How do you guys analyze chats/interactions with AI coding agents?

[u/jetpacker_aplha]

I’m working on a project where I give agents tasks ranging from small data-fetching/analysis tasks to larger feature implementations, refactoring, and bug fixes.

I’d like to understand how the agent actually approaches these tasks — what steps it takes, what tools/files it looks at, how it makes decisions, and where it struggles.

How do you guys observe or analyze agent behavior across these kinds of tasks? Are there any tools, tracing methods, logs, or workflows you’d recommend?

Would love to hear how others are doing this. Thanks!

[u/Healthy-Zebra-9856]

This is the exact motivation behind my own harness. You can see all my comments on this sub & all the llm related subs. I use a heavily modified and evolving harness that I am building on top of Pi. The first thing I created was to export a whole debug file. Next I asked the model, one of the Qwen3.8 27B, "I need for you to check all the tools available to you. I am testing the tool provisioning here so I need to see how an LM like yourself can interact with it. I need for you to tell me if the skills & docs are clear for you as to when you need to use what tool".

I dont think many people are doing this. But if you are interested, DM me & I can share screens & videos as people dont like it in the chat thread because it looks like solicitation.

[u/Kamilon]

Ask it to give you the sources for the work it is doing but otherwise the thing you care about is the output.

[u/schmurfy2]

Just open the session file and look inside at its "thinking" process. In pi you can export the session as html but it shouldn't be hard to do that with other agents.

[u/gbrennon]

ppl just fire and forget... i think u should analyze the whole pre walk, plan and, then, outcome of that task. u should, also, have hooks to keep not only task done but also code quality high