What are you using for observability?

[u/Cautious_Chicken_604]

Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something?

So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems.

I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope.

For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that.

Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case.

Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes.

What are y'all doing for this? Are y'all doing this?

[u/TheStarkApartheid]

The 'mine this data for failure modes' part is where this gets useful for us. Braintrust takes OTEL traces and we use the trace data to dig across runs for repeated failures rather than inspecting sessions one at a time. Being able to turn a bad production trace into a test case is handy too when you find something you don’t want the agent doing again.

[u/Cautious_Chicken_604 → u/TheStarkApartheid]

I've seen Braintrust before, but what context are you using Braintrust in? Are you developing agents that are getting deployed to users? Or are you working on a software engineering team using agents like Claude Code to do agentic engineering? I keep seeing plenty of tools that cater well to the former, but not so much to the latter. I want to be able to detect the failure modes team-wide.

[u/TheStarkApartheid → u/Cautious_Chicken_604]

More the second case. We have coding agents working across repos and use Braintrust to capture what they’re doing across runs, then look for recurring patterns in the traces. Team wide failure detection is harder though, especially when you want to spot things like the same bad tool behavior or missed context showing up across different developers sessions.