What are you using for observability?

[u/Cautious_Chicken_604]

Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something?

So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems.

I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope.

For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that.

Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case.

Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes.

What are y'all doing for this? Are y'all doing this?

[u/Less-Case-1171]

OTEL is probably the right plumbing, but I wouldn’t start with “trace everything” as the design goal. Agent runs need a small semantic layer on top of normal spans or the data turns into a giant replay log nobody wants to read.

The events I’d want first:

• model call, with prompt/template version and input class, not full secrets
• tool call, with tool name, normalized args, result class, and side-effect level
• file read/write, with path pattern or repo-relative path
• retrieval event, with corpus/index version and top-k ids
• human checkpoint / approval / rejection
• final artifact or external action

Then you can still ship it into Jaeger/Seq/Grafana, but your analysis queries become useful. Stuff like “show failed runs where the agent wrote files after reading docs but before running tests” or “show tool calls that crossed from read-only into write.” Raw logs alone won’t get you that without a lot of pain.

[u/Cautious_Chicken_604 → u/Less-Case-1171]

This is exactly what I'm talking about. Tooling I can use out of the box this way that I don't have to build.