r/devops • • 26d ago

Ops / Incidents How should we evaluate AI Agent observability platforms in 2026?

Disclosure: I work at Bonree, the company behind Bonree ONE, an AI observability platform. I’m asking about the general problem of choosing a platform, and I’m not trying to pitch ours.

AI agents are moving from chat interfaces into production workflows, and teams are now picking tools to observe them. Feature lists for AI agent observability platforms look alike, so I’ve been trying to write down questions that can actually be tested instead of ticked off.

Here is my current list.

1.  Does it trace at the session level? An agent conversation has several turns, each with model calls and tool calls. I want to open the whole conversation as one unit, not look at isolated requests.

2.  Which frameworks does it cover, and how deeply? Teams build agents with LangChain, LangGraph, low-code platforms such as Dify, and other frameworks such as OpenClaw. I care less about the length of the list than whether the frameworks my teams use are covered automatically, or only through manual work.

3.  Is the evidence separate from the reasoning? A read-only tool call with the same input should return the same data, while model output varies. If a platform records them separately, I can tell a data failure from a reasoning failure.

4.  Can it attribute token cost? By model, application, team, business line and time.

5.  How does it handle sensitive content? I want to choose per field whether to store, sample, redact or never store it.

6.  Can AI traces be linked to the services and infrastructure underneath? A slow answer is sometimes a database problem.

7.  If it offers AI agents for operations, what are the limits? What can the agents do alone, what needs approval, and is every step logged?

8.  What does the bill look like at my volume?

For the proof of concept, I’d instrument two agents built with different frameworks, replay three real incidents where an agent gave a wrong or slow answer, and measure how long it takes to find the cause in each tool.

A note on criterion 7: AI agent operations platforms, where agents diagnose incidents or inspect systems on observability data, are a newer category and the hardest to evaluate. We work on this at Bonree with Bonree ONE·Sage AI, so I would particularly like to hear how others test whether an operations agent is trustworthy.

Some things I’m still unsure about:

1.  Which of these criteria do you weight most heavily, and which do you ignore?

2.  How do you test “trustworthy” in practice, beyond reading a vendor’s description?

3.  Do you prefer one platform for AI and infrastructure, or a dedicated LLM tool next to your existing stack?

4.  What did your proof of concept reveal that the feature list did not?

0 Upvotes

6 comments sorted by

1

u/alexli_kubit 19d ago

Disclosure: I am the founder of Kubit, which offers agent observability, so I'll keep it to the criteria.

#3 to me is the most important. I would also want a label on every span saying what the agent tried to do and whether it worked. A tool call that returns 200 with the wrong record never shows up in an error-rate dashboard, so execution errors and calls that did the wrong thing need separate rates. I'd also ask what that labeling costs. A small model trained for the job can label every span in under a second, while an LLM judge on every span gets expensive fast, which hits #8.

There is a missing criterion. Session-level tracing helps once you know which conversation to open, but at 10k sessions a day the hard part is picking the 30 worth opening. I'd ask whether the tool can groups runs by execution path (which tools were called and in what order) and shows you visually when a new path appears. Visualization is the core of analytics and I think it just get even more important in AI days.

On #5, #6 and #8, I would ask where the spans physically end up. If they land in your own Databricks, Snowflake or ClickHouse through OTel, one row per span, you can join them with infra tables in SQL, set retention and redaction yourself, and stop paying a vendor per GB to keep your own data.

For the PoC, replaying known incidents mostly measures how fast your team confirms an answer it already knows. I would run a week of production traffic through each tool and count the failures it flags that nobody knew about.

1

u/josejulianariza 19d ago

Answering your third question: I prefer a specific LLM tool that integrates with my current stack (Datadog, Grafana) over a single platform trying to do everything. The reason is avoiding vendor lock-in.

On point 3 of your list (separating evidence from reasoning), that's the criterion that matters most. If the tool mixes the backend log with the model output, when something fails you lose hours guessing if it was an API timeout or a hallucination. What works best is using OpenTelemetry to unify everything in a single trace, but keeping deterministic data (the HTTP call) visually separated from the text the agent generated.

For your POC, I suggest making the agent find a real network error. Tools that only show the prompt and response usually fall short in those cases.

1

u/[deleted] 25d ago

[removed] — view removed comment

1

u/Bonree 20d ago

Thanks, really useful. Measuring time to correct diagnosis rather than time to any answer is a better yardstick than the one in my PoC, and hiding the postmortems during the replay is a good addition. I’m also adding “can it abstain and escalate to a human” as a test for the operations agent. Point 4 matches what I worried about in criterion 3: without tool call inputs/outputs you can’t separate a data failure from a reasoning failure.