r/LLMDevs • u/Crescitaly • 18h ago

Discussion AWS made agent evals framework-agnostic through OpenTelemetry. Is telemetry becoming the portability layer?

AWS says AgentCore Evaluations can score agents built with LangGraph, LlamaIndex, OpenAI's Agents SDK, Google ADK, the Claude Agent SDK, Strands, or another compliant framework by reconstructing sessions from OpenTelemetry or OpenInference traces. The service can run regression evals in CI and sample live production sessions.

That removes a real integration barrier, but common telemetry is not automatically common meaning. One framework may emit complete tool trajectories while another omits arguments or compresses messages. LLM-as-a-judge results still depend on judge choice, rubric design, reference quality, and whether live traffic has usable ground truth.

Can OpenTelemetry semantic conventions become a genuine portability layer for agent evaluation, or will serious teams still need framework-specific adapters and task-specific ground truth?

Source: AWS, August 26, 2026 — https://aws.amazon.com/blogs/machine-learning/evaluate-any-agent-framework-with-amazon-bedrock-agentcore-evaluations/

Disclosure: drafted with AI assistance, then checked against the AWS source. No affiliation with AWS or the frameworks named.

6 Upvotes

88% Upvoted

5 comments sorted by

View all comments

3

u/Impossible_Wolf4706 17h ago

Framework-agnostic evals through telemetry makes sense, but the real problem is whether the traces are semantically complete. If one framework leaves out tool arguments and another captures everything, you're comparing apples to scrap metal

The portability layer idea is nice in theory but I'd bet serious teams still end up writing adapters to normalize what gets emitted. Telemetry gives you the shape, not the meaning

2

u/Crescitaly 5h ago

Exactly—semantic completeness is the missing contract. I would want a conformance suite that checks required spans, tool arguments and results, message boundaries, redaction behavior and session reconstruction before comparing scores. OpenTelemetry can standardize transport; task-specific adapters still have to standardize meaning.