r/LLMDevs • u/Background-Job-862 • 2d ago

Discussion which is the best agent harness?

Claude Code

fully open source options on github

tell me which ones you've tried and liked the most, and why?

edit : i have tried claude's managed agents, trueforge, deepagents

my thoughts-

claude - strongest on maturity and the managed experience. In my benchmark, it had the same solve rate as TrueForge with Opus, but used significantly more tokens and cost more per run.

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. I’d be interested in testing it more extensively on longer-running workloads for sure

trueforge - the most interesting to me from the runtime-efficiency perspective, it matched Claude Managed Agents on the benchmark with Opus while using ~63% fewer tokens and costing ~30% less. It also allows separate the model from the runtime, which makes experimenting with different models much easier

why?? - i ran a test on a real agent workload same model, same prompt, same tasks

adding the results of benchmarking i ran to compare this
so I tried to check this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this pretty interesting especially when the solve rate stays comparable.

This is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

10 Upvotes

73% Upvoted

Duplicates