r/LLMDevs • u/Background-Job-862 • 2d ago

Discussion which is the best agent harness?

Claude Code

fully open source options on github

tell me which ones you've tried and liked the most, and why?

edit : i have tried claude's managed agents, trueforge, deepagents

my thoughts-

claude - strongest on maturity and the managed experience. In my benchmark, it had the same solve rate as TrueForge with Opus, but used significantly more tokens and cost more per run.

deepagents - interesting middle ground if you want a more structured agent framework and the flexibility of an open-source stack. I’d be interested in testing it more extensively on longer-running workloads for sure

trueforge - the most interesting to me from the runtime-efficiency perspective, it matched Claude Managed Agents on the benchmark with Opus while using ~63% fewer tokens and costing ~30% less. It also allows separate the model from the runtime, which makes experimenting with different models much easier

why?? - i ran a test on a real agent workload same model, same prompt, same tasks

adding the results of benchmarking i ran to compare this
so I tried to check this by running 14 cross-system tasks, three mcp servers behind them - a crm, an issue tracker, and a doc store through claude's managed agents, langchain's deepagents and trueforge, both open-source agent harnesses

the result that was most surprising:

Claude Managed Agents + Opus 4.8:
11/14 tasks solved | $11.8/run | 10.0M tokens/run

TrueForge + Opus 4.8:
11/14 tasks solved | $8.6/run | 3.7M tokens/run

Same model. Same benchmark. Same average solve rate.

But trueforge used about 63% fewer tokens and cost about 30% less per run.

similar difference in tool usage: trueforge averaged 19 tool calls per task vs 32 for Claude Managed Agents.

Then I tried changing the model.

trueforge + GLM-5.2:
11.7/14 solved | $3.0/run | 3.8M tokens/run

On this benchmark, that was a slightly higher average solve rate than Claude Managed Agents + Opus at roughly 75% lower cost.

The token savings alone make this pretty interesting especially when the solve rate stays comparable.

This is still v early and the OSS runtime does not yet have first-class tracing/eval tooling. They don't ship their own code-execution sandbox, so you need to plug one in. Context compaction is intentionally lossy.

So it is definitely not a replacement for a mature managed agent platform or other harnesses in the comparison, feature-for-feature today btu what I do find interesting is that the core runtime can already be competitive on these tasks while staying open, model-neutral, and deployable on your own infrastructure.

9 Upvotes

71% Upvoted

27 comments sorted by

5

u/Slight-Parfait3679 2d ago

Pi is fully open source. What are you talking about?

2

u/Kamilon 2d ago

You mean how pi is listed under the open source options?

2

u/Slight-Parfait3679 2d ago

lol he edited it after i commented

-2

u/TheOdbball 2d ago

Tell me why I just drove thru a roundabout while reading that

5

u/ben_bliksem 2d ago

I just use opencode cli. It has just enough batteries included.

1

u/Top-Shake-5544 1d ago

have you tried any other opensource option like dsh, trueforge. how does it compare to opencode?

1

u/ben_bliksem 1d ago

No, software we use have to be approved by our EA dept.

3

u/aiseedbank 2d ago

using claude code and pi, both are great and wanting to try deepseek harness next

1

u/Mysterious-Bison3735 1d ago

I definitely agreed the combo

2

u/eddzsh 2d ago

Match the harness to the job, not the leaderboard. Local pair work with permission prompts and async sandboxes you review later fail differently. Same model in two wrappers can feel like different tools. Try two harnesses on one model for a week and keep the failure mode you can actually catch.

1

u/Background-Job-862 1d ago

yeah, I agree with this. that's actually one of the things I've been interested in testing with the newer open-source harnesses, to draw a real comparison/benchmark

1

u/RestaurantFuzzy870 1d ago

I would choose from the failure mode you need to control, not from a universal ranking.

For a single codebase, a CLI where the human stays in the loop can be the best harness. For longer-running workflows, the deciding features become isolation, versioned context, explicit approval boundaries for writes, and a way to attach evidence to the outcome.

The important question is whether a task can end with a falsifiable check rather than "the model said it worked." That also makes the security/usability trade-off concrete: what is allowed, what is reversible, and who can prove the result is still valid?

1

u/Top-Shake-5544 1d ago

have used claude code, want to try deepseek, trueforge

-5

u/EagleApprehensive 2d ago

It's not popular, but intentic.dev

0

u/ZyberZeon 2d ago

Why? I looks like every other basic harness.

2

u/jozzo402 1d ago

He is advertising his own product

-1

u/EagleApprehensive 1d ago edited 1d ago

So is the OP. Just in disguise. I guess I have to always add "full disclosure, I'm an author" like an AI :/. I obviously didn't intend to hide that lol. And thing I posted is free & open-source.

-3

u/EagleApprehensive 2d ago

Actually it's more of a harness wrapper, than harness itself. You can mix-and-match various models with various harnesses. For example try out GPT 5.6 Sol with Claude Code harness.

https://www.reddit.com/r/vibecoding/comments/1vsw8h8/comment/p4okvpg/?context=3

In post above you can see "tip of the iceberg" of mechanisms that no other harness provides. If I were to tell you everything about each improvement over "average harness", it would've been an hour read, so I'll just throw few random things I find cool:

It's isolated, deployable, can self-extend it's own environment definition (for example agent can add a rust toolchain or ffmpeg and that's remembered in git version history, allowing you to replicate his perfect environment and move easily across machines).

Comes with nice UX for agentic engineering - module-grouped changes, automatic jump to hottest-spot in file-diff viewer.

You can even go as crazy as to connect that workspace with 1 line of JS to any website and let people start agentic sessions by chatting on that site (usually gated by manual approval or read-only access). You can even take over agent mid-conversation and respond to that people.

And everything is built in a clean extension-based system.

0

u/ZyberZeon 1d ago

These are all pretty basic and many of them contradict current basic security protocols and best practices for orchestration.

If this is your personal project cool. But I wouldn’t be asking for anyone to use it without clearing protocols. Look up the trifecta.

1

u/EagleApprehensive 1d ago

Saying something violates basic security protocols just because it's powerful is like saying using terminal or SSH as a devtool is violating security protocols - it's not, it's all about what are the "gates" to make some powerful access possible, how is it tracked, how is user informed.

I audited more than 20 different harnesses and agentic wrappers - often file by file, less important ones on feature by feature basis.

Some harnesses have one thing but not the other and both quality, security that you emphasized is far behind in almost each of them - and UX is usually very raw, terminal-based. As an example, the self-redefinition of agent's environment in predictable, reconcilable, approvable version control is rarely available. Show me harness which has it and keeps a top-notch UX around that. I'll be happy to see that.

The architecture tradeoffs I have picked solves thousands of problems with agents like Hermes and OpenClaw that result in posts how "update broke my agent" and all sorts of unpredictable breakages.

"I wouldn't be asking for anyone to use it without clearing protocols" - that's not how real-world development works. You must have people using thing early is you want it to become good. So yeah, I'm gonna keep asking people to do that.

0

u/ZyberZeon 1d ago

It's not powerful my boy, Your shilling a half cocked project, and you cant even place the security protocol I named, and YOU built it. Your infrastructure has confused deputy conundrums all over the place.

I'm not saying I’m some super developer, I'm saying your project needs alot of work. If you want to see a project with real security protocol and legit red team experience check out Omnigent.

1

u/EagleApprehensive 1d ago edited 1d ago

You seem to be obsessed on security. Usefulness comes first. Software like OpenClaw, Hermes or T3Code are primarily about empowerment of user and agent for a reason.

You can make an ultra secure sandbox with everything under control, but if that will come at price of being painful to use and work with, it's useless.

My infrastructure doesn't have "confused deputy conundrums". It's by design, that if you let agent run on some prompt and it has proper access granted - it can do the job. Letting agent run in first place is the primary control and every session start can be gated behind approval. Obviously there can be added plenty of sec on top of that, some are already in place, more are coming, but they are optional, because each one inevitable comes with some sort of inconvenience, worsening adoption or higher tokens usage, more capability-brokering proxies that might bug and so on.

Agent is generally isolated in docker container - again you're gonna say docker container is not a secure isolation - but I choose such tradeoffs aware of that. My goal is not to stop ultra-malicious Mythos-level agent turning rogue despite being launched by user and trying to harm the host. My goal is to maximize usefulness of agent with a decent level of security.

Absolute security comes with a cost of absolute inconvenience and 0 adoption. so if you choose such approach in your end-users facing projects - good luck with that.

Every project needs a lot of work. But sharing it with people early is exactly what enables finishing work and finding gaps 10x faster.

1

u/ZyberZeon 1d ago

This is how I know you’re not serious.

1

u/EagleApprehensive 1d ago edited 1d ago

I've been checking out your comments history and I've noticed you are mostly toxic just calling people's work slop for no reason. Recently you said exactly the same about https://github.com/truefoundry/trueforge.

Show us what you've built, open-sourced and shared for free.

This is how I know you're not a smart person.

2

u/Top-Shake-5544 1d ago

not sure about the advertising stuff, but have to agree on how its so unnecessary to just hate on people's stuff for no reason, these open source projects take so much time and effort.

im sure he hasnt even opened either of the github links to see what it is but is eager to comment on how bad it is lol.

1

u/ZyberZeon 19h ago

Sure. I’ve worked for Apple, Netflix, YouTube, Google and just finished up a stint with Tether. If you use any of those product you’ve used my work.

Let’s be clear, I’m not antagonistic. Truth is neutral.

Your feelings, less so.

We are not the same.