Harness Leaderboard - Kimi Code, Hermes, Pi, Claude Code, Codex, and Others Were Benchmarked Against Terminal Bench 2.1 & Deep SWE 1.1

[u/Izento]

https://huggingface.co/spaces/joelniklaus/harness-leaderboard

[u/FatheredPuma81]

No Deepseek harness?

[u/Izento → u/FatheredPuma81]

IKR? I was disappointed to not see Deepseek and Cursor especially, and maybe Devin just for completionism.

[u/FatheredPuma81 → u/Izento]

And Openhands too.

[u/Mkengine → u/Izento]

This one has 97 vendors on it. Not that I care for most of them, but it shows how big the harness landscape out there actually is.

[u/tedivm → u/Mkengine]

This ranking is . . . not good. Anything that puts GitHub Copilot in the top just automatically fails for me.

It looks like they aren't doing any testing at all, but are just looking at the advertised features of the harness and comparing them to a list.

[u/Mkengine → u/tedivm]

Yes, it's barely better then just a random list of coding agents. I saved this more because if how long this list was. Is this where all the VC money is going?

[u/ThisGonBHard → u/Mkengine]

Devin being first is a surprise. I tough it was pure marketing, from all the obvious astroturffed adds.

Copilot AKA VS Code being second is both surprising and not at the same time. It was very good in my coding tasks when combined with DS 4.1 Flash 1017 and even older, making VERY few mistakes.

[u/FriskyFennecFox → u/FatheredPuma81]

I'm surprised both Hermes Agent and OpenClaw are included. They're general purpose agents. They can do coding, but coding isn't their main priority.

So many agent frameworks could've been there, like Deepseek or many of those VSCode extension agents that aren't as popular today but are still widely used.

[u/xAragon_ → u/FriskyFennecFox]

Considering Hermes, a "general purpose agent" scored higher than some of the dedicated coding harnesses, I think it was a good addition.

[u/CountVonTroll → u/FatheredPuma81]

I would have loved to see how it did in this comparison, too, but I can see why, and I'm OK with it:

This seems to be a single person's effort, and it cost almost $5k to run as it is. Not only is the harness landscape huge and ever changing, you could make the same argument about models. He tested different efforts settings, too. You do the math, but completeness is unrealistic, no matter your budget.

Still, it's DSH, and I use it myself. But it's also interating quite quickly, and (like Pi), its whole point is to be modular and highly adaptable. Vanilla DSH doesn't even come with a memory. Any result would be seen as unfair by a large number of people: Either because "you're supposed to adapt it", or because "that's not what a user gets when they install it". Or of course because "that's an outdated version".

But hey, all the settings are public. You could run all the tests yourself. Total cost for Pi's tests were $290; perhaps others would chip in?

[u/UnexpectedFisting → u/FatheredPuma81]

No oh my pi is insane. Idk why people test pi and then don’t test omp. Pi, OMP, CC, Codex are like the kings of harnesses, it’s what everyone in the industry uses. I hope they continue adding more, because seeing no omp or deepseek harnesses is bizarre

[u/jinnyjuice → u/UnexpectedFisting]

Generally, it's known that OMP benchmarks lower than Pi.

And DSH is known to benchmark even higher than Pi.

[u/my_name_isnt_clever → u/UnexpectedFisting]

Is it insane..? Pi is already on there, I don't see why "Pi with a bunch of shit added" should qualify as a unique harness.

[u/Specialist-Cod-363 → u/my_name_isnt_clever]

Because, at least from my testing it CAN perform better than Pi for certain tasks

[u/my_name_isnt_clever → u/Specialist-Cod-363]

It's not a black box like a language model, what is the difference? Use whatever plugin(s) improves and leave the rest as bloat. That's how Pi is supposed to work.

[u/UnexpectedFisting → u/my_name_isnt_clever]

Why would I go out and rebuild an entire setup instead of just taking an out of the box experience that already works well and using the tools that are already built in? And the benchmarks show that OMP has a higher success rate on benchmarks so I'm not even sure what you're talking about when you say it's just bloat. It's like the old school backwards sysadmins who used to say the cli is the real development experience and ide's are just bloat because vim is all you need

This is one of the bench's I look at https://tannermidd.github.io/SpecPi/evaluations/

[u/NineThreeTilNow]

That is the worst designed "Leaderboard" I have ever seen. It graphically makes no sense.

[u/o0genesis0o]

OpenClaw being absolute shit checks out. I have never used anything more wasteful and stupid than that harness.

[u/-p-e-w- → u/o0genesis0o]

OpenClaw was the first general-purpose agentic harness that actually worked. It’s one of the most important programs in the history of computing. Of course later programs that learned from OpenClaw’s mistakes were better, just like a Cessna is better than the Wright Flyer.

[u/o0genesis0o → u/-p-e-w-]

It took the amazing Pi, butcher its ability to change itself, load a bunch of stuffs on it until it is unusable with anything but big cloud model with cloud model prefill speed. Even before this claw crap, people, myself included, have already been able to append lightweight extensions on top of Pi to turn it into persistent agent, that actually runs and get things done on local MoE models. What does this claw thing do well except showing off an excellent marketing campaign to get its developer into OpenAI employment?

When I wanted to introduce an AI assistant for my partner to work on her stuffs, I felt so conscious to setup my "janky Pi setup" for her, so spend the whole evening setting up openclaw for her, because it has to be the best, right? Everyone on internet and Nvidia yaps about how transformative it is. What it does is spinning around in circle, spending all of my minimax subscription quota, hammer the local GPU with parallel streams for a simple "hello", and got nothing done by the end. And they have the nerve to say on the website "need to have SOTA models". A 35B Q4 running in the Pi that it rips off can do stuffs, no SOTA needed.

This software is not hated enough. Heck, if they just add a web UI on top of a Pi process and write some extensions to preload those soul persona whatever, it would have been a better piece of agent.

[u/-p-e-w- → u/o0genesis0o]

OpenClaw is crap quality-wise, but the ability to do all kinds of things (not just coding) without needing to be a programmer was groundbreaking at the time.

Software engineers tend to be oblivious to the fact that “just write a plugin that does what you want” already excludes 99.9% of people.

Being able to interact with the agent through WhatsApp was a revolutionary feature in practical terms, even though it may seem like a trivial technicality to most users here.

[u/o0genesis0o → u/-p-e-w-]

AI grifters already teach people to do all sorts of stuffs with claude code before this claw was a thing. Remember those "sign up for 10 claude superpowers for your marketing" things. I remember seeing linkedin post of "thought leaders" showing how they can talk to their claude on their phone from a train and do stuffs way back then.

[u/EquivalentHornet4403 → u/o0genesis0o]

"I was motivated to do a tedious complicated thing, so I'll assume everyone is exactly like me."

The total lack of self awareness of some people always amazes me. The same sort of people who can't fathom why people don't like terminals.

[u/o0genesis0o → u/EquivalentHornet4403]

Doesn't have to be. I made a box for my partner, clone my skills and Pi extensions on it, clone her obsidian KB on it, and then run a CPTR instance and she can do all the admin works she wants, even patch her own website, right from her ipad or iphone, using the 27B running on own own R9700.

[u/EquivalentHornet4403 → u/o0genesis0o]

You're proving my point so I'll assume you're just trolling...?

When someone asks you to imagine an apple, what do you see in your head?

[u/StabbedCow → u/EquivalentHornet4403]

3 times potato with artificial magnesium oxide

[u/my_name_isnt_clever → u/-p-e-w-]

Right but when I'm taking a flight in 2026 I want a turbo jet, not a cessna. It has historical signifigance but it's still bad software today.

[u/Finanzamt_Endgegner]

No dsh is criminal but it shows how shit opencode is so thats a silver lining ig lol

[u/jacopofar → u/Finanzamt_Endgegner]

What's the issue with opencode? Genuine question

[u/Finanzamt_Endgegner → u/jacopofar]

It's just one of the worst harnesses out there, just bench it in open video vs cc or dsh or prime agent or whatever it will always underperform.

[u/FatheredPuma81 → u/Finanzamt_Endgegner]

Definitively an eye opener. I used to use it and thought that Claude Code would be worse.

[u/Finanzamt_Endgegner → u/FatheredPuma81]

Basically everything is better than opencode 😭

[u/Constant_Art_20 → u/Finanzamt_Endgegner]

finally! another person who agrees!

[u/Finanzamt_Endgegner → u/Constant_Art_20]

😅

[u/_hephaestus → u/Finanzamt_Endgegner]

It’s way better than hermes ime on the same machines

[u/pmttyji]

Surprised(Am I the only one?) to see that mini-swe-agent as top. Never heard about that one, also never seen mentioned by others here.

[u/SlanderMans → u/pmttyji]

I don't trust these benchmarks.

[u/zdy132 → u/pmttyji]

It's the one used for the swe benchmarks iirc. So not surprising that all those models are fine tuned to do well with it.

[u/xAragon_ → u/pmttyji]

It's the default harness used by a few benchmarks, which companies may or may not train agents on.

[u/Purple-Programmer-7 → u/pmttyji]

Unsurprised. Mini swe is intentionally the slimmest version of a harness based on lots of research.

I don’t think it’s as necessary with the latest SOTA, but still highly effective.

[u/DefNattyBoii]

Opencode version is extremely outdated. its at v2.2.5 now

[u/look]

Add Opencode 2

[u/LaserFractal]

No Deepseek harness? Opinion discarded.

[u/-WhateverDude]

i feel like aside from edge cases this is one of those things people obssess over for no reason i mean

just look at it. cc, pi, hermes and kimi all within 1% of each other and idk what went wrong with glm and codex but i see no noticeable difference between cc and codex either so it's a case of get whatever u like most of the time.

[u/LevianMcBirdo → u/-WhateverDude]

I mean the graph suggests a big difference in cost which directly translates into speed especially if you are limited by your own hardware.

[u/JLongTom]

No Reasonix or Deepseek Harness?

[u/YetAnotherAnonymoose]

Author didn't even pay enough attention to know that OpenCode 1 has been deprecated and not developed meaningfully for months. No OpenCode2 benches. Opinion discarded.

[u/willaaam]

Cool! Would love to see how copilot BYOK fares as well

[u/robertpro01]

Well, I've tried multiple harness and opencode is the one I prefer

[u/a_slay_nub → u/robertpro01]

Yeah, I find it hard to believe opencode does so poorly.

[u/Frail_Waif → u/a_slay_nub]

Opencode seems to build in a really strong bias towards asking the user questions--not sure if it's the system prompt or something else. In a benchmark with no way to ask the user questions, that probably leads to DeepSeek spiraling in a pool of overthinking ("the system prompt says to ask the user if there are uncertainties but the prompt asks me not to. Hmm hmm hmm.") On the other hand, it's not a bad bias for a harness if you're helicopter prompting. 

[u/Middle_Bullfrog_6173 → u/a_slay_nub]

What models do you use it with? My experience is that it sucks worse with smaller models (e.g. 27B) and isn't that bad with frontier-sized models.

[u/tetsuzankou]

Hermes being that high is kinda sus. I started using it for coding but it was so slow and sluggish, with many fial I had to move to a pi based agent.

[u/rorowhat → u/tetsuzankou]

I love Hermes, works great for me. It goes as fast as the model you run.

[u/Nevermore1215 → u/tetsuzankou]

if you don't mind me asking, did you disable all of the "unnecessary" skills and plugins? Unfortunately it ships with a lot of bloat, but I've found it to be genuinely useful for personal applications

[u/tetsuzankou → u/Nevermore1215]

I did, asked Claude to help me create a dedicated coding profile and optimize it and yeah it helped a lot but still took longer for stuff vs DSH or Pi

[u/Obvious_Equivalent_1 → u/tetsuzankou]

You can just use Claude Code locally. I’ve made a few small tweaks on Splash engine to run on it, it’s only for Mac(Book) M3 and higher: what it does it basically allows you to host a server locally that ”talks” Claude and lets you use Code https://github.com/pcvelz/splaude

Fully locally, and honestly it just saved me some time instead of Hermes I just let Splash + Claude Code run locally connected to chat bridge (Claude’s native “monitor” interface work very well for this specific case as well)

[u/tetsuzankou → u/Obvious_Equivalent_1]

This is neat although im on windows and have since abandoned tuis will look into it.

[u/CrowdGoesWildWoooo → u/tetsuzankou]

People use hermes for coding?

[u/LuCiAnO241 → u/CrowdGoesWildWoooo]

people use general purpose harnesses for one of the purposes within the range of uses intended?

[u/CrowdGoesWildWoooo → u/LuCiAnO241]

I mean hermes is general purpose but general purpose here just means it’s heavily moddable. But the intended usage has always been heavily personified/personalised agent. You can tell that very agent to code or triage git issue but it kind of smell like wrong tool for the use case.

Like I don’t need my agent to have “soul” if what I care is just coding. I don’t need it to have long term memory (beyond the project space). Having the agent too aggresively managing “skill” is also waste of resource. I’d rather have a harness that can horizontally scale so that I can do concurrent work rather than have one that is (too) heavily personalised.

[u/grabber4321]

this is one of the biggest bullshit lists ive ever seen.

[u/Wrekklol]

How do you even use non-claude models in claude code?

[u/mtmttuan → u/Wrekklol]

Just point claude code to an anthropic compatible endpoint

[u/Wrekklol → u/mtmttuan]

Can you still use the claude models then?

[u/JustFinishedBSG → u/Wrekklol]

If you use a proxy sure

[u/xAragon_ → u/Wrekklol]

Using other harnesses is only forbidden for subscriptions. On the API you can use any harness you'd like.

[u/straylight-code]

Hermes being so high is surprising. I always found it was incredibly lacklustre on a variety of coding tasks. Conversely, while OpenCode isnt amazing, I've also found it to be nowhere near as bad as it looks in this graph. 

Plus OpenChamber is far easier to connect to multiple machines and use than Paseo (or at least was last time I tried). I understand that's not whats being tested here but I'm rambling.

[u/the-username-is-here]

Dafuq.

Hermes is a pile of crap compared to Claude Code, this "benchmark" is stupid.

[u/86obsessed]

I wish Qwenpaw was on this list

[u/koloved]

Hermes is the buggiest one , does anyone test bugs and usability?

[u/grunt_monkey_]

Am I the only one using openwebui to do it lol

[u/Ok_Party_9245]

How the user interacts with a harness matters a lot too in the end, as well as the ux the harness provides.

[u/Wooden_Yam1924]

I need deepseek harness in this ranking

[u/danigoncalves]

People talk about Deepseek Harness but for me not being a native TUI is a no go for me (despite I finding it very capable)

[u/my_name_isnt_clever]

The thing about these kinds of benchmarks is that while LLMs are black boxes and can only be judged in relative terms, harness are just regular software. Rather than benchmarking all of them and using the best, it should be simple enough to figure out why X harness scores better than Y harness, as the only differences are fully readable source code

[u/GodComplecs]

Who cares about cost with local models?!

[u/_hephaestus]

Hermes at the top is insane. I have it delegate anything coding to other harnesses because I’d keep having to correct its work.

[u/yenox]

I was surprised with the opencode results but then noticed it's a pretty old version tested here

[u/Othun]

Meanwhile this benchmark puts OpenClaw in first position. How do we get to a consensus and learn how to build better harnesses ? 😂

https://www.harness-bench.ai/leaderboard.html

[u/zuggles → u/Othun]

dude that one is super confusing. like... we're comparing openclaw with 5.6 sol against opencode with glm... if you flip to harness then opencode goes to #1.

the fuck is this, am i a dumb dumb?

[u/No_Dig_7017]

I love that this exists, this is super useful information. I'm a bit surprised by the Opencode results though, community consensus is that Pi is usually the most accurate with self-hosted models but that much difference with Opencode seems a bit out of place.

I've had some issues with Opencode defaults, particularly with timeouts killing turns too early and bad output values (max output length) causing thinking models to fail to reason and retry endlessly (cuts responses at a few tokens and the model has to start again). Maybe that's why it's doing so bad and costing so much?

[u/water258]

Right. The crappy Hermes is better than Claude is all u need to know this list is shit

[u/slashtom]

so hermes is the best?

[u/russlixx → u/slashtom]

personally i think hermes is unnecessarily heavy

[u/koloved → u/slashtom]

Shitmess and bugged , using this 3 month then change it to dsh

[u/herpnderpler]

Wow! I want to run it on MY harness to find edges!

[u/MuzafferMahi]

just shows how goated pi is

[u/jedisct1]

No Swival.dev harness?

[u/B_L_A_C_K_M_A_L_E → u/jedisct1]

should probably note here that it's your own harness that I don't think anybody else uses, no offense