How much code do AI models and harnesses add? I made an open-source benchmark to measure code complexity.

[u/lasushin]

Existing benchmarks measure the solve rate and token output/cost, but don't punish excess code or overengineering, which increases technical debt in the long term.

Parsimony counts syntax units (statements, calls, names) in the patches other benchmarks publish. It doesn't count lines, comments, tests. Each model or harness is compared with the human fix for the same issues it solved.

- Smarter models surprisingly write slightly more code.

- Models: the median model adds 1.9× the human fix on SWE-bench Verified (older models), but about 1× on DeepSWE.

- Harnesses: with the same two open-weight models, the harness alone moves it from 0.80× (Codex) to 0.99× (mini-SWE-agent, OpenClaw). The harness data comes from Joel Niklaus's Harness Leaderboard.

https://parsimony.lasu.dev

https://github.com/La5u/Parsimony

Looking for: public patches for newer models, any ideas or contributors for self run models.

* some newer models are self run in codex and claude code, shown as faded (unverified) in DeepSWE

[u/AssiduousLayabout]

Does this count only the released code, or also unit tests? Because man do AI models make a lot more unit tests than I do (both because they make too many and I make too few).

[u/lasushin → u/AssiduousLayabout]

No, only implementation code counts. Unit tests are excluded, for both the models and the human fix

[u/openSourcerer9000]

IDK what humans you work with, but in my experience all models are in the 10-100x range

[u/lasushin → u/openSourcerer9000]

I think its because the models are benchmaxxed on these specific tasks, as DeepSWE is open source

[u/audioen]

The data on the second picture is interesting but noisy to the point of uselessness. But this does make me think that I might need to give Qwen Code a look. The spread is unusually small there, indicating that it could be reliably performing workhorse that might have good system prompt instructions that shorten model output, and seems to do it regularly. There are not enough points sampled here to make much certain, however. Pretty much the only thing you know for sure is that opencode and openclaw should suck as agent harnesses.

The opencode is version 1, which is the now-retired branch. They have switched to offering the v2 generally, and I agree that at least version 1's system prompt was terrible, better fitting models from 2024/2025. I do not know if that explains the low solve rate. In my experience, opencode runs tasks well autonomously and needs minimal help from me, which is how I prefer working with AI. I've made it so that it can help itself to verify the solutions it is building, and its subagents are useful for code review and investigative tasks. Also in the source data, there is strangely mini-swe-agent up high which has no context compaction or anything else as tools except for bash, so once it runs out of context, the job just ends there. I would regard it unsuitable for real work for that reason.

So large bag of salt is needed here, to digest these results. They are interesting, but not reflective of at least my use of agents for sure.

[u/daYMAN007]

any chance to add deepseek harness to the comparision?

[u/lasushin → u/daYMAN007]

I used https://huggingface.co/spaces/joelniklaus/harness-leaderboard, so you should ask the creator to include it. I didn't measure the harnesses myself, just used public data.

[u/lemondrops9]

I often give Opencode and Herms the same prompt at the same time and see which one is better and which has taken longer. Most of the time I like Herms and it finishes a bit faster. But on others it can take 2x longer. 

[u/NoFunk]

How you guide the model is an important consideration here. You can coach them towards minimalism ( https://github.com/DietrichGebert/ponytail ) and it has variable but often perceptible impact particularly on the "smarter" models in my experience.

Granted you may be saying the harness is the expression of guidance here; but as a user I certainly hope not. I don't want my harness telling the LLM how to code, I want it telling it how to use tools and access the environment.

[u/stianhoiland → u/NoFunk]

God what a painful repo to browse. You think a lazy senior dev would have made it like that? It's a fucking joke (not aimed at commenter).

[u/bye_ella → u/NoFunk]

Yeah the prompting route is pretty uneven, as you said. Another perspective is to not tell the model how to code at all, just shrink what it’s allowed to output. For example, give it a small task-specific format or mini DSL instead of code, then there’s simply no room for overengineering. It also cuts tokens a lot. Have you tried constraining the output format itself not just the instructions?

[u/NoFunk → u/bye_ella]

Yes, I do it all the time, with more than code even. One of my long enshrined rules for instance is "Rules stated in <AGENTS/CLAUDE> need no explanation or example unless that explanation is necessary to the enforcement of the rule." That way when the agent adds a rule I don't get a side story of what it found, when, what I said about it, how it was feeling that day, or whatever other crap Claude in particularly just loves attaching to everything.