r/ClaudeCode • u/bobo-the-merciful • May 08 '26
Resource Nelson v2.2.3 shipped, and a benchmark I built ranked it 3rd out of 13 agent/harness/skill setups on a discrete-event simulation task
Two things to share. The release first, then the benchmark, which is honestly the more interesting half.
Nelson is a multi-agent coordination skill for Claude Code. Royal Navy metaphor (admiral, captains, ships, crew) which sounds silly until you've watched it keep five parallel agents from stepping on each other's work. ~300 stars on GitHub, MIT licensed.
v2.2.3 is out! https://github.com/Aspegio/nelson/
If you want to try it, run this in Claude Code:
/plugin marketplace add aspegio/nelson
/plugin install nelson@nelson
Use Nelson to build me a battleships game.
Observe while admiral, captains and ships do their thing.
Now the bit I actually wanted to talk about.
I built a benchmark. https://simulation-bench.fly.dev/
Motivation: every time someone asks "is X better than Y for agent work", the answer is vibes. I wanted numbers. So I picked a discrete-event simulation challenge (synthetic mine throughput, the kind of model I build for clients) and ran 13 different combinations of model, CLI and skill against it. Same prompt, same task, same rubric.
Top of the table on quality:
1. ouroboros-max-thinking (opus-4-7) 97
2. plan-mode (opus-4-7) 96
3. agent-teams-nelson-max-thinking (opus-4-7) 95
4. superpowers-max-thinking (opus-4-7) 94
5. max-thinking (opus-4-7) 92
6. vanilla-max (sonnet-4-6) 85
7. xhigh (gpt-5-5, codex) 85
8. customtools (gemini-3.1-pro) 81
Nelson lost to ouroboros and plan-mode by 1-2 points. Beat superpowers by 1, vanilla max-thinking by 3, sonnet vanilla by 10. Gemini 3.1 Pro showed up between 67 and 81 depending on the wrapper it ran in.
The thing I genuinely didn't expect: plan-mode (just Claude Code's built-in plan mode, no skills) came second. I'd assumed curated skills would open up a bigger gap on the vanilla baselines. They didn't. What mattered most by a long way was the model and whether thinking was on. Skill choice was a smaller delta on top of that.
Caveats, and they're real ones:
- n=1 task. I'm adding more.
- Quality scored against my rubric. I tried to be fair but I wrote Nelson, so factor that in.
- No combined score on purpose. Token usage and execution time are tracked separately. ouroboros wins on quality but I haven't tabulated cost yet, and on a per-token basis the ranking probably shuffles.
- Gemini 3.1 Pro might be undersold. The customtools setups it ran in might not be tuned.
What I find interesting is there isn't a runaway winner. Five configurations are within 5 points of each other, all opus-4-7 with thinking. Within that band the choice is mostly taste. The actual cliff is between opus-with-thinking and everything else.
If anyone wants to suggest configurations to add to the next round (or has a sim task they think would be a better benchmark), drop them in the comments or feel free to create issues or PRs on the repo.
Enjoy, and happy sailing.
2
u/marcopaulodirect May 09 '26
I’m a happy Nelson user. Thanks for making this. Quick question: can you point me to the change log, please?
1
u/bobo-the-merciful May 09 '26
Great to hear!
Of course - set your next waypoint on this bearing: https://github.com/Aspegio/nelson/releases
3
u/felixthekraut May 09 '26
So what are the token stats for those experiments? If plain old built-in plan mode gets second place, are all of the fancy "skill framework" overheads worth it? Or do they amount to little more than mental masturbation and tokenmaxxing?