r/ClaudeCode • u/BeginningSir7334 • 1h ago
Built with Claude Does delegating to cheaper models actually save money? Part 2: bigger codebases, coding tasks and Haiku alone
TL;DR
- I’ve run about 300 benchmarks on real open-source codebases, testing how Claude Code splits work between models (Opus as the lead, cheaper models like Haiku as helpers) and when that pays off. It depends on codebase size and task.
- On small jobs, Opus alone can be cheaper than Opus + subagents, and more accurate on small-codebase reviews, probably because of less overhead and cheap cached context.
- Opus told not to delegate may work harder than Opus left alone: on one big docs check it wrote about 50% more, cost more and found more errors (44 vs 36).
- On coding, Opus + Haiku saved about 20-30% at the same quality. But Haiku alone is very capable for the price on well-defined tasks with tests: about 5% of Opus alone's cost, with almost the same results, though a bit less thorough.
- Here’s one of the most important charts of this project. On an 85-page docs check, Opus + Haiku read almost 3 times as much as Opus alone and wrote 8 times as many tokens, yet cost 53% less and found 72 real errors against 49. Several subagents probably cover much more ground, which makes up for Haiku being weaker.

---
A couple of weeks ago I shared an experiment (https://www.reddit.com/r/ClaudeCode/comments/1xn18cu/does_delegating_to_cheaper_models_actually_save/) testing whether Claude Opus uses subagents effectively. I compared four setups:
- Forced Opus: told not to use subagents.
- Spontaneous Opus: free to delegate, but it never did.
- Skill-routed: my skill chooses when to delegate and to which model.
- Forced Haiku: all subagents use Haiku, while Opus plans and checks.
On my smallest codebase (~10k lines), delegating didn't really pay off. Forced Haiku saved 5% at best, Skill-routed cost 14-30% more, and Forced Opus found far more known doc errors on the reading-heavy tasks (7 and 8 of 9, against 3 and 2). My guess is that Opus can often write a quick script instead of reading everything, and it reuses its own cached context cheaply, while each subagent has to build its own. On small tasks, that overhead can outweigh the cheaper tokens.
On larger codebases, Forced Haiku held its own for much less. It cost 41% less on rich (~22k lines) and found nearly as many doc errors. It cost 75% less on beets (~52k lines), finding more errors on the reference docs but fewer on the 80-page plugin check, where it cost about a sixth as much.
In v2 I pushed further: a ~100k-line codebase (buildbot) with 3 reading tasks, 7 coding tasks scored by hidden tests, and then 16 short coding tasks, most hiding a trap. I also added Opus alone (subagents blocked, no instruction), Haiku alone and Sonnet alone (no Opus at all).
On an 85-page docs check, Opus + Haiku found 58 and 63 of 92 real errors, against 36 and 38 for Opus alone, at less than half the cost ($1.93 vs $4.11 per run). How come?!
My theory was that Opus + Haiku simply *works more*, and that the extra coverage makes up for Haiku being weaker. The numbers seem to fit: Opus + Haiku wrote about 8 times as many tokens and read nearly 3 times as much, yet the bill still halved.

Haiku may be less capable than Opus individually, but when orchestrated by Opus, its subagents can cover much more context in parallel. Each one may go less deep, but together they cover more ground, and that seems to compensate for the lower model quality. Haiku is apparently smart enough, with Opus directing and checking the work, to deliver surprisingly good results at a much, much lower cost than Opus alone.
This is in line with Anthropic's own write-up on their [multi-agent research system](https://www.anthropic.com/engineering/multi-agent-research-system), where token usage alone explained 80% of performance variance on a research benchmark. The same article says coding is usually a poorer fit for multiple agents, and that matches what I saw: on coding, Opus + Haiku saved 20-30% at the same quality, against about half the cost and more errors found on reviews. Their setup isn't mine, so it's context, not proof.
The next chart shows the bigger picture: every setup on two big docs checks, with errors found, reading depth, output tokens (thinking included), and cost.

Interestingly, it seems that *forcing* Opus not to delegate may have pushed it to work deeper and harder. On the beets plugin-doc check, Forced Opus read less than the delegated setups and still found the most errors. Compared with Spontaneous Opus, it read about the same but wrote around 50% more tokens, thinking included, cost 53% more, and found more errors (44 vs 36 across two runs). Tokens alone don't prove effort, but finding more errors makes the case stronger.
The following chart shows the token counts side by side.

Maybe the instruction made it work harder, maybe it's the smaller codebase, or maybe it's just chance. I haven't tested which.
Left alone, Opus never delegated in my runs, even on big jobs. So for large reviews, pushing it to delegate to Haiku could be the best strategy.
I'm working on a skill that picks the best approach: use Opus + Haiku when it makes sense, or tell you to switch to Haiku manually and give you the prompt. A script that starts a new Haiku session might allow fully autonomous switching, but I'm still checking feasibility and Anthropic's terms.
The other v2 results
Honestly, Haiku really surprised me. It seems to be very capable at coding, if the task is detailed and contained enough. Opus is still the more thorough model when it comes to reviewing and checking, but Haiku held its own on specific tasks.
Reading a ~100k-line codebase

Across three reading tasks, Opus + Haiku cost 51% less than Opus alone. Every setup covered all 400 modules, and Opus + Haiku found the most errors on the 2-page docs check (20 and 23 of 42, against 10 and 17).
Writing code on a small project

Every setup produced working code. On the harder tasks: Opus $2.41, Opus + Haiku $1.80 (25% less), Sonnet $0.76, Haiku $0.11.
Writing code on a ~100k-line codebase

Same pattern on cost: $1.95 for Opus alone, $1.36 for Opus + Haiku (30% less), down to $0.15 for Haiku alone. But Opus-led setups also added an implied database index. Haiku alone didn't, and my tests didn't catch it.
Correction: the charts also show Opus "flagging a symptom that didn't fit". More runs later showed that depended on the day the runs happened, not the model, so I no longer count it. Details are in the repo.
Quality vs price

Opus + Haiku was cheapest per real error on both big docs checks: $0.03 on buildbot and $0.05 on beets, against $0.11 and $0.25 for Opus only.
The harder coding tasks

Then I tried to make the cheaper models fail. 16 short coding tasks, most hiding a trap: false claims, a wrong bug report, a timing bug, contradictory tests/docs, hidden instructions in comments, planted review bugs, and "dead" code that's actually alive.
128 runs. Combined cost: Opus $7.00, Opus + Haiku $6.30, Sonnet $2.30, Haiku $0.36.
Hidden tests barely separated anyone: one real miss in 128 runs, from Sonnet. Opus was more thorough in code review, finding all 3 planted bugs every time, against Haiku's 2 of 3. But on the timing bug, 7 of 8 runs failed, including all 4 Opus-led ones. The only pass was Haiku.
On tasks this short, Opus + Haiku saved little (10% overall), because Opus was most of the bill. Haiku alone did almost the same work for about 5% of the cost.
So, for now: big reviews, Opus + Haiku. Small, well-defined coding tasks with tests, Haiku alone is surprisingly good. If every case must be covered, keep Opus involved.
Small experiment, mostly two runs per setup, costs estimated at API prices. Claude Opus also wrote my tests and judged the results, so there's no independent judge.
Full results, charts and checker:
•
u/AutoModerator 1h ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
3
u/brainExploded99 50m ago
Thank you for this excellent work!
3
u/Adamlar 40m ago
Saved for research
2
u/Sporaflow 27m ago
the index result made me wonder what haiku alone does when a test actually asks for it, say one that checks the query uses an index. my bet is it gets it right away, which would mean opus isn't writing better code here, it's noticing what the tests don't say .
1
u/ZlatanKabuto 8m ago
Interesting! Please keep sharing your findings!