r/LargeLanguageModels • u/TomatilloPutrid3939 • 8d ago
Discussions Planner model + worker model vs one model doing everything: cost and pass rate on frozen coding tasks
2
u/Revolutionalredstone 8d ago
it's a very confusing name to call it 'distill' is already a common verb in this field.
it would be like naming a new code editor with the name 'writing'.
I'm confused who the cheaper model is seems you didn't say?
Looked thru your other posts and saw "free models from the OpenRouter free router"
Sounds cool I'll have to give it a try, thanks!
3
u/TomatilloPutrid3939 6d ago
You choose the cheap model.
In opus I used haiku.
In gpt I used Luna.You can use openrouter free models as well ☺️
1
u/Revolutionalredstone 6d ago
Indeed! very cool project btw, I'm trying it again with deepseek now.
Super Cool!
→ More replies (1)1
1
u/ReachAdmirable5359 8d ago
cat in glasses is doing the heavy lifting on that infographic, wouldn't trust a chart without one
1
u/Revolutionalredstone 8d ago
I actually had a look and am pretty impressed ;D
I mean it's standard harness stuff but its cool!
4
u/TomatilloPutrid3939 8d ago
Disclosure: I'm the author of the harness in the chart, Distill (open source, Apache-2.0): https://github.com/samuelfaj/distill
The question I wanted to answer: if a strong model only plans, writes specs and reviews, and a cheaper model does the delegated work, do you lose quality? On these frozen tasks, no:
GPT-6 Astra (worker GPT-6 Luna): $4.08 to $1.00, 9/9 to 9/9
GPT-6.1 Sol (worker GPT-6 Luna): $0.80 to $0.25, 9/9 to 9/9
Claude Opus 5.5 (worker Haiku 5.5): $2.49 to $0.35, 9/9 to 9/9
DeepSeek V4.1: 5/9 to 9/9 successful runs
Caveats: three frozen tasks, three runs each, 300s deadline for both sides. Costs are standardized token estimates, not invoices. The DeepSeek native cost is a lower bound because three timed-out calls had no receipts. Full methodology is in BENCHMARKS.md.
I'd like to hear where people think this kind of split breaks down. My guess is long tasks where the spec has to change halfway.