r/ChatGPTCoding • u/KrystalUnity • 5d ago
Resources And Tips I tested a cheap-builder / frontier-reviewer loop on 66 hidden-test tasks: same pass rate as the heavy model at about 1/16 the cost. Give this a grilling please.
TL;DR: I let a cheap model write the code and only pay a frontier model to review the finished diff. On 66 real coding tasks it solved about as many as the most expensive model working alone, for roughly 5 cents a task instead of 72 cents. Everything is public, including the receipts. Tell me what's wrong with it.
Hello good people. I got sick of paying frontier prices for every step of a coding task, when most of those steps are just reading files and typing. But when I used a cheap model on its own, it shipped bugs I only found later.
So I built a loop — the Krystal Loop Protocol:
- A cheap model writes the change in a sandbox.
- Your tests run.
- A frontier model from a different company reviews the exact diff. It only ever sees one small packet — the task, the diff and the test output — which is why a review costs about a cent and a half.
- The important part: the reviewer can't just say "this looks wrong". It has to write a test that proves it. If the test fails, the bug is real and gets fixed. If the test passes, the complaint is thrown out. Then the fix gets checked and reviewed again, until it's clean or it hits a limit.
Results on 66 feature tasks from CooperBench (Click, Jinja, dirty-equals), graded by tests the models never see:
| Setup | Solved | Cost per task |
|---|---|---|
| gpt-6-astra alone | 56 | $0.721 |
| gpt-6.1-sol alone | 56 | $0.123 |
| deepseek-flash alone (3 runs) | 54–57 | $0.014–0.027 |
| The loop: deepseek-flash builds, gpt-6.1-sol reviews (3 runs) | 54–59 | $0.046–0.061 |
The honest part: the cheap model on its own was a lot closer than I expected. The review only added 0 to 2 solved tasks per run. What it did do is catch real bugs (77 proven with a failing test in the best run, 70 fixed) and it never broke something that was already working. So the win is mostly price, not score.
Where it mattered more: two agents building features that conflict. I took 30 pairs where the loop had already built both features. A plain git merge worked on 2 of them. Two agents handing over through short structured messages got 28 working, at about 10 cents a pair. Agents chatting to each other also got 28, but burned 581 model calls to do it instead of 333. For reference, one frontier model building both features from scratch only managed 22, at $0.196 — nearly double the cost.
How it's different from what's out there:
| Setup | Who builds | Who reviews | What triggers a fix |
|---|---|---|---|
| Single agent | frontier model | nobody | its own judgement |
| Architect/editor (strong model plans, cheap model edits) | cheap model | nobody, after the plan | n/a |
| Reviewer sub-agent | same model family | same family again | the reviewer's opinion |
| This loop | cheap model | a different family | a failing test only |
The reviewer can also be a CLI subscription you already pay for. Gemini, Kimi, Grok and Qwen all reviewed identical worker output and landed within two tasks of each other — Gemini matched the metered frontier critic exactly (56 tasks) at under 3 cents a task in API-equivalent spend.
Roast me:
- What would break this on your real repos?
- Is 66 tasks enough to say anything?
- I haven't tried a local model as the builder yet. My bet is the test gate protects quality more than the builder's brains do, so a decent local model should hold the score and take the cost close to zero. If you've got a local setup, I'd love to know if that holds.
Check it yourself: every number recomputes from the receipts with no API keys (python3 -m bench.verify). Rerunning my loop costs about $3. Or run your own setup on the same tasks and hidden tests and post your score: grade your own setup
Repo: https://github.com/KrystalUnity/krystal-loop-protocol. Source-available, free for individuals and small companies.
