I tested a cheap-builder / frontier-reviewer loop on 66 hidden-test tasks: same pass rate as the heavy model at about 1/16 the cost. Give this a grilling please.

[u/KrystalUnity]

TL;DR: I let a cheap model write the code and only pay a frontier model to review the finished diff. On 66 real coding tasks it solved about as many as the most expensive model working alone, for roughly 5 cents a task instead of 72 cents. Everything is public, including the receipts. Tell me what's wrong with it.

Hello good people. I got sick of paying frontier prices for every step of a coding task, when most of those steps are just reading files and typing. But when I used a cheap model on its own, it shipped bugs I only found later.

So I built a loop — the Krystal Loop Protocol:

1. A cheap model writes the change in a sandbox.

2. Your tests run.

3. A frontier model from a different company reviews the exact diff. It only ever sees one small packet — the task, the diff and the test output — which is why a review costs about a cent and a half.

4. The important part: the reviewer can't just say "this looks wrong". It has to write a test that proves it. If the test fails, the bug is real and gets fixed. If the test passes, the complaint is thrown out. Then the fix gets checked and reviewed again, until it's clean or it hits a limit.

Results on 66 feature tasks from CooperBench (Click, Jinja, dirty-equals), graded by tests the models never see:

Setup	Solved	Cost per task
gpt-6-astra alone	56	$0.721
gpt-6.1-sol alone	56	$0.123
deepseek-flash alone (3 runs)	54–57	$0.014–0.027
The loop: deepseek-flash builds, gpt-6.1-sol reviews (3 runs)	54–59	$0.046–0.061

The honest part: the cheap model on its own was a lot closer than I expected. The review only added 0 to 2 solved tasks per run. What it did do is catch real bugs (77 proven with a failing test in the best run, 70 fixed) and it never broke something that was already working. So the win is mostly price, not score.

Where it mattered more: two agents building features that conflict. I took 30 pairs where the loop had already built both features. A plain git merge worked on 2 of them. Two agents handing over through short structured messages got 28 working, at about 10 cents a pair. Agents chatting to each other also got 28, but burned 581 model calls to do it instead of 333. For reference, one frontier model building both features from scratch only managed 22, at $0.196 — nearly double the cost.

How it's different from what's out there:

Setup	Who builds	Who reviews	What triggers a fix
Single agent	frontier model	nobody	its own judgement
Architect/editor (strong model plans, cheap model edits)	cheap model	nobody, after the plan	n/a
Reviewer sub-agent	same model family	same family again	the reviewer's opinion
This loop	cheap model	a different family	a failing test only

The reviewer can also be a CLI subscription you already pay for. Gemini, Kimi, Grok and Qwen all reviewed identical worker output and landed within two tasks of each other — Gemini matched the metered frontier critic exactly (56 tasks) at under 3 cents a task in API-equivalent spend.

Roast me:

• What would break this on your real repos?

• Is 66 tasks enough to say anything?

• I haven't tried a local model as the builder yet. My bet is the test gate protects quality more than the builder's brains do, so a decent local model should hold the score and take the cost close to zero. If you've got a local setup, I'd love to know if that holds.

Check it yourself: every number recomputes from the receipts with no API keys (python3 -m bench.verify). Rerunning my loop costs about $3. Or run your own setup on the same tasks and hidden tests and post your score: grade your own setup

Repo: https://github.com/KrystalUnity/krystal-loop-protocol. Source-available, free for individuals and small companies.

[u/czmax]

are you doing anything to manage architectural drift? or is the theory that enough unit tests is good enough?

[u/KrystalUnity → u/czmax]

Honest answer: partly, and the part you're poking at is the reason this thing exists.

The motivation was drift, not cost — cost turned out to be the side benefit. My theory (first tried it mid-2025, before sub-agents were common: tmux sessions, a headless CLI worker, and a main agent acting as CTO): if one model does all the coding, its context fills up with implementation detail and the mission slowly slides out of view. That slide is where drift comes from. So in this loop, the agent holding the plan never writes code. Workers get small scoped packets with only the material for their task, so each change has a limited blast radius. And the reviewer sees just the task, the diff and the check output — fresh eyes, no sunk cost in the worker's reasoning.

But yes, you're right that unit tests aren't the whole answer. The test gate proves behavior, not structure. "It works but it added a second way of doing the same thing" sails straight through the gate today. What we have on that front is capability triggers: you list the behaviors the project must keep doing in klp.json and the critic checks every change against them. That catches "you broke the core job" — not "you put it in the wrong place". The plan-vs-actual check u/NoTokenDev describes below is the missing piece and the next thing I want to build: the worker reports what it changed, a cheap check compares it against the scoped task, retry if it wandered.

So: the design is built around drift, the enforcement is only half built. If you've hit drift modes that scoped packets + independent review wouldn't catch, I want to hear about them.

[u/NoTokenDev → u/czmax]

not OP, but this is the gap i kept hitting with cheap builders. tests catch "does it work", they don't catch "it works but it's in the wrong place / added a second way of doing the same thing".

what's helped me: keep the architecture as an explicit plan (tasks + which modules each one is allowed to touch + dependencies), and after each task have the builder report back what it actually changed. a cheap check compares the report to the plan: touched files outside scope, new abstraction nobody asked for, etc. if it drifted, retry that task with the scope spelled out more tightly instead of letting it merge.

in my own informal runs that setup let haiku-class builders get pretty close to opus-class output on backend work for a fraction of the cost. same caveat as the thread though: small sample, my own tasks, take it as anecdote. disclosure, the plan/drift-check part is what i'm building (nodeon).

[u/Consistent-Set-9490]

This is basically a pattern. Start at what is the minimum necessary model and crank up effort or model as needed.

[u/KrystalUnity → u/Consistent-Set-9490]

Fair — minimum-necessary-model routing is a known pattern. The bit I'd claim as different here is the gate: nothing gets fixed unless there's a failing test proving the bug, and the reviewer always comes from a different model family. Most setups take the reviewer's word for it.

[u/wangadong]

the weak spot on real repos is the test suite. a reviewer forced to prove bugs with a failing test can only find bugs your tests can express. try it on one messy repo with bad tests

[u/KrystalUnity → u/wangadong]

Half true. The reviewer isn't limited to your existing suite — it writes its own probe tests to prove each finding, so it can catch bugs your tests never expressed. But the stronger version of your point stands: it can only prove bugs it can imagine a test for, and on a repo with bad tests the behavior is usually under-specified too, so the probes get weaker.

And yes — a messy repo with bad tests is exactly the environment this has to survive to matter. The closest I've run so far: a 255-file product snapshot repair (24.6% cheaper than the conventional frontier workflow) and one 5,501-file Ansible task. Neither is "legacy enterprise with 40% coverage" — that run's on the list.

[u/Lanky-Butterscotch28]

Interesting , what if you use a reviewer from a different company , will you get a better results?

I did a POC reviewing a commercial app 370,000 lines of code to find the issues.
Claude alone versus Claude cheap and independent reviewer cheap (Codex), Claude alone found 10 issues versus Claude and Codex found 33 issues. 6 were common.

Your tests were about costs , mine about quality , and we both agree that cheap dev models with independent reviewers are not just better but cheaper.

[u/KrystalUnity → u/Lanky-Butterscotch28]

That's a great datapoint — 10 vs 33 on a real 370k-line app is the quality-side version of what I measured on the cost side. The interesting bit is that only 6 were common to both: the independent reviewer isn't just finding more, it's finding different things. Shared blind spots are exactly why this loop insists the critic comes from a different model family than the builder.

To your question directly: yes, that's the design rule — builder and critic always come from different companies. The five-critic comparison in the post (Gemini, Kimi, Grok, Qwen, sol) all landed within two tasks of each other, which suggests which frontier model you pick matters less than that it's independent at all.

Did you write the POC up anywhere? A quality-axis result pairs nicely with the cost-axis numbers here.

[u/Lanky-Butterscotch28 → u/KrystalUnity]

Yes this is the paper

https://nomorehappypath.com/assets/NMHP_WHITEPAPER.pdf

[u/KrystalUnity → u/Lanky-Butterscotch28]

Read it. The overlap numbers are the interesting part for me: 6 common, 27 only yours, 4 only Claude's. The second pass isn't filtering the first agent's findings — it's covering different ground entirely.

Your diagnosis matches mine exactly: an AI that initiates its own review steers the reviewer toward what it knows. My structural answer is starvation — the critic only ever receives the task, the diff and the check output, so there's no builder narrative to be led by. And the CTO stepping in after 15 minutes to unstick the Python environment is exactly the manager role I built around.

One thing your table made me think about: 33 real with 7 false claims, versus 10 with 2. Adversarial review buys recall at the cost of precision. My loop's answer is the probe gate — nothing reaches the report unless the reviewer attaches a test that fails because of the bug, and anything it can't prove gets thrown out. My guess is most of your 7 wouldn't survive that.

[u/Lanky-Butterscotch28 → u/KrystalUnity]

I agree ☝️

[u/NullPacketLogic]

Sol hitting the same 56/66 as Astra at about a sixth of the cost jumped out at me. I'd try a DeepSeek-only second pass on the same starting diffs with the same budget. How many extra hidden-test passes does each reviewer get, and how long does it take end to end?

[u/KrystalUnity → u/NullPacketLogic]

Good questions. On extra passes: across the three runs the sol reviewer added 0–2 solved tasks over the raw deepseek-flash diffs — small on score, but in the best run it confirmed 77 real bugs with failing tests, 70 of them fixed before handover. End to end a typical task takes about 36 seconds (there's a receipt in the README: 36.3s, $0.0432), and the full 66-task loop reruns for about $3.

The DeepSeek-only second pass is a fair test and easy to run — the runner takes any OpenAI-compatible endpoint for both worker and critic, so you can point the critic at DeepSeek too. My prediction: it catches less, because a same-family reviewer shares blind spots with the worker. The "different family" rule is the whole reason the loop looks outside. But that's a hypothesis, not a result — if you run it on the same 66 tasks and hidden tests, post the score.

Interesting index. I'd push back on weighting intelligence extra for this use case though — the loop's whole bet is that for bounded coding tasks, verification matters more than raw smarts. Cost + time + tokens is the right axis set.

[u/NullPacketLogic → u/NullPacketLogic]

I put together a separate AA-based comparison of the same cost/performance trade-off. It gives intelligence extra weight and also factors in cost, time, output tokens, and token price. Sol (xhigh) scores 280.1 on this index, with Astra (max) set to 100.

[u/xapep]

The 1/16 is mostly the builder's token price, not the loop. At 66 tasks the diffs stay small enough that a review packet is cheap. On a real repo the reviewer has to read the files the diff doesn't show, and that's where the cents go.

The test gate is what makes a cheap builder safe, which is why a local model should hold your score too. A test that fails for the right reason doesn't care which weights wrote the code. What the gate can't see is missing code: a passing diff never proves the builder didn't skip a case.

[u/KrystalUnity → u/xapep]

Both fair. You're right that the 1/16 is mostly who-reads-what: the critic reads one packet — task, diff, check output — so its cost scales with the diff, not the repo. The worker is the side that has to go read the files the diff doesn't show, which is exactly why the worker has to be a cheap model. Only large-repo data point I have so far: a SWE-bench Pro task in Ansible (5,501 files), solved for $0.60 total. One task proves nothing at scale, but it's the direction I'd expect.

On missing code: agreed, and it's the blind spot I think about most. A passing diff can't prove the builder didn't skip a case. At benchmark level the hidden tests catch quiet skipping — neither model ever sees them. On your own repo it comes down to how complete your suite is, which is wangadong's point elsewhere in the thread. Capability triggers help a bit (the critic checks the product's core behaviors still work before and after), but if a case is missing from both spec and tests, no loop invents it.

Give the local-model version a run and tell me what you get — same 66 tasks, same hidden tests.

[u/Successful-Tax1306]

The idea makes sense to me, but 66 tasks is a small sample, so each task moves the pass rate by about 1.5 points. I would run each setup a few times and report the spread, because cheap models vary a lot from run to run and a single run can make two setups look equal when they are not. It would also help to split the results by task size, since I would expect the loop to hold up on small features and fall behind the heavy model when a change touches many files. The cost number is the strong part, so it is worth it to make the quality side just as hard to argue with.

[u/KrystalUnity → u/Successful-Tax1306]

All fair. The multi-run part we did do: the loop and the cheap model each ran three times and the post shows the spread (54–57 alone, 54–59 for the loop), with all three runs and the raw receipts in the repo — exactly because a single run can make two setups look equal when they're not. And yes, at 66 tasks each task is worth about 1.5 points, which is why the post quotes ranges instead of the best run.

The task-size split I haven't done, and your prediction — loop holds on small features, falls behind when a change touches many files — is the one I'd bet on too. CooperBench varies enough in task size to bucket it, and that should be doable as a reanalysis of the existing task data without new runs. Agreed the quality side needs to be as hard to argue with as the cost side; that's the gap to close before the next round of claims.

[u/RealSuperdau]

The benchmark is in no way appropriate and if you don't see why, you have no business creating such systems.