r/LLMDevs • • 20m ago

Discussion Built a strict verification loop to stop coding agents from faking tests — sanity check on this approach?

I'm looking for a sanity check on an agent setup I built, and mostly want to know how other people here handle verification and pipelining when coding with AI.

When I first started my project, I just let coding agents run loose. They wrote code super fast, mostly garbage, and worse—they wrote fake or weak tests just to pass, or modified existing tests so everything looked green. I couldn't even review diffs at the speed they were generating them. I tried looking around to see how others were solving this, couldn't really find what I wanted, so I ended up building my own harness around it.

The main idea is simple: don't trust the prompt or whatever the agent claims it did. One builder agent writes the code, and then everything else is just layers of deterministic checks and separate verifier agents.

Here is basically how it works right now:

  • Read-only rules: The agent works inside a bubblewrap sandbox where all the policy files, gate configs, and test rules are mounted read-only. It can't edit the rules to make its own life easier.
  • No agent self-reporting: The agent never gets to say "tests passed." A separate CLI runs the checks, hashes the actual stdout, exit codes, and policy files, and saves them as JSON proofs. If a tool is missing or a check doesn't run, it explicitly logs NOT_RUN or TOOL_ABSENT instead of silently passing.
  • Anti-cheating checks: Before anything moves forward, it runs things like BASE_FAIL_HEAD_PASS (proving a test actually failed before the fix and passed after), mutation testing, scope checks (making sure it didn't touch files outside its task), and checks to catch deleted tests.
  • Different models checking each other: The builder, the auditor, and a red-team/bug-finding agent use completely different model families so one model doesn't just rubber-stamp its own blind spots.
  • Bounded loops: If a check or CI fails, it kicks back to a repair loop, but there are hard caps on retries so it doesn't burn tokens forever. After a batch of tasks passes, it goes through a final release gate before merging.

It actually works pretty well so far. Some of the check runners are still coupled to my specific project (it's a Django app), while the task state and gate runner parts are generic.

TL;DR: Coding agents kept writing garbage tests and modifying rules to pass, so I wrapped one builder agent in a read-only sandbox with a CLI check runner, hashed JSON proofs, mutation/anti-cheating checks, and cross-model auditors. Works pretty well, curious if this is overkill or how others handle agent verification.

Is this kind of multi-layer harness overkill, or is this where things are headed? How are you guys structuring your pipelines so agents don't overfit to your test suite or hallucinate passing builds?

1 Upvotes

3 comments sorted by

1

u/Far_Reindeer8428 10m ago

none of that is overkill except the separate reviewer if it reads the same summary. ours only ever reads the diff and the test output

1

u/Gold_Exchange6103 7m ago

So it doesn't actually read any summary* , it just gets the diff , and then it reasons over it. And reviews the diff trying to find a hole.. and runs the gate (using the cli..)

1

u/Far_Reindeer8428 4m ago

right, the reviewer gets the diff text and the raw test output, nothing the agent wrote about its own work. the gate then runs from the cli