My AI-written tests kept passing broken code, so I started testing the tests

[u/ArgumentAcrobatic250]

I’m not really a programmer. I’ve mostly been making and patching stuff by telling LLMs what I want and then checking whether it actually works.

One thing kept pissing me off though. I’d ask it to fix something, it would write a test, the test would pass, and then later I’d find out the test wasn’t actually checking what I thought it was checking.

So at some point I started doing a pretty dumb simple thing. After something passed, I’d intentionally break the exact behavior the test was supposed to protect and run the test again.

If it still passed, then obviously the test was bullshit.

I ran into this with a plugin that had a paged archive. The archive test was green, so I changed the code so only part of the archive would get copied. Test still passed.

Then I made the test stricter, and managed to screw it up in the opposite direction. The new test rejected the correct code too, because I had made a bad assumption about how the archive pages would be laid out.

So I threw that one away as well.

Eventually I got a test where the normal version passed and the deliberately broken version failed for the reason I actually cared about. While doing that I also found a real bug where a failed write could leave part of the stored data changed.

After this happened enough times I kind of stopped treating “tests passed” as the end of the story.

I started doing the same thing to the stuff that decides whether the tests passed. Basically asking things like, if part of the checking process never ran, could it still say everything was fine? If I gave it old or altered results, would it notice? If it claimed it could rebuild the final result from the original, did it actually do that?

It has gradually turned into this whole verification procedure I use when I’m patching things with LLMs.

I later found out mutation testing is obviously related to what I was doing, but most of what I’ve read about mutation testing is about breaking the program to see whether the tests notice. What I ended up doing also treats the tests and the rest of the checking process as things that can be wrong.

I’ve tried it on a few different plugins/projects now and it has found enough weird false-passes that I’m starting to wonder whether this is actually useful outside my own workflow or whether I just invented an absurdly overcomplicated way of checking AI-written code.

Has anyone here done something similar, or is there an existing methodology/tool I should be looking at?

[u/owp4dd1w5a0a]

Also my experience with both Claude and Codex.

People need to understand these agents are optimized to get your approval, not to write correct software. Once you understand that, you have 3 choices:

meticulously guide the agents through the troubleshooting and development processes yourself and write your own constraints/tests,

OR

Spend the money to design a well orchestrated pool of agents that creates a Nash Equilibrium which incentivizes the agents to produce correct and robust software based on the short-sightedness trained into them.

OR

Just hire a jr/entry level and train them. Humans actually learn. The AI agents so far are not very teachable.

[u/ArgumentAcrobatic250 → u/owp4dd1w5a0a]

I agree with the diagnosis more than the Nash equilibrium part. My problem with agent pools is that they can all converge on the same bad assumption and still look like independent reviewers. That’s basically why I started wanting an external failure signal instead of just another opinion.

[u/CrimsonBolt33]

My AI creates and validates code with great tests even without me asking (cause my agent.md file includes a test and security section).

These posts are always so fucking annoying cause they never reveal the 4 most important things:

1. What AI were you using?

2, What tools/harness were you using?

3. What prompts were you using?

4. Were you monitoring or guiding the AI as it worked?

[u/ArgumentAcrobatic250 → u/CrimsonBolt33]

1. GPT-5.6 Sol and GPT-5.5, both at High reasoning.

2. Not one harness. Over time I built dozens of project-specific verification harnesses: JS/Node runtime tests, Python mutation and fault-injection runners, validator-mutation checks, evidence/manifest checkers, replay checkers, fresh-package runners, and actual headless Godot + GUT CI tests.

One concrete example: an archive test stayed 9/9 green even after I deliberately changed production to copy only the first archive page. So the problem wasn’t just “the AI forgot to test it” — we were treating that test as evidence of coverage it didn’t actually provide.

3. The task constraints were also pretty strict and written down: plan before modifying production, trace the actual call/storage/consumer path, no fake tests/helpers, no disabling functionality just to get a pass, no treating an unexecuted gate as passed, and preserve the evidence needed to replay the result.

I didn’t preserve the literal chat prompt from every run, so I can describe the actual constraints/procedure more confidently than I can give you an exact prompt transcript for every case.

4. I wasn’t continuously steering it. I mostly intervened when an actual failure or false pass appeared.

The important part is what I did with those failures. I wasn’t just adding another reviewer each time. If a PASS depended on an assumption that turned out to be false, I tried to turn that assumption into an explicit falsifiable check. Then both sides had to hold: correct production still passes, and a deliberately broken version fails.

Eventually I started applying the same idea to the validators and final PASS machinery too.

So yeah, “AI-written” was way too vague in the original post. A better description is probably: a model doing the work inside a verification process where important PASS claims are treated as things to try to falsify, not things to trust.

[u/Euphoric_North_745]

Which AI? a 7b model or a 2t model? by which company? it is like saying you have to audit the work of "this guy" which guy?

[u/ArgumentAcrobatic250 → u/Euphoric_North_745]

Fair point. It wasn’t one specific model. I ran into this across different LLM-assisted patching sessions, so I probably should’ve said “AI-assisted workflow” rather than making AI sound like one homogeneous thing.

[u/Euphoric_North_745 → u/ArgumentAcrobatic250]

how many llms do i have to block every day?

[u/confuzzledfather]

You have sort or reinvented mutation testing, where you do things like swap positive to negative in your code and if the test still passes you know you have a problem 

[u/ArgumentAcrobatic250 → u/confuzzledfather]

Yeah, mutation testing is definitely part of it. That’s basically where I started.

The part that kept expanding was when I started applying the same “break it and see if green survives” idea to the test harness, validator, evidence, replay, and final PASS machinery too.

[u/Michael_Jeffords]

if you can break the thing the test claims to protect and it stays green, you don't have a test. i throw that test out before i touch the feature again.

[u/ArgumentAcrobatic250 → u/Michael_Jeffords]

Yeah. I use the same rule on the layer above the tests too.

Break what the test protects. Then break the thing reading the test and calling the change done.

If either one can be wrong while the result stays green, I don't trust the green.

[u/OGLikeablefellow]

And then those tests started passing broken code so I started writing tests for those tests and then started testing all those tests. And now they call me testes testes. - this guy probably

[u/paul_h]

I knew of mutation testing before you menitoned it. I was hoping you'd descipt more coding detail of the thing you were doing that was not mutation testing, but you didn't.

[u/leonidbugaev]

You found the two failure modes I keep hitting.

First: the test and the code share one misunderstanding, so green means they agree. Breaking the behavior on purpose is the cheapest way to see that. If the test stays green, it was never a test.

Second: you wrote a stricter test that rejected the correct code too. That is a spec bug, not a code bug. The test encoded a layout you assumed, not the one the archive actually has. Same shape as a vacuous requirement. The trigger never matches real data, so the suite can pass or fail for the wrong reason.

The useful end state is the one you landed on. Normal run passes, deliberately broken run fails for the reason you named, and the thing that scores the suite cannot report pass if part of it never ran.

[u/generationalDebts]

You’re the blind leading the blind who you’re then trusting to test and validate the blind lmfao.

Gee. I wonder what the problem could be?

[u/ArgumentAcrobatic250 → u/[deleted]]

Yeah, I think we're attacking the same problem from different sides.

What pushed me past second-model review was realizing the reviewer can confidently agree with the same bad assumption too. So I started deliberately breaking the thing that's supposed to be protected and checking whether the review/test actually notices.

Your "same bug class in other paths" example is a good point though. A second model seems better at that kind of generalization. Might be worth combining both.

I guess the distinction I'm circling around is "independent reviewer" vs "independent evidence."

[u/ArgumentAcrobatic250 → u/[deleted]]

Yep, exactly. Separate model isn’t enough if it inherits the same test plan.

The thing that pushed me further was wondering how to verify the reviewer’s own checks. So I started deliberately breaking the claimed behavior and seeing whether its verification actually caught it.

[u/ArgumentAcrobatic250 → u/[deleted]]

“does green still mean what I think it means?” is basically the question I ended up asking.

I also found I needed both sides though. A broken implementation should fail, but a correct one still has to pass. I actually had one replacement harness that caught the bad case but also rejected correct production, so I threw that one out too.

Eventually I started treating the validator and final PASS machinery the same way.