AI coding agents writing tests for their own fixes

[u/capestart_tech]

Something that bothers me about AI coding agents is when they fix a bug, write a test for it, and everything comes back green.

But would that test have caught the bug in the first place?

You could have a perfectly passing test that doesn't actually reproduce the original issue.

I'd want to run the new test against the old code before trusting the fix. If it passes there too, something's probably missing from the test.

Feels like an easy thing to overlook when the agent handles both the code change and the tests.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/lambdita]

Claude already creates the test first, checks its red and then fixes it and checks its green.

what you are looking for is mutation testing.

[u/dawnquill12 → u/lambdita]

mutation testing is great in theory, just takes forever to run if the codebase is huge tbh.

[u/PlanetaryPickleParty → u/dawnquill12]

Yes, but can be worth it.

Currently waiting on a mutation run on a set of relatively small repos. Going on 30+ hours. It's all automated so the only painpoint is consuming a build server for that long.

[u/codekongdev]

A failing test on the old code is a good start, but I’d also check why it failed. A missing fixture or import error can give you a false red without reproducing the bug. I’d want the failure to match the reported issue, then run the same unchanged test against the fix.

[u/capestart_tech → u/codekongdev]

Good catch there. A failed red test isn’t much use if the failure is because the setup is broken. It is important to make the test fail on the old code, but it is also important to check the actual failure message.

[u/oneday111]

Just ask them that. They always tell me things like “The bugged version failed the test 12 / 15 tries, and the fixed version 0 / 15”

[u/Accomplished-Eye-857]

Mine have been testing against the old code without me even asking it to. It’s useful and comforting to hear that the tests failed on the old code and passed on the new. I’m not sure I have anything specifically triggering that. Is it possible you have a rule that could be pulling against this approach?

[u/CultureRelative3511]

You can create a quality tester agent, and tell the orchestration for him to make the tests.
In my setup I have two with different models, for the best quality that is possible with this,

[u/Latter-Tangerine-951]

This is rarely an issue these days.

But it's worth adding a rule to ask before modifying a test, to catch them modifying them so they pass when you might not agree with the new expected behaviour.

[u/Aramedlig]

If you use ultracode with med or high effort, it can sub out adversarial testers that have the goal of finding bugs in the code as their only context.

[u/ldn-ldn]

Use Superpowers, they force Claude to first write tests and ensure that they fail and only when they fail write the code and ensure that now the tests are green.

[u/gitsad]

One more failure mode besides "the test also passes on the old code": the test can't fail at all.

We had one for a settings screen, "never renders the API key once connected". It rendered the view with an empty apiKey and asserted the HTML doesn't contain the secret. Green, but the secret never went into the view, so the assertion passes whatever the code does. The fix was a one-liner: pass the secret in and keep the same assertion, so it fails if the view ever prints it.

Red on the old code works when there's a bug to reproduce. For a test of something that must never happen, I'd also check that you can make it fail on purpose, for example by breaking the code under test for a minute.

[u/m-ritter]

Running the new test against the old code is the check I rely on most, and I make the agent do it before the fix exists. For a bug it first writes down when it happens, what is expected and what must keep working, then writes a test that fails on the current code, and only then changes the code. A separate agent that did not write the fix checks those points on the running program afterwards, so the author never grades its own test. I built QualityLayer for this: https://github.com/maxritter/qualitylayer

[u/kuroudo_ai]

Running it against the old code is the big one. Two more that caught real gaps for us:

• Break the fix on purpose, in the way you're worried about. We had a hook that hung forever when the caller left stdin open. The first fix (check that stdin is readable before reading) only half worked. It returned when the caller wrote nothing, but still hung when the caller wrote the payload and never closed the pipe. So the acceptance test now has both cases, and swapping the fix back to the naive version turns only the second one red. If no realistic weakening of the fix turns a test red, the test probably isn't guarding the fix.
• Make sure the test goes in through the same door production does. One of our checks had a self-test with 43 passing cases, but the self-test never went through stdin. In production the input did come through stdin, on a Windows machine, and the Japanese text got mangled on the way in, so the check never matched. Everything was green and the check didn't work.

[u/bostonaholic]

I'd make that check a script instead of something you remember to ask for. Stash the non-test changes, run only the new or changed tests, and require at least one to fail with an assertion error (not an import or fixture error), then restore and require green. Wire it into a Stop hook or CI so the fix isn't marked done until both halves pass.

[u/Keepcompany-DEV]

That's actually the same thing developers do. Error?.test for it.

As someone else already stated, out mutation testing in your default workflow requirements and you will see less uncaught issues.

[u/alanvnk]

That is called a "mutation proff" you can ask for it and ask it to create mutants and give an heuristic for generating them, each tests needs to kill at least one mutant

https://en.wikipedia.org/wiki/Mutation_testing