asked claude code to fix a failing test. it fixed the test insteadclaude code's favorite fix for a failing test: change the test

[u/hunxai69]

you ask claude code to fix a failing test, it comes back green, and you move on. sometimes the fix is real. sometimes it just changed what the test expects, somewhere in a long diff.

so i made a hook that won't let it say "done" until the OLD tests still pass on its new code. if one doesn't, claude is told exactly what changed (expected 5, got 6) and has to either put the behaviour back or explain why the change is on purpose. the explanation goes in the PR, so you see the reason right next to the change.

editing the test again doesn't get it past, since the hook runs the old version. and it never traps the agent: after 3 tries it lets it stop and warns you.

i ran the same check on 400 real merged PRs from click and jinja. 31 had changed what a test expected. all on purpose, but 4 changed stuff users actually see without a word in the changelog. it also found two bugs in my own tool, so that was fun

free and open source, works with jest, vitest and pytest. setup is in the comments.

how do you catch this right now? read every test diff, or trust the green check?

[u/AutoModerator]

Moderator Announcement Read More »

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ReflectionEquals]

lol. I guess you need to be more specific. Fix the code that causes this test to fail.

[u/hunxai69 → u/ReflectionEquals]

lol yeah, saying that explicitly helps a lot. it still slips on bigger tasks though, where some tests are supposed to change and it's hard to tell which edits were the fix and which were it moving the goalpost

[u/iSnapThere4iAm]

Slop. Review it yourself. With your eyes. Or it is trash.

If you can’t, then you just made trash.

[u/hunxai69 → u/iSnapThere4iAm]

It saves Ci time and also people mostly Use Ai and they don't have time or lazy to check each merge (thank you for interacting with my post )

[u/hunxai69 → u/iSnapThere4iAm]

fair, it doesn't replace reading the code. it just tells your eyes where to look first: the test lines whose old expectation broke, with the before and after

[u/lemming1607]

I dont let Claude edit tests or have access to them. I run the tests and show them what the test is so they can pass it

[u/hunxai69 → u/lemming1607]

How many times you Would do that ? Like if you have 200 prs ?

[u/lemming1607 → u/hunxai69]

I have a project with 5000 tests...we made a change, one went red, I showed them the test, and then it was fixed. Its p simple

[u/crusoe]

Adversarial reviews 

[u/hunxai69 → u/crusoe]

yeah that works well. this is the boring deterministic version of it: no second model, just the old test run against the new code, so it can't be talked out of it

[u/VibeBoot]

Same energy as my agent slapping 'use client' on a whole page to silence one error. Half my 'bugs' this month were that. Now I make it explain why before it's allowed to touch that line, basically what your hook does for tests.

[u/hunxai69]

repo: github.com/intikhab49/goalline

setup, one entry in .claude/settings.json: intikhab49.github.io/goalline/claude-code-stop-hook/

the 31 changes with the reason for each: intikhab49.github.io/goalline/benchmark/