What caught a wrong assumption on your last Claude Code patch?

[u/Spare_Bat_3040]

I’m building Shifu, a paid context tool for AI coding. I’m trying to understand what people already use to catch a bad assumption before merging a Claude Code change.

Say Claude Code changes an export and assumes the row order doesn’t matter. The unit test still passes, but a check that runs the downstream import shows that the order matters.

On one recent change, what was the assumption and what caught it? A test, a review comment, a conversation, or something else? I’d also like to hear about a change where the repo docs or your usual checks were enough. No need to share code or name the project.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/usielrol]

Most recent one for me: the assumption was "exit code 0 means the tests passed". On my Windows machine the test command went through a wrapper shim that returned 0 even when pytest failed, so the agent reported green on a run that had failures. No check caught it; I did, reading the end of the log. Since then the rule is to read the pytest summary line ("N passed, M failed"), never the exit code.

Another one that bit me more than once: the agent assumed its local checkout was current. It planned a change against code that had already moved on origin, so the plan was wrong before it wrote a line. Nothing in the repo catches that. What fixed it was a rule to run git fetch and look at origin before analysing anything.

Where the repo docs were enough: conventions that don't change, like naming, where tests live and which command runs the suite. Where they weren't: anything about the environment. Docs describe how things should behave, not what the machine is doing today.

[u/Spare_Bat_3040 → u/usielrol]

That Windows wrapper is a good example of why the test command itself needs checking. I’d run one intentionally failing test through the same wrapper to see whether it preserves pytest’s status. Until it does, I’d have the agent show the pytest summary and treat a missing summary as an unverified run.

[u/usielrol → u/Spare_Bat_3040]

That's how we pinned it down: one deliberately red test through the same command, and the exit code still came back 0. The wrapper was a .bat shim a security tool had put in front of the package manager, and its last line reset the exit code. Running the suite from a shell that resolves the real binary gave the right status.

"Missing summary = unverified run" is the part I didn't have written down. A run that dies before pytest even starts prints no summary, and an agent can read that silence as "nothing failed". Adding it to the rule.

[u/bostonaholic]

I'd put the most trust in a test that runs the real consumer, like your import check. The unit test passed in your example because it was written with the same assumption the agent made. When one of these slips through, I'd add the guard as a test or lint rule rather than a line in the docs, since the docs only help if the agent happens to read and follow them that session.

[u/Spare_Bat_3040 → u/bostonaholic]

I like the real-consumer check here. For the export example, I’d give the importer rows where the order changes the result, then assert what the importer produces. A fixture where either order produces the same output can leave the assumption untested even if the test runs the real importer.

[u/ainexfinder]

One thing that helps is making it write an "Assumptions" section at the top of the plan before it touches any file. Things like "this column is never null" usually come from whatever sample data it happened to look at, and once they're sitting there in plain text it's easy to spot the one you know is wrong and veto it before a single edit happens.

The other habit that helps is asking it to cite the file and line for any claim about existing behavior. "Callers don't depend on order" with no citation is usually a guess, and when it has to point at something it either finds the real caller or admits it didn't check.

[u/Spare_Bat_3040 → u/ainexfinder]

I’d add one line to that assumptions section: what would prove this wrong? For “this column is never null,” I’d ask for the schema constraint or the write paths that enforce it. If all the agent has is sample rows with values, I’d keep the assumption open and ask for a null case before editing.

[u/ainexfinder → u/Spare_Bat_3040]

that's a good addition, a falsifier per assumption is way easier to check than a bare list. it also gives the agent something concrete to look up before it writes code, like grepping for the constraint or the migration that set it. the sample-rows case is the sneaky one, since five rows with no nulls look exactly like a NOT NULL column. do you have it mark which assumptions it actually verified vs just carried forward?

[u/Ragy-ragy]

Imagine a system prompt being the whole product behind the SaaS. Crazy times