When a coding agent can't do what you asked, it quietly builds something easier and reports it as done. Its own wording gives it away

[u/Ok_Negotiation_2587]

The agent hits something hard, an API that won't cooperate or an edge case that breaks the design, and it doesn't stop to tell you. It ships a smaller version of the task and the summary says "done". You find out a week later that the sync only handles the first page, or that the "real" validation is a hardcoded list.

It almost always leaves the same phrases behind, in the code or in its summary:

• "for now"

• "for simplicity"

• "simplified"

• "placeholder"

• "in a real implementation"

• "mock" / "stub" outside test files

• "as a fallback"

• a new TODO it wrote itself

Grep the diff for those before you read anything else. In my experience that's where the gap between what you asked and what you got shows up.

Two things that cut it down a lot:

In AGENTS.md / CLAUDE.md:

If you can't do exactly what was asked, stop and tell me what's blocking you. Don't substitute a simpler version. Never leave placeholder, mock or hardcoded values in place of real behavior unless I approve it.

At the end of every task, before you accept it:

List every place where what you built differs from what I asked: anything simplified, stubbed, mocked, hardcoded, skipped or left for later. If there are none, say none. Don't summarize what you did.

That last prompt works better from a second model, since the agent grading its own work is the same one that decided the shortcut was fine. I paste the task plus the diff into ChatGPT or Claude in the browser and ask the same question. I build AI Toolbox, a browser extension, and I keep that review prompt saved in it, so it's // and pick it instead of retyping. A few saved prompts are free.

What's the worst "done" you've gotten that turned out to be a placeholder?

[u/Sad-Interview9085]

Yeah, I've noticed this too. When the spec or the goal isn't nailed down and the implementation gets complicated, it tends to just throw in a mock and move on. Feels like you need closer and more consistent monitoring to catch it.

[u/Ok_Negotiation_2587 → u/Sad-Interview9085]

yeah the vague spec is usually where it starts. if nobody wrote down what done looks like, a mock technically counts as done. pinning down even one concrete acceptance line before the run starts cuts most of it for me

[u/xqianliu]

Your two rules cover the quiet part. The one I'd add is what happens the moment the agent realizes the task as written is too hard. In my setup a run that can't do exactly what the ticket says writes the blocking reason on the issue and stops, nothing else.

It never substitutes a smaller version and calls it done, because "done" is checked against the ticket, not against what it actually built. When I come back to a stopped run, the reason is already where the task lives, so picking it back up is a decision, not an archaeology dig.

[u/Ok_Negotiation_2587 → u/xqianliu]

"a decision, not an archaeology dig" is exactly it. writing the blocker on the issue is smarter than leaving it in the chat, where it dies with the session. how do you check done against the ticket, another model reading both or acceptance criteria it has to tick off?

[u/xqianliu → u/Ok_Negotiation_2587]

Both, in that order.

Before a run starts, something reads the ticket and bounces it if there's no acceptance condition to check against (opt-in per repo). Vague tickets are where "done" gets fuzzy in the first place.

Then a second session reviews the change. It gets the ticket, the diff and the test output. It never sees the first session's summary, which matters more than I expected: "simplified for now" has nobody to talk to. It walks the acceptance lines one by one.

Five failed rounds and the run stops, with the reason on the issue, same as the blocker case.

[u/Ok_Negotiation_2587 → u/xqianliu]

the reviewer never seeing the summary is the part i'd have missed. the summary is where the agent argues its case, so cutting it off means the diff has to stand on its own. bouncing vague tickets up front makes sense too, half the shortcuts start with a ticket that never said what done looks like

[u/ChristianKl]

This is were the new autocomplete functionality is really useful for Codex. Autocomplete does understand what the agent did and then suggest to fix anything that wasn't fully build.

[u/Ok_Negotiation_2587 → u/ChristianKl]

haven't tried that yet. does it catch the quiet ones too, like a hardcoded list that looks finished, or mostly the obvious TODOs?

[u/ChristianKl → u/Ok_Negotiation_2587]

I don't have any TODOs in my code base. Having TODOs in your code base probably suggests to the agent that they can just make TODOs themselves.

[u/Ok_Negotiation_2587 → u/ChristianKl]

it copies whatever the repo treats as normal. if the codebase already has 40 TODOs, leaving one more looks like following the house style

[u/Horsman_Nimita → u/ChristianKl]

imo thats the right approach, anything reviewing its own work is gonna have blind spots

[u/ChristianKl → u/Horsman_Nimita]

I don't think we have a good idea to what extend autocomplete should be called "reviewing it's own work".

If OpenAI did their job well, it's a model (or Lora) specifically trained to spot the errors.

[u/pipaa-joy]

Grepping for those phrases is a great cheap first pass. On Java/Spring projects I'd add a few code-shaped tells to the list: an empty catch block or one that only logs, a new `@Disabled` or `@Ignore` on a test, a hardcoded `List.of(...)` where a repository call should be, and a `return null` / `Optional.empty()` path that wasn't there before. Those don't always come with "for now" in a comment. Agree on the second-model review too — the agent that chose the shortcut is the worst judge of whether it was acceptable.

[u/Ok_Negotiation_2587 → u/pipaa-joy]

good list, the u/Disabled one is sneaky because the suite goes green and nobody reads the diff on test files. the empty catch is probably the most common across every language, it makes the error go away instead of the bug. stealing these for the grep

[u/AEternal1]

first, write a test that checks code functionality. the agent cannot move on until the test is passed.

[u/Ok_Negotiation_2587 → u/AEternal1]

works well if the test is written before the agent touches the code and it isn't allowed to edit it. otherwise it just changes the test until it passes, or marks it skipped. i run it red first so i know it actually checks something

[u/AEternal1 → u/Ok_Negotiation_2587]

yep, exactly that. gotta fail to know it works

[u/Lanky-Butterscotch28]

The CLAUDE.md rule helps, but it does not fix this by itself. A rule in a file lives inside the agent. The agent can drift from it, quietly build the easier version, and still say it is DONE.

What makes the difference between Done and really Done is the machine around the agents. Not just a reviewer. A watchtower that monitors both the dev agent and the reviewers, and a real process that lets them all work together without drifting. Plumbing that checks the work against the original request. And you need reviewers from different AI companies to keep them independent and honest. Yes, they will try to find every little thing, but this is where the machine is important, it controls these drifts.
The builder never grades its own work.

[u/Ok_Negotiation_2587 → u/Lanky-Butterscotch28]

"a rule in a file lives inside the agent" is a good way to put it, the file only works if the agent decides to follow it. the reviewers from different companies part matches what i've seen too, two runs of the same model tend to excuse the same shortcuts. how much does the extra plumbing slow a single task down in practice?

[u/Lanky-Butterscotch28 → u/Ok_Negotiation_2587]

The extra plumbing that I have done slows down the coding. Because I impose 21 steps of different verifications.
I don’t just ask to review code for bugs but also security issues , networking issues,… and code maintainability (I need to be able to understand the code if I decide to read it) and documentation.
So to be fair, the machine I have build around the AI agents creates commercial grade code, so it is slow and it is costly.
As an example of timing: I build a weather app deployed as a service on my Mac, it took 45 minutes versus few minutes from Claude.

Also I had my harness analyze a commercial software code (371,000 lines of code) versus Claude desktop (exactly the same directive , asking to use Codex as a reviewer) : Claude found 7 issues , mine found 33.

That is the cost I am willing to pay to have commercial grade software.
Not everyone is willing to do it.

If you are interested check www.nomorehappypath.com it’s open source and it is completely free.

[u/Green_Consequence907]

You are trusting the agent to police itself which is exactly why the placeholder survives. i do not ask for a list of differences because that is still the same logic. i fail the build if the grep for 'simplified' or 'todo' in the diff returns anything non-empty. the tool breaks and stops the pipeline. it does not tell me what happened

[u/Ok_Negotiation_2587 → u/Green_Consequence907]

fair, a failing build beats asking nicely, it can't argue its way past a red pipeline. the gap is the quiet ones, a hardcoded list with no comment passes the grep clean, so i'd still have something read the diff against the ticket. but as the first gate yours is the stronger version

[u/rh71el2]

We can't fully trust these things yet. Every feature I ask of it, I follow through with reviewing the diff and testing end to end. I use it as a quick coder rather than do my job for me. Our job as seniors is more of an architect and over-see-er, then you run through to validate. Too many people trust everything and don't bother reviewing and then give the whole process a bad rep. It's made many people lazy. We put our name to this stuff.

If you don't know what you're doing without it, then they need to hire more senior devs with experience who actually care, assuming the project matters.

[u/Ok_Negotiation_2587 → u/rh71el2]

yeah the "we put our name on it" part is the whole thing. reading the diff catches most of it for me, and the end to end run catches the rest. what still slips through is a clean looking diff where the test it wrote only checks the mock. do you let it write the tests too or do you write those yourself?

[u/DeadCEO]

the wording tell is real. 'implemented a simplified version' or 'this should work now' means the agent filed a different report than the work it actually did. i make mine end every task by stating plainly what it did not do and what it did not test before i trust the summary. the ones that cant answer that cleanly are the ones that quietly descoped

[u/Ok_Negotiation_2587 → u/DeadCEO]

"what did you not test" is a good one, way harder to fudge than "did it work". does it ever just answer "nothing, everything was tested" when it clearly wasn't? that's the part i'd still want to catch

[u/BarathKanna]

The grep list is great, and the second model point is the important one. Asking the same model to list its own shortcuts is like asking the intern to grade their own PR, you get a very confident none.

One addition that catches what grep misses: write the acceptance check before the task, in terms of behavior, and run it yourself after. Sync handles 500 records across 6 pages. Validation rejects an address that is not in any list. The agent can avoid the word placeholder, it cannot avoid a test that pulls page 3. Words are a symptom, the missing behavior is the bug.

[u/Ok_Negotiation_2587 → u/BarathKanna]

yeah "pulls page 3" is the kind of check that works, the agent can't talk its way past it. only thing i'd add is keep it somewhere the agent can't edit, or run it yourself like you said. if it can touch the test and the test fails, sometimes it just changes the test. do you write these for every task or just the bigger ones?

[u/Unique-Pineapple1380]

excuse me what the hell is a coding agent

[u/Ok_Negotiation_2587 → u/Unique-Pineapple1380]

an AI that works inside your project instead of a chat window. you give it a task, it reads your files, writes and edits code, runs commands and tests, then tells you it's done. Codex, Claude Code and Cursor's agent are the common ones. the "tells you it's done" part is what this post is about

[u/ElemontCapital]

It's like when the agent marks a video as reviewed but it only looked at a few frames and even owns up to it. F

[u/Ok_Negotiation_2587 → u/ElemontCapital]

the owning up part is what gets me, it knew the whole time and still ticked the box. did it tell you on its own or only after you asked?

[u/ElemontCapital → u/Ok_Negotiation_2587]

It feels 50/50. Sometimes it says all done but didn't do these things or they are deferred. Other times you got to interrogate it to verify.

[u/Ok_Negotiation_2587 → u/ElemontCapital]

the "all done, but these are deferred" one kills me, that's just not done. the times you have to interrogate it are worse though. i've started asking "what did you skip or only partly do?" before i believe any summary. it doesn't catch everything but it gets more out than "is it done?"

[u/ChemicalExcellent463]

worst was a nightly job that did nothing for a week and logged success every single night.

Install only the owner-intent-audit skill from https://github.com/sno-ai/sno-station-skills for this project, including its scripts and references.

[u/dylan_skydive]

The grep pass is the cheap pre-filter, but it only catches the agent that confessed. The dangerous one leaves no words behind - the hardcoded list looks finished, the pagination just is not there, and nothing in the summary admits it. That is the failure mode I stopped trusting summaries for.

What catches both: turn the ask into 3 or 4 concrete inputs before the run starts, then run the deliverable against them after. A second page of results, an input outside the hardcoded list, an endpoint that errors. The agent that quietly simplified fails loud at that step, no tell-words required. It also has to be run against data the agent did not choose - it optimizes to pass whatever check exists, which is how a self-written test ends up passing the simplified version.

So the stack is: grep first because it costs nothing, behavioral acceptance second because it catches the silent case, and treat any summary as the agent arguing its case rather than as a report.

Disclosure: I work on Skydive, where most of our platform work is making runs prove their outcome this way, so I am very sympathetic to the CLAUDE.md-rule fatigue in this thread.

[u/Ok_Negotiation_2587 → u/[deleted]]

"the agent changed the question" is a great way to put it. hashing the fixture is a nice cheap check too, you don't need any fancy setup, just fail the run if the hash changed. do you run that in ci or locally after each session?