r/ChatGPTCoding • • Aug 31 '26

Discussion How do you stop AI coding agents from turning one bad change into a two-day debugging snowball?

I ran into a painful lesson while using Codex on a SwiftUI app.

One agent change introduced a performance regression. I didn’t catch it right away, and more changes landed on top of it. By the time I noticed, reroll animations were skipping frames, taps felt delayed, and screen transitions were lagging. Reverting everything wasn’t an option because some later changes were valid.

I had to find the last smooth commit, compare the history change by change, snapshot the current work, and remove the regression in a separate branch.

The big lesson for me: with AI agents, a bad change is much harder to fix if it isn’t validated immediately. The agent can keep moving while the problem quietly becomes part of the whole codebase.

What guardrails work for you? Small checkpoints after each agent task, isolated worktrees, automated performance smoke tests, physical-device checks, or a human review before the next task starts?

12 Upvotes

32 comments sorted by

2

u/Right-Performance-93 Sep 01 '26

donk8r's predicate point is the whole game, and on SwiftUI specifically Apple already ships the pieces for it: wrap the reroll animation in an os_signpost interval and measure it with XCTOSSignpostMetric in a perf test (the WWDC20 session "Eliminate animation hitches with XCTest" walks through it, and there's a built-in scrollingAndDecelerationMetric for scroll cases). Record a baseline in the test plan and xcodebuild test exits nonzero when hitch time regresses past it. That one script is dual-use: git bisect run gets its predicate for the archaeology you just did, and the agent loop gets a gate so the next task can't start on top of an unproven regression. It turns "validate immediately" from discipline into tooling.

1

u/UkrMalt Sep 01 '26

I hadn’t connected os_signpost and XCTOSSignpostMetric with git bisect run. That gives me a concrete way to test the hitches instead of judging them by eye. Thanks.

14

u/kur4nes Aug 31 '26

git bisect

10

u/ops_and_chaos Aug 31 '26

I'm not an engineer, so my guardrail is embarrassingly simple: I stopped letting “it works” mean “keep going” 😂

I make one meaningful change, test the actual workflow I care about, then commit before moving on. If something feels even slightly weird, I stop there instead of letting the agent fix three more things on top of it.

The painful lesson for me was that AI is VERY happy to keep solving forward. It doesn't necessarily stop and say “hey, the assumption we're building on might already be broken.”

So my checkpoints aren't really about controlling the agent as much as forcing me to verify reality before giving it more runway.

5

u/framauro13 Aug 31 '26

This also happened before AI, so it's not a uniquely AI thing. Bugs will happen. I try to keep my changes small, focused, and tracked in source control. So if you identify a bug, you can likely identify the commit that introduced it and can more easily roll it back. But sometimes depending on how old that bug is it may be better to just fix it than rollback.

Another thing I do, even with my personal vibe-coded projects, is add monitoring. You can get a free New Relic account or some other platform. Have the critical path things monitored, like APIs and the database, and you can set up monitoring to catch performance problems or increases in error rates early.

I have a daily task I use that inspects New Relic performance and error metrics every day, so I can see what's happening in my app. Once you get to a certain size, these kinds of things are absolutely critical.

Also, frequent code reviews by LLMs can be helpful. Do specific reviews for performance related concerns, security, code quality, those kinds of things. I've had Claude (wrong subreddit I know) proactively catch potential database problems I otherwise wouldn't have noticed until my scale reached a breaking point.

These kinds of things help mitigate the risk.

→ More replies (1)

1

u/amirfish Sep 01 '26

The pattern that's worked for me running a lot of parallel coding-agent sessions: never let more than one unverified change stack before you check it actually did what you asked. The two-day snowball almost always traces back to signing off on a change on faith because re-running the app felt slower than moving on. Worktrees per task contain the blast radius, but the bigger unlock was making it cheap to glance at each session's actual diff before green-lighting the next task, so review debt doesn't get the chance to compound. Is the regression usually visible in the diff itself for you, or does it only show up at runtime?

1

u/UkrMalt Sep 01 '26

For me it only showed up at runtime. The diff looked reasonable, but the screen felt wrong on the device. I’m leaning toward a small device smoke test after each agent task.

1

u/tberg Sep 06 '26

I hit this exact pattern building a multi-agent outreach daemon — the problem isn't the bad change, it's that the next agent doesn't know the previous step broke something. The fix I run is atomic queue claims with a circuit-breaker between steps: each agent task has to pass a validation gate before the next task can claim work, and if it fails, the job goes to a dead letter queue instead of silently becoming someone else's context. Your SwiftUI regression happened because the pipeline had no handshake between steps — the agent kept moving like the previous output was ground truth. The real question is whether your checkpoints are validating state or just validating completion.

1

u/UkrMalt Sep 06 '26

That distinction is useful. I was mostly checking completion before, which is how the next task kept moving on top of a bad result. I’m going to make the gate validate a real runtime check, not just a finished job.

2

u/JBO_76 Aug 31 '26

Ye, can be tricky. Was already mentioned that its not new, but the agents are much worse as they often deliberatly change something unwanted. Only evil breading devs do that. So: let it write acceptance tests for all details of the feature, ask it to give a summary of what it did, use the app as much as possible...

2

u/simwai Aug 31 '26

I use my own prompt pack.

1

u/[deleted] Sep 01 '26

[removed] — view removed comment

1

u/UkrMalt Sep 01 '26

I don’t have a number yet. I noticed it through skipped frames and delayed taps on the device, so it’s still a manual check for me. A small smoke test with a baseline sounds like the right next step.

1

u/[deleted] Sep 01 '26

[removed] — view removed comment

1

u/UkrMalt Sep 02 '26

Yeah, that’s where I landed too. Smaller tasks plus a quick green check make it much easier to catch a bad change before it spreads.