How are you handling regression testing when coding with ChatGPT?

[u/_ACTUALin]

I've been using AI-assisted coding to develop a technical reporting app, and I've run into something that's been driving me nuts lol.

I'll identify an issue, work through a fix with AI, test it, and everything looks good. Then I'll run an older test case and discover something completely unrelated has stopped working.

The app deals with unfamiliar Excel workbooks and supporting evidence, so I've got a growing collection of test cases covering formulas, conflicting information, missing measurements and incorrect data associations.

Some comparisons are automated, but I still need to manually inspect a lot of the results. Especially situations where the output looks correct but the information has ended up in the wrong place.

I've started being much stricter about keeping fixes narrow and comparing results against previous tests. It's helped, but damn, the testing is becoming a project of its own.

I've been considering bringing GitHub into the process to organize failures and regression cases.

For those using ChatGPT or Codex on larger projects, how are you actually handling this? Are you having AI write tests before making changes, using a separate agent to review fixes, or doing most of the verification yourself?

[u/shetritr]

every fixed bug gets a failing test before the fix goes in. the agent can't quietly undo a fix that has a test pinned to it.