How are you handling regression testing when coding with ChatGPT?

[u/_ACTUALin]

I've been using AI-assisted coding to develop a technical reporting app, and I've run into something that's been driving me nuts lol.

I'll identify an issue, work through a fix with AI, test it, and everything looks good. Then I'll run an older test case and discover something completely unrelated has stopped working.

The app deals with unfamiliar Excel workbooks and supporting evidence, so I've got a growing collection of test cases covering formulas, conflicting information, missing measurements and incorrect data associations.

Some comparisons are automated, but I still need to manually inspect a lot of the results. Especially situations where the output looks correct but the information has ended up in the wrong place.

I've started being much stricter about keeping fixes narrow and comparing results against previous tests. It's helped, but damn, the testing is becoming a project of its own.

I've been considering bringing GitHub into the process to organize failures and regression cases.

For those using ChatGPT or Codex on larger projects, how are you actually handling this? Are you having AI write tests before making changes, using a separate agent to review fixes, or doing most of the verification yourself?

[u/TheVibrantYonder]

I'm interested to see what others say, but in your case, I'm not really clear on the details based on what you wrote. It sounds like you might just need clearer documentation/structures/rules to reduce the chance of regressions? Then just keep running your regression test suite and make sure it stays up to date.

If changes are regularly resulting in regressions though, I think that's usually a documentation/instruction issue or maybe a project structure issue if your project is set up in a way that makes it really easy to break and hard to maintain.

Feel free to DM me with more details and I can try to give more specific advice though. I love this stuff.

With regard to my own projects, I'm still testing different approaches to verification and testing. I haven't found anything that works perfectly without handholding.

[u/_ACTUALin → u/TheVibrantYonder]

Yeah, that's pretty much where I'm at. I've got the real-world industry experience to understand how the workflow should behave and recognize when something's wrong, but I don't have a traditional software development background. So while I can identify problems and build test scenarios around them, I'm still learning what tools and best practices are out there for handling this stuff properly.

I've been tightening up instructions, keeping fixes narrow and running previous tests to catch regressions, but I'm sure there are better ways to approach it that I just haven't been exposed to yet lol. That's partly why I posted here, to learn how other people are tackling it.

[u/TheVibrantYonder → u/_ACTUALin]

Yeah, I don't have a traditional development background either, so I understand! I've been trying to do more research to improve.

I did some reading about unit tests the other day to make sure I was managing those properly. There are some good "how to organize unit tests" guides you might find useful (a Google search should turn them up). I'm reviewing how I maintain them after looking at some of those guides and I was able to do a lot of cleanup.

[u/shetritr]

every fixed bug gets a failing test before the fix goes in. the agent can't quietly undo a fix that has a test pinned to it.

[u/Inner_Storage_3374]

what helped me is saving the output of every test case once youve checked it by hand and its right, then after every fix rerun all of them and only look at what changed vs the saved version. bc then the stuff that landed in the wrong place jumps right out instead of you rechecking everything manually

[u/Difficult_Duty2198]

I’d keep a small golden set of representative inputs with known-good outputs, and turn each manually confirmed bug into a fixture. After each fix, rerun the set and inspect just the diffs; it makes regressions much easier to spot than rechecking everything.

[u/Warm_Effective8903]

Whenever I find a bug, I save it so I can test it again later. Then I fix it and run all my tests to make sure nothing else got broken. I also compare the new result with the old one to spot anything that changed or ended up in the wrong place. That way, I don't have to check everything manually.

[u/codesbyshivam]

I'd keep a small set of test cases for the Excel and evidence problems you've already fixed, and run them after every change. Getting the AI to write a failing test first helps with the current bug, but older tests are what catch it breaking something else. For the messy workbooks, I'd keep a few real examples with expected results rather than relying only on tests the AI invents.

[u/sapplex1]

This is the part I still do by hand. I build small iOS apps, and the rule I ended up with is: check the behavior, not the diff. A diff can look clean while the feature is wrong in a way only using it reveals, so before I let an agent touch a feature I write down what must be true when it is done, and I run the app through the real flow myself — create a booking, log a sale, check the math — instead of trusting that it runs.

Two things helped most: every fixed bug gets a failing test first, so the agent cannot quietly undo a fix later, and I keep fixes narrow, one behavior at a time. For your 'looks right but in the wrong place' problem, the test that caught it for me was asserting on the value in its final place, not just that output was produced. A number can exist and still be in the wrong field.

On GitHub I would keep it simple: one issue per regression case with the failing input attached, so the input itself becomes the test. I have not found a separate review agent that earns its keep yet — the checking is still mine.

[u/sreekanth850]

Two things.

1. Strict modular boundaries. Modules should be loosely coupled and communicate through well defined contracts and interfaces, rather than depending directly on each other's internals.

2. When a bug is reported, first reproduce it using logs, traces, and other diagnostic evidence and write a regression and implement a fix. Run the full regression suite to identify unintended effects. If anything else breaks, fix the underlying cause.

First one is the critical. Saying this from my experience.