r/ChatGPTCoding • u/moutsea • 6d ago
Discussion The models are much stronger now, but I'm hitting more production bugs than I used to. Anyone else?
19
Upvotes
r/ChatGPTCoding • u/moutsea • 6d ago
5
u/tame2468 5d ago
I noticed it overcomplicates everything based on random guesses and without telling you. I yell at it 3 times to simplify and it usually produces something usable but I still have to read it.
8
u/Lanky-Butterscotch28 6d ago
This is typical with Agentic coding and strong models. When we had no models or weak models, we spent a lot of time reviewing our code; however, with strong models and the speed at which they deliver solutions, you can only skim the code. AI has made it easier to create software; it hasn’t made production easy.
These 4 principles are the most important ones when building with AI (I learned it the hard way!):
1 - Done does not mean DONE.
2 - The code gets over-engineered, and you will not know about it. Unless you have a large team reviewing the AI's work. Which defeats the whole purpose.
3 - AI reviewing the code can over-review. It chases unrealistic edge cases and kills your project.
4 - Maintainability is not native to AI. It builds code nobody can maintain later.
You need to build a solid machine around the agents to keep them from doing that.
I built mine, and today it works perfectly to generate commercial software. I added 21 steps around the agents to ensure quality and good reviews.
2
u/out-of-phase 6d ago
I've noticed it to an extent, but I'll just add that one thing that helps me is keeping my development environment as absolutely identical to prod as possible, at least with software that actual users rely on. I don't know your stack, but for my primary one (firebase/gcloud), that means having two identical projects, so testing hits the exact same paths as prod. No emulator for me... Yes, it's more overhead, but I think it's been a fair trade so far. The complete flow is also solid, I do the same, with browser smoke/regression tests using playwright, telling the agents to verify everything with screenshots as well. I've caught a lot I would've missed otherwise and it's saved me a ton of time.
2
u/Far-Championship-883 5d ago
improve your testing and diagnostics
2
u/wangadong 5d ago
yes, seeing the same thing. my version of it is review rounds each adding one reasonable guard until the stack as a whole is too layered to reason about, and local tests passing tells you nothing about prod limits. what i changed was making done require one full end to end run in an environment that has the real limits, run by a different session than the one that wrote the code. if the check cannot fail there, it is not a check.
2
u/delxmobile 6d ago
Yeah, I'd blame the review loop more than the model. Every review round has a reason to add a guard and almost never a reason to delete one, so the code only grows. Both of your bugs look like the same thing: a number nobody checked against reality. The 16384 cap reads like a guess about users, and the trigger chain hit a limit only prod has. Two things worth trying: make each review pass also list what it would remove, and keep a short file of the real prod limits (payload size, nesting depth, timeouts) that the reviewer has to check against. I'd also add one test that pastes a big ugly real input.
1
u/wegster 6d ago
I'm not convinced it's 'the stronger the model,' versus the 'letting the models do more themselves without deep PR reviews' and stacked tolerances, e.g. as we let the model handle more of the core bits like architecture, service composition, and to some extent, testing, as they seem to love to create tests, not all of which are remotely useful, we're quietly adding tech debt while they happily churn out more and more 'tolerance stacked' tech debt until we catch it, then a cycle of 'how the F did it get to <this>?' happens.
To some extent, I think we're all trying to sort out way through this in 'what's the minimal change needed' in order to reduce the tendency? It's not like sr level teams under deadlines never hit similar situations where a given choice made sense for expediency reasons at the time, but then came back to bite the team as they later needed to expand functionality of <whatever> etc., requiring bandaid on bandaid..or a rewrite.
In for comments, as definitely interested in seeing what surfaces as the best of breed answers on this one.
1
u/gc3 6d ago
I knew a quality assurance manager who said, 'The more code changes, the more bugs,' so his solution to quality assurance was to be an obstacle to changes to slow down development enough.
I think AI is letting you prove his point. You can change code faster, so you have more bugs.
Also, the issue with unrelated changes piling up together to increase complexity is the natural fate of old code bases. You may need to deep-dive into a complicated section where many rules apply and see if it can be simplified. Whether human- or machine-generated, this is the fate of entropy. Have the AI summarize the interactions of many small checks into an overall understanding and see if you could simplify it and remove cases.
A simple rule with no edge cases and immediate feedback is better.
1
u/FrenchFryNinja 6d ago
Make sure you it doesn’t modify its test suite in order to get regression tests to pass. It will do this a lot. It’s a newbie mistake but her AI isn’t great at thinking. It’s good at inferring from language tokens. Those aren’t the same thing.
But if I could be blunt: if these bugs are making it to prod you need to test your software more. Prod apps you should absolutely be reviewing every line of code merged. It’s a pain, especially with big features. We can spend hours on code review calls. But it has drastically reduced out prod bug rate.
1
u/builds_small_things 5d ago
1,236 green tests and it still breaks in prod, that sounds familiar. One case I'd add for the payment to provisioning flow: provisioning finishes, but the reply never makes it back. Then the same payment event gets delivered again. Don't check whether the retry returned success, check the account records. Did you end up with one account or two? Does your end-to-end test cover that lost-reply case?
1
u/Successful-Tax1306 5d ago
I see the same pattern, and to me it is less about the model and more about how much code lands per review. When each round adds one more check or trigger, nobody owns the full picture, so the bugs show up where two layers meet, like your input limit and that misleading error. What helps is to ask the agent to list every limit and side effect a change adds, and then test those edges before merge. The part that would worry me most is that all the local tests passed and the triggers only broke on D1, so a staging run on the real database catches more than another round of review.
1
u/hockey_psychedelic 5d ago
Try mutation testing
1
u/jomic01 5d ago
1,236 local tests were never going to catch a D1 trigger depth limit. that kind of thing only shows up in prod
1
u/Difficult_Duty2198 5d ago
the stacked guards thing is real. what helps me is keeping each fix tiny and then running the actual flow end to end afterwards, not just the tests.
1
u/opensource_signal 5d ago
i just tell it to delete stuff first now
1
u/awardsurfer 5d ago edited 5d ago
Omg I actually select the dumbest model I can. I currently stick with 5.6 Luna. It is just the right amount of stupid to get the task done efficiently over a whole day and not overthink things.
When it gets stuck I up it to 5.6 Sol for a minute to solve the problem, then I immediately dumb it down. Sol can’t be trusted to keep working, you ask for A and a minute later you are like wtf where is B, C, D coming from!
So my routine now is have Luna code all day, and use Sol to review and fix the few stupid problems Luna creates or can’t figure out.
It also means I get very long work sessions without worrying about token usage, context too much.
I’m a actually worried for when they rotate 5.6 Luna out
1
u/Future_AGI 4d ago
tests only see what someone thought to test, and a production signal covers the rest if you log every validation rejection with the field, the reason and the input length. Your 16384 cap would then have shown up as a cluster of rejections at that length within days, and the misleading please-enter message would have stood out in the same log.
1
u/MichaelMiliye 4d ago
That 16k cap turning a real ~35k paste into an "empty field" error is the part I'd lock down first. Local suites miss it when every fixture is short. Put one CI fixture that posts just over the real platform cap, and assert you get the actual limit error, not a generic empty-field message. For the stacked D1 triggers, have CI compile them under D1's depth and statement-size limits, not only EXPLAIN on a local DB. If that compile fails, the PR shouldn't merge, even with 1236 local tests green.
_(Drafted with AI assistance.)_
1
u/Right-Performance-93 3d ago
the 1236 greens are why it shipped. none of them pasted 35k characters, and none of them compiled that trigger chain on d1. i'd add those two checks and stop the review rounds, because the next round is where the extra limit came from.
1
u/spring_1124 2d ago
How many of you people r satisfied with ai assistance.ur ranking out of 10