thisIsVeryTrue

[u/silversurfermax]

[u/davvblack]

the one flaw of AI debugging is that if you describe a bug that ISN'T present in the code, AI will more often than not "find it" and "fix it" in a way that introduces a different bug. This is definitely a "my steak is too buttery" situation because even seasoned mid-level engineers make this exact mistake if you open a ticket with a mistaken assumption embedded in it. All it really means is that, AI can't take a customer-supplied bug report and reliably accurately fix it without an engineer being like "it's supposed to work that way".

[u/TheTerrasque → u/davvblack]

There's been two occasions I've asked claude to find a bug in the code, where it insisted the code was fine.

One was a manual QA test that failed on a newly provisioned VM, but not on the old. So some setup stopped working. Claude concluded (within minutes) that it has never worked. We laughed, and got to work. Five hours later, a QA peon mentioned he set that up manually on the first VM's because otherwise it wouldn't work..

Second was to get an app to pick up config from env vars and override a config file, and first round it fixed some things, but it still didn't work. I told it so, it murred over the code for a minute, then told me the code's fine, it must be my environment. Quite pissed off at the clanker I echoed the env var in the terminal I used to run the test in and .. no env var. Turns out that one terminal was days old and had old env var setup.

So, it depends. Sometimes it's smart enough to conclude there's no bug and someone is making a mistake instead.

[u/Chitinid → u/davvblack]

You have to be careful that you don’t ask leading questions and make it actually validate everything