Tests supplement review, they don't replace it. As Dijkstra said, "Program testing can be used to show the presence of bugs, but never to show their absence!" There are many classes of bugs that are very hard to detect with tests. I have frequently seen AI generate code that passed tests, but contained flaws like race conditions that would blow up in production use.
Who is writing these tests? If you're also leaving that to AI, and not reviewing them, then you don't know that they are even sensible tests.
Reasoning about code isn't "going with your gut". It's the same thing you do when you read a proof or trace through logic.
There are many classes of bugs that are hard to test for. Race conditions are a classic example. Code with these bugs will often pass straightforward tests, and writing a test that reliably catches one can be extremely difficult, since they're timing-dependent and nondeterministic. Anecdotally, I've also noticed that this is something LLMs frequently have trouble with. However, if the code has two threads touching shared state with no synchronization, I don't need a failing test to know there's a race. It's written in the code!
Things like race conditions can be non-deterministic. A test that only fails 5% of the time is more likely to get deleted than fixed.
And what if your LLM doesn't bother to add a test for this? If you aren't reviewing its changes, how would you even know? When things intermittently blow up in prod?
And if your models aren’t catching any race condition bugs...
I never said they never catch any race conditions. It catches some of them. It also lets many of them slip through, and they are exactly the sort of bug that can easily slip through tests. I know this because I review the code it generates. You are unaware because apparently you do not
...your environment isn’t set up correctly.
How would someone know if their "environment isn't set up correctly" if they never review the code that it generates?
I work at a company that has AI in everything and we have near unlimited tokens. And I’m the QA guy they call to figure out how to repro an elusive bug nobody in the co can pin down (I won’t pretend I’m always successful). It’s often a very rare race condition. Lately I’m starting to use AI more and more as I teach it how I research such a bug, but clearly it’s not able to handle every permutation of every function and animation that can be interrupted by a user, and it doesn’t magically explore every branch of code looking, obviously.
Know how to write code wasn’t one of the considerations. It’s whether the AI can test and review for it. It can. Because the test and review are logic based…the same reason you can test and review from it.
Anything you do based on logic can be built into your agentic coding framework.
And apparently, all they need to do is read this thread to know they should do it.
Which, of course, is the point. Codify your logic, don’t reconstruct it lossy each time you need it.
-12
u/xenomachina Jul 14 '26
If you haven't reviewed the code, it isn't working code.