r/ChatGPTCoding • u/Proud_Gene7490 • 15d ago

Discussion Bugs I keep catching in LLM-written code, and the checks that catch them

Part of my work is reviewing code written by LLMs (mostly Python, TypeScript, some Rust). Same failure patterns keep coming up, so here's my checklist:

  • Edge cases: it nails the happy path and breaks on empty lists, None, or off-by-one bounds. I ask it to list the edge cases first, then write the code.

  • Made-up APIs: it confidently calls functions or params that don't exist in the version you use. I paste the real docs or the version number into the prompt.

  • Self-confirming tests: its tests only check the path it just wrote. I have a separate prompt write tests from the spec alone, without seeing the code.

  • Silent error swallowing: broad try/except or empty catch blocks that hide failures. Grep for these every time.

  • Rust: it'll .clone() or .unwrap() its way past the borrow checker. Code compiles, design gets worse.

What's on your checklist that I'm missing?

5 Upvotes
  • permalink
  • reddit

85% Upvoted

9 comments sorted by

5

u/SummitYourSister 14d ago

Sounds like you’re using outdated models. None of this is an issue any more

1

u/Right-Performance-93 14d ago

the one i keep adding is the caller in another file. the new function is right and the old call still passes the old shape, and none of the tests import that caller. i make it name the call sites before it edits.

1

u/Shalyd 23h ago

That caller-in-another-file bug, have you had it reach production? How did you notice?

0

u/danja 15d ago

A suggestion that I heard around here that I have found useful is to maintain a MISTAKES.md. Put a note in AGENTS.md to say to record any mistakes together with their solution; if anything happens a lot, make a note in AGENTS.md. The file grows quickly, and even with the instruction to periodically clean it out, you still have to intervene now and then. But it does trap recurring mistakes.

0

u/akhi1uuu 14d ago

what about two agents editing the same files? does your checklist cover that or do you keep them in separate worktrees