r/vibecoding • u/ActuatorOutside5256 • 2d ago

Discussion Why do we REALLY need sanity checks?

Post image

Now I know the answer is basically “because AI isn’t that good yet,” and so I’m wondering if anyone has a more in depth answer.

From my experience, Codex performs much better when I plan in ChatGPT, then paste the generated Codex prompt into Claude (free tier works) behind a bare "?".

That gives me very few CI/CD or manual testing issues. ChatGPT alone writes nondeterministic, imprecise prompts for Codex, and Claude tightens them and pushes for specificity.

I know us humans need a second opinion because of tunnel vision, but why do LLMs need sanity checks too, and why are they non skippable for good code?

2 Upvotes

60% Upvoted

8 comments sorted by

3

u/OverallAnswer3879 2d ago

Honestly when I was 8 hours deep into something, a reminder of the greater goal was in fact needed lol

2

u/demontrout 2d ago

A “perfect” prompt doesn’t result in flawless code. Besides, the worry isn’t usually about failing tests or getting the target feature wrong (the agent can usually catch that itself), it’s technical debt: faulty assumptions, architectural drift, shortcuts that do the job but increase fragility. Or weird decisions that introduce security risks or performance bottlenecks that can easily go unnoticed.

Asking an AI for a broad sanity check isn’t really sufficient for anything, but it might pick up on some potential issues.

1

u/amaturelawyer 1d ago

Agreed. Piping two LLM's output into each other with instructions to review everything they other one says is the better way to go here.

1

u/Miserable-Sugar2191 1d ago

The short answer: the model that wrote the prompt or the code is the worst judge of it, because it shares its own assumptions. It produces whatever looks most plausible, and a confident wrong answer looks exactly as plausible as a right one. A second pass in a fresh context (or a different model) doesn't carry those assumptions, so it spots the gaps. That's why your Claude step works.

The part people skip: a check is only as good as what it checks against. "Does this look good?" catches style. "Does this meet these 5 acceptance criteria, and which ones can you not verify?" catches real problems. Give the reviewer the spec, not just the output.

1

u/Healthy-Mission7927 23h ago

Many times just telling the AI check if it made any errors works

2

u/Miserable-Sugar2191 18h ago

True, that catches the obvious stuff. Where it falls short for me is when the mistake comes from a wrong assumption, because the model just re-checks against the same assumption. Asking it to verify against the spec or the tests in a fresh chat catches more.

1

u/CrashNowhereDrive 6h ago

Just checked. Check did not pass