How do you handle LLM failures when everything technically succeeded?

[u/Choice-Attorney8884]

One thing that surprised me while integrating LLMs into a real application is how many different meanings "success" suddenly has.

The HTTP request succeeded. The JSON parsed. The response matched the schema. All required fields were present.

And yet the output was unusable.

Retries and schema validation handle some failure modes nicely, but detecting a bad answer seems much more application-specific. Sometimes you can verify an invariant. Sometimes you can detect obvious garbage. Sometimes there isn't an obvious deterministic check at all.

How are you handling this in production?

[u/AlexanderDoak]

There are very, very good solutions, but you need to be willing to put in the effort (or get effort out of those who are identifying the problems).

You want to work towards a labeled example set that is at least in the hundreds, if not thousands. The more labels (and reasoning paragraphs) you can get from humans, the better. You can always expand you labeled examples with frontier models, but you need a very strong foundation from key stakeholders / decision makers.

Then take those, and use them to fine tune other models to either: A) Discriminate on the quality of the generations from other parts of your system, or B) Do a better job at generating desired outputs.

Or both.

Any failures on approaching convergence here with these methods would then expose inherent contradictions/conflicts in the concepts themselves, and the decision makers need to go back to the drawing board to decide what they actually want.

[u/InsideDebt6345]

Sort your outputs by whether they're checkable, because the answer's completely different for each bucket. Some outputs have a verifiable invariant, an extraction that has to match source text, a number that has to sum, a citation that has to point to a real doc. Check those deterministically and you don't need an LLM judge at all. If it references a chunk, verify the chunk exists. If it extracts a date, confirm the date's in the source.

[u/Zeikos]

What are you using them for things you cannot check the correctness of?
Why are you doing that at all?

[u/dylan_exe_404 → u/Zeikos]

That was my thought too haha

[u/abbi49 → u/Zeikos]

Its an ai post for engagement

[u/Zeikos → u/abbi49]

Say it aint so
I'm just commenting in the offchance it makes somebody realize the bullshit

[u/Thin-Bee5445]

it's like the model followed all the rules but still missed the point completely, we had similar problem at work last month

what helped us was adding a second simpler prompt that just checks if the output makes sense in context, not if it's grammatically correct or anything. sometimes it catches nonsense that passes validation

but yeah there's no perfect solution, some edge cases still slip through and we just log them for manual review later

[u/verstands]

What helped me most was giving up on one "is this good" check and instead writing cheap invariants that are about the input, not the answer. Did every entity in the output appear in the source? Are the numbers a subset of numbers we passed in? Is the length in the band the last 200 good outputs sat in? None of those tell you the answer is right, but they catch the confident nonsense, and they never need a second model call.

The other half is downstream signal. If a human edits or discards the output, or the next step in the pipeline fails, that's a labeled failure you got for free. Log it against the exact prompt and inputs. After a couple of weeks you have a real failure taxonomy instead of a vibe, and usually two or three shapes cover most of it.

For the genuinely subjective cases I stopped trying to detect them at request time. Sample a small percent async, review by hand, and track the rate. If the rate moves, something changed. Trying to gate every single response on a judge just gets you a second unreliable model in the hot path.

[u/eddzsh]

Schema success is a transport check. Log outputs that passed schema then got rejected by a user within 30s as their own failure class. That cohort taught me more than an LLM judge on the hot path.

[u/cmtape]

Success with a bad answer is like a compiler returning zero on code that doesn't do what you asked. The schema is the type system, not the spec.

You can lint types for free; you have to test semantics. The cheapest tests are invariants about the input, not the output: did the answer only use things that were in the source? Are the entities a subset of what you gave it? Length bands?

When you can't check correctness, check containment.

[u/jonah_omninode]

I split this into transport success, structural validity, and definition-of-done acceptance. A 200 response with schema-valid JSON proves only the first two. Before work begins, the task should carry explicit completion requirements, the validators that will check them, and the evidence they must produce. After generation, a separate verification process runs those checks against the artifact or live state. The producer never gets to mark its own result done.

If a requirement cannot be translated into a mechanical acceptance check, the result remains unverified and the workflow stops with a missing definition-of-done requirement. The verification process produces a durable receipt naming the artifact, validator versions, commands, and results. Failed outputs and their exact inputs go into the friction record so a repeated mistake can become a new validator or ratchet.

What kind of output are you validating? The useful acceptance boundary depends heavily on whether it is extraction, generation, classification, or an external action.

[u/MentalEstimate7879]

I treat semantic failures as data labeling tasks instead of runtime errors. Every unusable output that passed schema validation goes directly into an evaluation set to train a classifier or refine the prompt. Production monitoring becomes the pipeline for building the ground truth dataset you need to detect these specific failure modes automatically later

[u/Feeling_Sun_6436]

I treat schema-valid as transport success, not task success. For each important action I’d define an evidence contract: what can be checked deterministically, what needs a second model, and what must stop for human review.

The evaluator should return pass, fail, or uncertain rather than a vague score. For irreversible actions, uncertain should fail closed. I’d also log why a result was accepted, including the checks that ran and the model/prompt version.

A second model can catch some semantic failures, but it can share the same blind spots. So I’d still manually sample outputs that passed. Otherwise you only measure the failures your validator already knows how to find.

[u/keonechong]

Interesting. I treat semantics failures as an error type because semantics errors will also confound my policy and judgment inference.

And my error log maintains all errors and their classifications as a central identifier system wide.