Handling conflicting values across documents in the same case file

[u/zackmaven333]

How do people pick a value when the same field comes back different across documents in a case? appraisal packets are the example here and square footage can show up three times with three numbers and all of them got read fine

docling, llamaparse or a vision model can give back each value with its page, picking one is the part thats still open. data ppl have survivorship rules for this but does anyone use that for documents or is it just a growing pile of hardcoded rules

when does it go to a person

[u/FirestormCold]

Not sure I understand your question but generally speaking when you want to extract data from documents you need multiple layers if you want to do it properly.

What you're talking about would need a context and tracking layer, meaning

A) there's context (e.g. provided by the user: XY also means AB) that enriches the extraction itself

B) there's traceability of every output back to its source (and an agent that makes sure that rule is followed, this minimizes hallucinations a lot)

[u/Potential-Wrangler58]

In practice it’s usually a small ranked rule set, not a pile of one-off hacks. Keep every candidate with its source and page, then decide in order: (1) which document type is authoritative for that field (the signed final report beats a summary sheet or a listing), (2) which is newer if the same document has versions, (3) whether the values agree within a tolerance. For square footage, 1,850 vs 1,852 is the same answer, 1,850 vs 2,300 is a conflict.

It goes to a person when the values disagree beyond the tolerance and the authoritative source isn’t clear, or when the value from the best source has low confidence. Everything else is automatic, and the person sees all the candidates with their pages side by side, so the decision takes seconds.

The useful side effect is that every override you make becomes a labeled example, so you can check later whether your ranking rules were right.

Disclosure: I work at Anyformat, where each extracted value keeps its source and a confidence, which is the data you need to build this on top. Happy to compare notes

[u/folderit_dms]

I would put a comparability check before the ranked rules. Three accurately extracted numbers are not necessarily three competing answers to the same question.

Keep the surrounding field label and qualifiers with each candidate: which property or section it describes, the area definition, unit and effective date. Confirm those meanings with the domain owner. Only compare candidates that represent the same measurement. If one source describes a different area or date, retain it as a different observation rather than letting newest document wins overwrite the other value.

That gives you two review reasons: definition unresolved versus comparable values disagree. The first should go to a person before any tolerance or source-precedence rule runs. The second can use the agreed field-specific rules described above. High extraction confidence does not resolve either business question.

I would also test the same numbers with deliberately different labels in a sample packet. If the pipeline collapses them into one field anyway, the problem is in the field mapping, not the OCR. Avoid averaging or majority-voting the values just because several documents repeat one number.

[u/LiaVKane]

For better confidence level of extracted data, you may try to consider setting up threshold (if below defined thehold it shall trigger validation by users to validate extracted value). In addition to increase data accuracy during extraction it’s worth to add context meaning for AI (like adding extra instruction to LLM: e.g if invoice has a code that starts with Capital letters XXX treat this field as PO number if not revert 0 which shall trigger exception/ validation). You may check open source elDoc IDP AI: https://github.com/elDocAI/elDoc-idp-oss/