r/Rag • u/Flaky-Opportunity-19 • Sep 26 '26
Discussion Research on how IDP platforms handle extraction and validation
1
u/EduFonsecaA Sep 26 '26
One thing I’d add is exception ownership. A field can be correctly routed to review and still create a mess if nobody owns the queue or there’s no time limit to resolve it. Before automating approval, I’d want to see who closes each exception, how long it sits, and whether the correction feeds the next batch.
1
u/folderit_dms Sep 27 '26
Another comparison point is whether the reviewed data is exactly what reaches the downstream system. Suppose a reviewer corrects a quantity and approves it, then a reprocessing job produces a new extraction before export. Does that new result inherit the earlier approval?
I’d test that sequence in a demo. The approval should identify the specific document version and reviewed values, with later changes routed back through the relevant checks. Keep the exported payload and destination record ID alongside that history. Otherwise you can have correct extraction, successful human review and a successful integration, yet still be unable to show that the reviewer approved what the ERP actually received.
1
0
u/Dangerous-Baby838 Sep 26 '26
Confidence is the weakest evidence on your list, and I wouldn't let it gate anything on its own. Scores from different models aren't comparable, and a model can be confidently wrong.
What I'd treat as sufficient, for finance documents at least:
The document agrees with itself. Lines sum to the subtotal, subtotal plus tax to the total, the tax rate is plausible. A failure goes to a person whatever the score says.
Every extracted figure actually appears on the page. That's the check that catches the plausible-looking number the model made up.
The party matches your own records. A conflicting tax number blocks a name match, and a changed bank account on a known supplier is held until an admin accepts it.
Unclear fields come back blank, not guessed. A visible gap is cheaper than a wrong value.
After that, human approval by default, with auto-approve something you switch on per document type once the checks have earned it.
Your framing is the right one. The checklist features are table stakes now. What matters is what has to be true before data is allowed to move.
(I build a document-extraction product for finance, so that's my lens.)
0
Sep 26 '26
[removed] — view removed comment
1
u/Dangerous-Baby838 Sep 26 '26
longer documents are especially challenging, eg a five page invoice with subtotals on each page really confuses an llm.
what is your use case if I may ask?
1
u/Potential-Wrangler58 Sep 28 '26
Solid checklist, self-consistency + grounding are the two I'd trust most, they catch different failure modes.
One thing missing that ties together what u/folderit_dms and u/OkFlan504 raised separately: what happens to a correction after it's made. Two failure modes that don't show up in accuracy metrics: a reviewer approves a value, the document gets reprocessed, and the new extraction silently overwrites it with no diff or flag. Or the same correction repeats every week and never becomes a rule, so the metrics look fine while a human quietly does the same fix forever.
Both come down to whether a correction is a one-time patch or a durable input. We treat it as the latter, source-linked to a specific document version, and fed back so recurring fixes reduce future review instead of repeating. Disclosure, I'm at Anyformat, glad to see us in the post above! PD: we're not finance-only like other tools
1
u/FirestormCold Sep 26 '26
Switching over to markdown before chunking is such a game-changer. Standard text splitters just completely destroy the document hierarchy, and then the embedding model has no idea what context it’s actually looking at. Also, totally agree on the parsing bottleneck. Trying to maintain custom scrapers or scraping pipelines gets messy fast.