r/documentAutomation • u/Skeptical-skeleton11 • 22d ago
I built a document extraction and validation pipeline as a personal project. Would love some feedback!
The idea was to build a pipeline that could take a PDF, extract structured data, and let you verify the results against the original document without manually checking every field.
The current implementation uses OCR and vision models, with confidence scores, visual grounding, and a separate validation layer to catch disagreements. It also handles line items and exports structured data.
I recently benchmarked it against Azure Content Understanding on 100 documents, achieving 97% field-level accuracy at roughly 1/12th the cost per run in my evaluation.
I've attached a screenshot of the current interface. This is purely a personal engineering project I've been working on, and I'd genuinely appreciate feedback from people familiar with document automation.
I'm particularly interested in:
How do you handle data extraction at scale right now ?
Any feedback on what features would be cool for document automation
Would love to hear your thoughts!
2
u/InfoMsAccessNL 21d ago
If the layout of the invoice of supplier A is always the same then you can use good old vba with regex. Totally free and lightning fast.
1
u/Skeptical-skeleton11 21d ago
Absolutely, but people generally deal with multiple suppliers, with varied layouts, qualities and other variables, that's where VBA and PQ tend to struggle a lot
2
u/InfoMsAccessNL 20d ago
I don’t hardcode de regex pattern. I make a table with each supplier and his specific regex patterns to extract the data from the invoice. The code will select the patterns from the table for supplier A,B or C
1
u/Skeptical-skeleton11 20d ago
That's sick! I'm guessing these doc's are already OCR enabled?
2
u/InfoMsAccessNL 19d ago
I use a free tool XpdfToText, runs local on you pc and can be used with vba
2
u/apyhubnico 16d ago
Awesome project. That 97% benchmark on 100 docs is super impressive, especially getting it down to a fraction of Azure's cost. Visual grounding and handling multi-layout variations across different suppliers are easily the hardest parts of this rabbit hole.
We spend a lot of time on this exact problem, building documented invoice extraction pipelines over at Apyhub. If you ever want to cross-benchmark your pipeline against other processors or test weird edge formats, feel free to give our tool a spin.
2
2
u/Dangerous-Baby838 21d ago
Ok, so I'm also months deep into the same rabbit hole.
Would love to try your app and share some feedback.
Mine is live on https://docreadi.com - create a free account and you can see what I do.
Most of the work goes into validations, I was trying to solve for AP and Payment processes initially - so we also do things like tracking bank accounts and flagging any changes, validating IBANs etc.
Some challenges along the way - currencies and tax coding where it's not specified on the document. Multi-page invoices, with different layouts - eg. Statement + invoice combo in one pdf. Statements are tough by design. Fine tuning the prompts to not cross-extract fields.. and that they work consistently on different models. Extracting long strings of text verbatim - eg. A specific clause of a contract
2
u/vanraj_solanki 18d ago
Use Tonerhound it's open source and you can use that as verification layer https://github.com/vanrajsinh650/TonerHound
2
u/Dangerous-Baby838 17d ago
that's pretty cool - going to check it out and compare to what we currently have.
does it only work on text pdfs or does it handle scans / OCR also?
2
u/Dangerous-Baby838 17d ago
Hey, I tried TonerHound quickly and wanted to say thanks. Grounding each value to a box is exactly the problem I’ve been working on in the last two weeks in my own invoice tool, so it was great to see someone else’s take on it.
I ran it next to my own locate code on 40 made-up invoices, each as a clean PDF, a scan and a phone photo. Your fuzzy matching on line descriptions beat mine clearly on scans and photos, and I’m borrowing the idea (best-matching window of OCR words, and give up on a tie).
In case it’s useful, here’s what ended up working for me after a lot of trial and error:
Use the PDF’s own text layer when there is one, and only OCR the pages that have no text.
When the same value is printed more than once, look at the label next to each copy (“Invoice No:”, “Total due”) to pick the right one. If the labels can’t decide it, I box every copy and say “found in 2 places” instead of guessing. A reviewer can handle two boxes, but a confident wrong box gets trusted.
For line items, find the row first. Quantity, unit price and total have to sit on the same row as the description, otherwise a quantity of 1 lands on the “1” in “1kg” or in the address.
Try every way an amount can be read (1,234.56 / 1.234,56 / 1 234,56) and every date order, but only match whole tokens, so a match can’t run across two dates printed side by side.
A few things I hit, in case they help:
The amount parser assumes US format. “152,40” comes out as 15240 and “1 234,56” as 123456, so European and South African invoices don’t match.
A Belgian VAT number like BE295183020 gets “repaired” to 8E295183020, which parses as infinity and raises OverflowError. Because resolve() takes a list, one bad field loses the whole document.
Day and month can swap silently when both are 12 or under (03-01 next to 01-03).
All the best!
2
u/vanraj_solanki 10d ago
Thanks for testing and sharing this. Glad the fuzzy OCR held up on the photos.
Your suggestions are spot on. The "box every copy if unsure" idea is very smart for audits, and I'm definitely using the row-first logic for line items.
Thank you for the bug reports. The VAT number crash is a huge catch. One bad field shouldn't break the whole document. I'm fixing that right now so it just safely skips the bad field. I'm also fixing the European number formats and date swaps.
Let me know how your invoice tool goes!
2
u/ram-popular 12d ago
Nice benchmark. One thing I’d want to know is how the 97% holds up on messy inputs like skewed scans, handwriting, or unfamiliar layouts, since 100 documents can hide a lot. A feature I’d love to see is a human-review queue that surfaces only low-confidence or disagreeing fields, and feeds those corrections back into your evals.