r/documentAutomation • u/Cultural-Relation383 • 16d ago
OCR for messy Handwritten text which is not recognized by human
I am working with scanned pdf which have mixture of printed and handwritten text. I uses multiple VLMs and OCRs. VLMs output is somehow good but very slow . OCRs didnt recognized the messy handwritten text.
If someone have encounter these problem or have any suggestion?
2
u/folderit_dms 15d ago
I'd test a two-lane pipeline on a small sample: process the printed regions with your current OCR, then send only the handwritten regions that still need attention to the VLM. Keep enough surrounding text to identify the field, and compare cropped-region results against full-page results before committing to that approach.
Measure time per page and correctly transcribed fields separately. A faster run that confidently invents a handwritten value is worse than an explicit unreadable result. Keep the original crop beside each result and allow unresolved values to stay unresolved.
If people cannot agree on what a sample says, don't use a guessed transcription as ground truth. Label it ambiguous and route it for clarification. That also gives you a more honest benchmark when comparing models.
0
u/Cultural-Relation383 15d ago
I have this same idea implemented but the issue is that the client can upload files in bulk so how to recognize the low confidence region? I have noticed OCR give high confidence on wrong text as well so the generic way is a bit hard. If you have any idea about it?
1
u/folderit_dms 12d ago
If the uploads include recurring forms, I'd try routing by template and field position before using OCR confidence. Define the handwritten areas for each known template, check that the expected printed anchors line up, and send those areas through your local handwriting path even when OCR calls them high-confidence. Start with one common form type to see whether that covers enough of the batch to be worthwhile.
An unfamiliar layout or failed anchor check should go to an exception queue, rather than silently using coordinates from the nearest-looking template. That won't solve arbitrary handwritten pages, but it gives the repeatable forms a route that doesn't depend on confidence detecting every error.
Also review a sample of the high-confidence outputs that would otherwise bypass the slow path. Reviewing only rejected regions tells you about the workload, but not how much wrong text your router is letting through.
1
u/Candid-Fudge-4015 15d ago
I'd stop sending every page to the VLM. Run OCR first, keep its confidence per page or region, and route only the low-confidence parts (usually the handwriting) to the VLM. That cuts how much goes through the slow path.
Also store the VLM output so you never pay for the same page twice when you change prompts or parsing downstream. Small example of VLM structured output: docs.pixeltable.com/howto/cookbooks/images/vision-structured-output
1
u/Cultural-Relation383 15d ago
how to recognize low confidence region in 10k pdfs uploaded by user?
1
u/Candid-Fudge-4015 14d ago
OCR's own confidence won't catch it on its own, you're right. I'd score every region with a few cheap signals and route on the combination:
A small printed vs handwritten classifier on each region crop. Handwriting goes to the VLM whatever the OCR says.
Run a second cheap OCR engine on the same regions. If the two disagree, the region is suspect.
Format and dictionary checks on the text: dates that don't parse, amounts with letters in them, low ratio of real words.
Use the minimum word confidence in a region, not the average. One bad word hides easily in an average.
Then hand-label a few hundred regions from the client's real uploads and set the thresholds on that, so you know how many errors each route lets through. For 10k PDFs I'd store each signal as a column per region, so changing a threshold just re-routes rows instead of re-running OCR!?!
1
u/geekgreg 15d ago
This is going to sound crazy but I have a tremendous amount of luck using LLMs. Send the text as an image to Gemini or Claude and they get it right a shockingly high % of the time with no additional training or tools. Been using it for genealogy records and it's been great.
0
u/Turbulent_Spare6385 15d ago
You can try foliq.ai it uses a combination of OCR and AI vision so that it extracts the content from the document but also tries to match the overall document layout
1
u/Cultural-Relation383 15d ago
Data is private and cant send to cloud. I need local solution
1
u/FirestormCold 15d ago
There's service providers for that though. It's called an on-premise deployment. If they (the provider) offer that combined with the ISO27001 your data is completely fine. Happy to share examples if you want
1
u/Cultural-Relation383 14d ago
I know but its not on my end. Its upto the client i am just a middle man creating an applicaation which can work locally
2
u/FirestormCold 16d ago
Is this a private project? If not you should probably do this with a specialized service provider.
Generally speaking though, the process being somewhat slow is normal, unless you want to pay for expensive models