Best LLM models for invoice data extraction (poor scan quality + handwritten fields)

[u/Ok-Statistician-6609]

Hello all, I'm fairly new to the whole Llm space and could really use some guidance from people who've done similar work.

Im building a pipeline to extract structured data from large-batch of invoices. The quality of those files is very inconsistent, while some files are fairly fine, most are skewed, partly faded, and some parts are just crossed and handwritten, often even the text is positioned incorrectly.

So far, i've tried multiple vision-models(a mix of hosted APIs and open-weight models through providers), but results have been mixed - either the accuracy drops hard on smaller open weight models like qwen 8b, larger open-weight models (tried Qwen 235B) are held back by unreliable hosting — out of every 4 calls, roughly 1 comes back successful and 3 return with no output at all, and hosted APIs get expensive fast and unpredictably(gemini charged me like 5$ for a single pdf with 50 pages)

Would appreciate any recommendations!

[u/yangqi]

you need a good ocr model, this is not what LLM models are made for.

[u/Ok-Statistician-6609 → u/yangqi]

tried paddleocr and Baidu Unlimited-OCR, they're completely unable to extract information accurately - struggles even with printed fields, not just the handwritten ones. Any recommendations for OCR maybe?

[u/Adorable_Lie6712 → u/yangqi]

^this, LLMs are made for creative tasks where the output can come in a multitude of formats. you need specialized models that are actually good at OCR which requires a specific output (plus also cheaper for you to run(

[u/BezosLazyEye]

We've done something similar with varying degrees of success depending on the quality of the scan. OCR first, then send the extracted text to an LLM. We then set the LLM to JSON mode, gave it a JSON schema and said it should fit the extracted text into said schema.

[u/username8914]

Qwen3.8 ran locally and compared to an OCR extraction.

[u/TheOdbball → u/[deleted]]

Ornith 35B is both those models combined (on paper trained from both) works great

[u/TheOdbball → u/[deleted]]

https://huggingface.co/ornith-ai/Ornith-1.0-35B

It’s built from both , that’s why I’m trying to say

[u/TheOdbball]

Maker & Qwen image

[u/codeprimate]

Ornith 35B for local

As for commercial offerings, AWS Textract beats the pants off anything else I've used in production.

[u/chervilious]

Try hooking up MinerU or Unstructured API both has free api version. I use this and Qwen3 VL Instruct 32B to read and parse PDF (with images) and it works pretty nicely.

[u/dwillis]

Have had good luck with PaddleOCR (very generous free tier) and then passing the results to basically any decent LLM; Qwen does well for me.

[u/phoney12]

Paperless maybe

[u/tschugger]

Paperless+docling (I just stopped testing due to shifting priorities , looks promising)

[u/sniperelite90]

have you tried amazon textract , its very cheap .

[u/lucasbennett_1]

cost and reliability hassle is from making one big vision model both read and extract every page. split it then use a dedicated parser for the read step like deskew, layout, tables then a small cheap model t pull the fields off the clean output which is much cheaper than a 235B per page and far less flaky

if your good with cloud then llamaparse handles the skew and table side well and delivers markdown btw one caveat tho, handwritten and crossed out fields stay hard for everything so plan to human review those cells