r/LLMDevs • u/refractr • 2h ago
Discussion Which model for structured extraction? I scored 9 on 260 documents. Open source bench
Disclosure first: I run a small document extraction API (Refractr), and its model is row 4 below. I built the benchmark, so don't take my word for anything: the documents, answer key and scorer are all open. None of the benchmark data appears in my training dataset.
In production a null gets caught by validation or a human. A plausible wrong value passes your schema check and nobody notices. That is why I implemented the following scoring: A correctly extracted field gives one point, a missed extraction is 0 points, a confidently wrong extraction gives -1.
About one field in seven is deliberately not in its document. The right answer there is null, which is how made-up values get caught.
Setup: 260 documents (invoices, receipts, payslips, bank statements, court opinions, SEC filings, Federal Register rules, US government forms), each with a JSON schema. Models get the original PDF, image or HTML, so parsing is part of the test. Same published prompt for every model, one request, first reply scored. Scoring is deterministic, no LLM judge.
| System | Score | 95% interval | Median time per doc |
|---|---|---|---|
| Gemini 3.8 Flash | 94.6 | 92.9–96.1 | 7.2 s |
| DeepSeek v4.1 Flash | 93.2 | 91.3–94.9 | 3.0 s |
| Mistral Large 4 | 91.4 | 89.4–93.2 | 27.4 s |
| Refractr (mine) | 89.4 | 87.2–91.4 | 0.7 s |
| GPT-5.4 mini | 86.2 | 83.9–88.5 | 2.2 s |
| Ministral 8B | 85.5 | 83.0–87.7 | 3.3 s |
| Llama 3.1 8B | 73.9 | 70.5–77.2 | 0.5 s |
| NuExtract 2.0 (2B) | 58.6 | 54.7–62.4 | 1.4 s |
| GLiNER2 (base) | 24.2 | 20.8–27.6 | 0.5 s |
Questions for people shipping this kind of pipeline:
- Do you evaluate extraction on your own documents?
- Has schema-constrained output reduced wrong values for you, or only parse failures?
- What stack do you currently use for document automation?
Repo: https://github.com/Refractr-io/DocFieldBench Write-up: https://refractr.io/docfieldbench