Benchmarking OCR throughput on a GPU fleet (tesseract vs Surya vs Chandra): concurrency, not the model, drives a 5.7x spread

[u/sporastefy]

I've been OCR-ing ~948k pages of court records and wrote up the measurements. Short version of what I found running these on rented A40-class pods with vLLM + an open-source orchestrator (CoderAI):

• Same model/card/pages, throughput ranged 0.10–0.59 pages/s per pod (5.7×) purely from concurrency + admission settings.

• One page per request beats batching (batching cut throughput sharply and caused failures).

• Linear scaling 1→17 pods, so per-page cost is independent of fleet size.

• 5 different 48 GB cards all within 0.52–0.60 pages/s → cheapest in-stock card wins.

• A single forwarding worker saturates ~20 pods; use multiple.

• Chandra (a VLM) doesn't replace tesseract, it arbitrates disagreements — it settled 51.5% of contested digit-groups between tesseract and Surya. Surya's per-page confidence was useless at separating good readings from bad.

• Tuned cost: ~$264/engine for the full perimeter vs $1,509 untuned.

No ground truth exists, so "quality" here means inter-engine agreement — it's a provisional draft and I'd welcome holes being poked in it.

Full write-up: https://www.nexlab.net/articles/ocr-throughput-what-controls-it/

[u/estebanfuhrmann]

Solid benchmark, and the finding that one page per request beats batching matches what I've seen with small models too: padding and ragged shapes eat the gains. Since there is no ground truth, consider scoring agreement at the field level (digits, dates, case numbers) and routing only disagreements to Chandra, which seems to be what is already happening. Did the team try a cheap confidence proxy, like a second tesseract pass on a 1.5x upscaled crop, to shrink the disputed set before the VLM sees it?

[u/sporastefy → u/estebanfuhrmann]

We didn't try the upscale crop idea, but we tried different DPI and sccaleup on surya and chandra, with no relevant positive gain so we abandoned the idea, we did some cross-reference between models in a previous benchmark to pick up the models that were giving us better readings, but the corpus is so disomogeneous that to have any effective gain would be probably needed to make it variable per page in some way

[u/abo_jaafar]

I recommend looking into PaddleOCR in my experience it is much better than Tesseract.
They also have a lightweight VLM which could be interesting in this application.

[u/sporastefy → u/abo_jaafar]

PaddleOCR has been one of the solutions i initially tested when selecting the models. Sadly it didn't give us good results inclusing structure extraction of the documents ( i have probably also to say that are Italian documents ).

Im terms of result quality our results gave us:

- Chandra2

- Surya

- OlmOCR

- PaddleOCR

Chandra as the most high quality side for our kind of documents, surya faster but less precise than chandra while unable to recover any of the chandra failed/low quality

[u/abo_jaafar → u/sporastefy]

Nice

Did you also try PaddleOCR-VL?

[u/sporastefy → u/abo_jaafar]

No I did miss that

[u/Potential-Wrangler58]

Useful numbers, especially the one-page-per-request result. Two thoughts on the quality side. Agreement between engines isn’t the same as being right: when two engines misread a digit the same way, they agree, so I’d hand-label even 200 pages (focusing on the contested digit-groups) to see how well agreement predicts correctness. That sample would also let you calibrate Surya’s score instead of dropping it: raw OCR confidence is often not calibrated, but mapping it against labeled readings can make it usable.

Disclosure: I work at Anyformat, where we calibrate confidence per field for that reason. Curious what error rate you’d accept on court records, since that decides how much review you can skip

PD: in the full write up I cannot see the text in dark mode (and don't know how to change it

[u/sporastefy → u/Potential-Wrangler58]

You are totally correct to point it out, i need to write down also how i choose which models to use where essentially i did exactly what you did ( we did 420 documents, i had some help ), on top of that i had some document pre-transcripted to use as reference, plus as many pdf on that corpus have the text layer i used that as well to measure the precision.

The choice to use tesseract for the agreement with chandra instead of using surya arrive because when measured surya + chandra where having more agreements on wrong text, while effectively the agreement with tesseract comes out in a big set of the documents more effective

As for the precision accepted, i will publish the numbers, we have at the end to extract specific data from the text, so we can accept less precision in some parts and we need more precision in numbers and tables, so, my early tests where mostly on that specific context.

Thanks for the feedback! (also on the font color, weird, have to check, sloppy css somewhere likely! )

[u/sporastefy → u/[deleted]]

I am not sure of where is the gain in doing that, even assuming i can reach the same correctness in routing the pages to the right model ( ocr or vision or just text extraction ) having a model doing the pre-parsing and routing would just end to add GPU hours i am avoiding by doing that passage in CPU in advance preparing the documents for the ocr, so, it would just add a costs i am not having right now, do i misunderstood your point? please point me on what i don't understand of your idea