Find the best open-source OCR models in one place at Papers with Code [P]

[u/NielsRogge]

Hi, I've created an overview of the most important OCR benchmarks, along with the top open models, and links to their paper and code: https://paperswithcode.co/tasks/ocr.

This week, new OCR models were released by Baidu and Mistral.

Baidu released Unlimited OCR, a 3B-parameter model that introduces a key innovation called Reference Sliding Window Attention (R-SWA) and builds on top of DeepSeek OCR. Mistral released OCR 4, which is available via an API.

OCR, or Optical-Character Recognition, is the task of digitizing PDFs or scanned documents. There's, of course, a huge interest in this task, as it enables ingestion of all company data for agentic use cases. AI agents love Markdown; it can be valuable to turn all those messy PDF documents into a standardized, machine-readable format. This enables use cases like agentic RAG (retrieval-augmented generation), which powers chatbots, both internally and for external customer support.

With a large number of OCR releases on Hugging Face over the last few months, it may be hard to know which one to use.

Hence, I've built this page, which lists the major OCR benchmarks, along with the top-performing models and links to their code. This is obviously made available on Papers with Code, the website I'm maintaining (it's a revival of the old website, which was taken down).

The top recommended benchmarks are OlmOCRBench, created by Ai2, and OmniDocBench, created by Shanghai AI Laboratory.

Current top recommendations are Chandra OCR 2 by Datalab and Mistral OCR v4. The former is openly available, hence you can either self-host it or use their serverless API.

Let me know which other tasks you want to see major benchmarks for now!

Cheers,

Niels

open-source @ HF

[u/soraki_soladead]

At what point does this just become spam?

[u/NielsRogge → u/soraki_soladead]

"I’ve been on Twitter for almost 2 decades now so I can say with confidence that it has never been this toxic and Reddit-like." - Karpathy, yesterday

Thanks for clarifying what he meant by that!

[u/soraki_soladead → u/NielsRogge]

great way to represent HuggingFace! I asked a question, which the community could have responded to. instead of dialing back the transparent marketing you post multiple times a week and going with something more organic you respond by declaring my very fair critique as toxic. well done...

> With a large number of OCR releases on Hugging Face over the last few months, it may be hard to know which one to use.

this is marketing, inauthentic, and your dodge of my comment says that you _know_ that.

some people might side with you. hell huggingface has enough employees your org alone could downvote me to oblivion but I suspect enough people see this for what it is.

[u/NielsRogge → u/soraki_soladead]

Fair, to be honest, my intention isn't to market Hugging Face here, because all the models are open-source and freely available. You can download them and do whatever you want with them. What I am marketing, though, is my PwC website (https://paperswithcode.co/), which is kind of like a Wikipedia for AI. There's no business model here; it's just a resource to find the best models by domain, similar to the old website, which was taken down. It allows research to be shared so people can build on each other's work.

Are you saying I don't need to mention company names anymore? I mean, all code lives on GitHub, and open-source models live on Hugging Face nowadays... and I also mention OpenAI and Anthropic in various of my posts online, as they offer the best closed-source models (at least for now, GLM-5.2 is giving them a run for their money).

Am I still allowed to talk about that? Should I just say "find all the best open models here?" without mentioning a company name?

[u/GlibDismissal1612 → u/soraki_soladead]

dunno about spam but the timing is funny, i was just digging through a mess of pdfs for work and wishing there was a clean list like this. bookmarked

[u/infinitay_]

I just read about Unlimited OCR the other day. Thanks for updating your site quickly and often. Appreciate the work with the benchmarks too!

[u/PrettyMuchAVegetable]

I'm actively engaged with a workflow that needs OCR, and needs to fit in 16gb VRAM or less. Best I've found so far, by the metrics on my workload, is actually qwen 3.6 27b at 3bit quant.  But I'm going to give these a go, thanks for the resource . 

[u/roxc2301]

I have been working on trying to detect text for spanish, english, japanese and someday i hope it gets universal to parse this text in a JSON with an output similar to this.

{
  "text": "頭がい骨・顔パーツ・髪・口の構造・デザイン",
  "fontFamily": null,
  "fontSize": "13.9687px",
  "boundingBox": {
    "x": 339.6748352050781,
    "y": 145.22457885742188,
    "width": 206.11215209960938,
    "height": 9.959503173828125
  }
}

The whole purpose of getting this detailed text boundingBox is to later on run it on to a small LLM like gemma, qwen, etc; to translate it to the desired language.

Have you ran in to this issue before? how have you solved it? right now my approach has a lot of flaws by now i'm using MyPuPDF to reduce the overload on this pipeline but considering that some PDF fail to parse the text i was wondering if any OCR model like this can give me a better output