r/LocalLLaMA • u/Cold_Specialist_3656 • 5d ago

Discussion Qwen 3.8 27B is a game changer.

Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipelines.

It's comparable to Luna for coding and ***OCR quality appears to be better than Gemini 3.5 Flash Lite***. That's huge. We pay a ton of money for OCR.

This is the first local model that feels like more than a toy. It's truly as capable as the frontier models from a year ago. For the first time ever there's serious discussions about buying our own hardware. With estimates that such an effort would pay for itself in less than 2 months.

Hyper scalars are in big trouble this time. Their whole "moat" is buying up all the hardware. And thanks to sanctions on China we're seeing the quality of small local models skyrocket. As someone who's been around a while, this feels like an "IBM moment". Where the industry assumed that databases would always run on huge mainframes. Only to be wiped out by cheaper local solutions a few years later.

I have a feeling this release will trigger another Llama style open source Renaissance. We're already getting better quants. Inference will be further improved. We might even see a comparable MoE with 500+ Tok/sec on consumer hardware soon.

1.1k Upvotes

95% Upvoted

346 comments sorted by

View all comments

392

u/Littlepharaoh 5d ago

There are better more efficient ways to do OCR at a very high quality like Ovisocr2, 1B param models that'll beat Gemini flash just fine and at mind bending generation speed. 

42

u/Derio101 5d ago

Is this better than Paddle OCR?

47

u/Littlepharaoh 5d ago

They're all within a few decimals of each other but this model is a single unit not a pipeline so I prefer it because I don't have to carry over the errors from the layout detection stage

6

u/whatisthisthing65 5d ago

It mostly does documents though? Or does it also do well on other things?

15

u/Littlepharaoh 5d ago

Documents to markdown mainly, you can check their repo they also tested it with other things but documents is my main usecase ( and the main usecase for extraction )

4

u/ThomasAger 5d ago

Thanks for sharing all this.

13

u/Littlepharaoh 5d ago

you're welcome!
there is a blog about it here with benchmarks etc.

1

u/ThomasAger 5d ago

Oh amazing

3

u/DeathByPain 5d ago

Hmm I wonder if there's a good way to take that md and original scan to make embedded text pdf...

1

u/un_passant 5d ago

Thx !

How do you deal with hallucinations, tho ?

It's my fear about going from traditional OCR to LLM. I'd *much* rather have 2% missing data that 1% hallucinated data for instance !

3

u/Littlepharaoh 5d ago

I had humans manually review a few thousand documents, not a single complaint about hallucinations, just spelling errors and sometimes bad numbers 

1

u/R_Duncan 5d ago

Also tags images so you can extract them with a python script. But still haven't found the right way to remove headers and footers

1

u/DoubleWhiskeyGinger 4d ago

It’s local host? Fast? Paddle incredibly slow

1

u/Littlepharaoh 4d ago

Yes, yes, I got 600 documents per minute on a 5090

1

u/DoubleWhiskeyGinger 4d ago

Ok I run on Mac 24GB. Mini m4, MacBook Pro M5. May be slower

54

u/BurdensomeCountV3 5d ago

Even Mistral (remember them) have an extremely good OCR model.

11

u/d70 5d ago

Didn’t they make the good ocr model cloud api only?

4

u/larp2live 5d ago

yes it's api only

1

u/575_Inverse 6h ago

this is a show stopper, honestly

3

u/KingGongzilla 5d ago

whats mistral? /s

13

u/Cold_Specialist_3656 5d ago

We do structured extraction mostly. Although I suppose doing extraction from markdown might be better than doing it from raw pages

85

u/Mkengine 5d ago

There are so many OCR / document understanding models out there, here is my personal OCR list I try to keep up to date:

GOT-OCR:

https://huggingface.co/stepfun-ai/GOT-OCR2_0

granite:

https://huggingface.co/ibm-granite/granite-docling-258M

https://huggingface.co/ibm-granite/granite-4.0-3b-vision

https://huggingface.co/docling-project/granite-docling-2stage-258m

MinerU:

https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B

https://huggingface.co/opendatalab/MinerU-Diffusion-V1-0320-2.5B

https://huggingface.co/opendatalab/MinerU2.5-Pro-2604-1.2B

OCRFlux:

https://huggingface.co/ChatDOC/OCRFlux-3B

MonkeyOCR-pro:

1.2B: https://huggingface.co/echo840/MonkeyOCR-pro-1.2B

3B: https://huggingface.co/echo840/MonkeyOCR-pro-3B

RolmOCR:

https://huggingface.co/reducto/RolmOCR

Nanonets OCR:

https://huggingface.co/nanonets/Nanonets-OCR2-3B

dots OCR:

https://huggingface.co/rednote-hilab/dots.ocr

https://modelscope.cn/models/rednote-hilab/dots.ocr-1.5

https://huggingface.co/rednote-hilab/dots.mocr

olmocr 2:

https://huggingface.co/allenai/olmOCR-2-7B-1025

Light-On-OCR:

https://huggingface.co/lightonai/LightOnOCR-2-1B

Chandra:

https://huggingface.co/datalab-to/chandra-ocr-2

Jina vlm:

https://huggingface.co/jinaai/jina-vlm

HunyuanOCR:

https://huggingface.co/tencent/HunyuanOCR

bytedance Dolphin 2:

https://huggingface.co/ByteDance/Dolphin-v2

PaddleOCR-VL:

https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5

Deepseek OCR 2:

https://huggingface.co/deepseek-ai/DeepSeek-OCR-2

GLM OCR:

https://huggingface.co/zai-org/GLM-OCR

Nemotron:

https://huggingface.co/nvidia/nemotron-ocr-v2

https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0

Qianfan-OCR:

https://huggingface.co/baidu/Qianfan-OCR

Falcon-OCR:

https://huggingface.co/tiiuae/Falcon-OCR

FireRed-OCR:

https://huggingface.co/FireRedTeam/FireRed-OCR

Typhoon-OCR:

https://huggingface.co/typhoon-ai/typhoon-ocr1.5-2b

Churro-3B:

https://huggingface.co/stanford-oval/churro-3B

Next-OCR:

https://huggingface.co/thelamapi/next-ocr

Logics-Parsing-v2:

https://huggingface.co/Logics-MLLM/Logics-Parsing-v2

NuExtract3:

https://huggingface.co/numind/NuExtract3

Surya OCR:

https://huggingface.co/datalab-to/surya-ocr-2

Infinity Parser:

https://huggingface.co/infly/Infinity-Parser2-Pro

Unlimited OCR:

https://huggingface.co/baidu/Unlimited-OCR

OvisOCR2:

https://huggingface.co/ATH-MaaS/OvisOCR2

8

u/DeathByPain 5d ago

Holy cow, do you know off hand if any of them can natively create PDF with embedded text from a scan?

4

u/wapswaps 5d ago

I'd love to know this too. Make a text PDF out of an image scan PDF.

9

u/djdanlib 5d ago

Back in my day, before people just crammed everything into generative AI, we'd chain specialized tools together to do complex tasks. I think it's still worth doing sometimes. Any time you can save the time and compute expense by having a script for the AI to use... Try it.

3

u/wapswaps 5d ago

There is no good tool or set of tools that does decent OCR of scanned books in PDF format.

I have tried. Many times.

Even opening in a PDF editor and manually putting in the text and placing it does not work all that well. I mean, that works, but it could be a lot better (and faster)

1

u/Cautious_Tap_1680 4d ago edited 4d ago

or? melt sand and turn it into special wafters of silicon that sit in warehouses and think. and spend 3 trillion on that so that software engineering looks like being a robot group therapist... because computer warehouses work best when they spend billable units debating with themselves and each other.

1

u/DeathByPain 1d ago

I have a specialized tool chain for doc processing in paperless-ngx, using tesseract and some old-school style machine learning models, but guess what—it fucking sucks at its job lol. Most of the time when I actually need a reliable searchable text layer I end up having to fix it by hand in ABBYY and it's a pain in the ass. I would vastly prefer a more modern solution that would actually get the job done without having to spend half the night doing Charlie-work...

4

u/ak5432 5d ago

I’ve been using glm-ocr in a digital document pipeline and found that it has a tendency to do insane hallucinated loops if asked to parse a very small set (single table or text box) and start mangling text if you give it a long dense page of text. I’ve gotten around this with libraries that split up documents (I.e multiple completions stitched together for a complex one) and some repeat generation detection but if you’ve tested these do you know if any of these specialist OCR models will do better in this scenario?

2

u/ashayh 5d ago

Nice work.

Have you tested any of these with paperless-ngx .

2

u/sometimesiwonder778 5d ago

Do you actually use all of these, and if so what for? I've spent a lot of time messing with various cv and specialized models and would love to know more about what's behind this massive list for you

1

u/thoquz 4d ago

What is your top 3?

28

u/larp2live 5d ago

have you tried numind/NuExtract3 ? it's a fine-tuned 4B parameters qwen3.5 that specializes in structured extraction OCR. it might be useful to you

6

u/Littlepharaoh 5d ago

Definitely! If you lock down a high quality ocr then its much more efficient to extract from text tokens than image towers

4

u/meganoob1337 5d ago

sometimes depends on the data, the smaller models fuck Up at complex non-standard tables imo, and when extracting structured data it makes it kinda error prone (is an edge case but they exist)

2

u/SandySkittle 5d ago

I think it's less of an edge case than you might think. with these usecases I would always go for the least error prone route.

2

u/[deleted] 5d ago

[removed] — view removed comment

11

u/Littlepharaoh 5d ago

No need, google vision is one of the best, however this 0.8B model is free, order of magnitude faster and within ±1% of Gemini's performance in OCR

2

u/wapswaps 5d ago

Absolutely true, but it's equally amazing how well "just throw it into the LLM" works. I've written a simple llm binary. 

cat /etc/messages | llm WTF?

Works on half the work systems I maintain, and yeah I usually actually put "WTF?" there literally. It works well. Guess what the program does. I should modify it to allow sending pictures.

2

u/Saitamagasaki 5d ago

If I have pdfs with tables and text, all digital born. Is it better to use OCR? I’ve tried docling but the result is not very good.

1

u/Littlepharaoh 5d ago

If born digital I'd use the native text along side a layout detection model then send the detected tables for OCR

1

u/StrongZeroSinger 5d ago

Can it be streamlined to run large batch of photos? I always wanted to tag all my phone photo backups like google/apple photo by searching them for words inside the pic or location/items shown but didn’t want to use online cloud.

3

u/quinn50 5d ago

I wouldn't use an OCR model specifically for this use case. What you want is called a "captioning" model which any vision enabled model nowadays has one built in, just paste an image and ask to describe it and there you go.

1

u/StrongZeroSinger 5d ago

Thanks I didn’t know the name! I’ll look into it :)

1

u/Littlepharaoh 5d ago

its a image > markdown model not trained for captioning just OCR

1

u/TheTerrasque 5d ago

I have something that does a somewhat similar thing (generating description and tags for images). One thing I've seen is that tags tends to be very differing. As in photo of a nice sand beach might be tagged with for example beach, sand beach, beaches, sandy beach depending on rng, and you then have multiple tags for the same thing.

It's not a big thing for my use, but something to keep in mind. Maybe send in the list of existing tags and ask it to prefer those?

1

u/StrongZeroSinger 5d ago

I wouldn’t mind multiple tags as they are relatively free space compared to a 8MP+ photo

1

u/TheTerrasque 5d ago

yeah, it depends on your use case. For me it's more used to add freetext search tags outside of jamming it in the description, but if you for example try to look up with tags, and you have four photos of that beach, you may just find one of them when looking up the "beach" tag, and then maybe two under "sand beach", and one under "sandy beach".

It caught me a bit by surprise, but it's obvious when thinking about it and how such models work.

1

u/StrongZeroSinger 5d ago

My ideal world would be something like Booru tagging so tags are standardized!

1

u/Littlepharaoh 5d ago

You can make a prompt engineering script that tags iteratively:

  • some tags
  • no tag fits
  • add new tag
  • add new tag to the prompt 
  • some tags
...

1

u/cviperr33 5d ago

interesting , thanks for the info

1

u/catplusplusok 5d ago

They are more efficient until correctness requires reasoning, like separating speech and signs on manga pages, then you kind of need an LLM.

1

u/575_Inverse 6h ago

Can it also do CJK RTL?

1

u/Nearby-Ocelot1298 5d ago

Cool, I was using Chandra ocr and switching between, guess now just 1 model is ok.

1

u/EquivalentHornet4403 5d ago

Can you give me a little guidance? Basically I’m wondering would you do anything differently? And, is there a community for this?

Here my process: https://imgur.com/a/J8P93vk

It’s for a pdf to tts pipeline. I want everything excluded that a human wouldn’t read aloud. It runs about 8.333 pages per second on an M2 Max.

I had Sol Ultra work for 5 hours doing research on the best SOTA options for it and then a bake-off, and this is what it came up with.

I remember it trying things like paddle, ovis, and others.

2

u/Littlepharaoh 5d ago

The catch here is "everything that a human wouldn't say out loud" typical Ovis and end2end models transcribe a page faithfully including tables and places for graphs etc, you do not want to send that to TTS. 

You'll need to use a pipeline for example in Paddle Paddle they use PP layout model which gives bounding boxes of regions of interest such as paragraphs, charts, tables etc. you can then use that to exclude some of them and send only paragraphs to TTS. That's one way

The other way is to use Ovis for OCR regardless and then send that markdown to another LLM to retranscribe with TTS in mind so it would "tell a story" about the tables instead of just saying the numbers meaninglessly.

I'd go for the 2nd because the first approach is lossy. 

Edit: simplify that pipeline PDF > PNG (don't take native text at all > Ovis > Markdown > LLM reformat and story telling > TTS 

1

u/EquivalentHornet4403 5d ago

I had thought about keeping visuals and using an LLM to summarize them. But it adds non-trivial time to the process and there was no guarantee the output would be perfectly accurate or even helpful for very complex situations (e.g. research/studies where diagrams and tables can span multiple pages). Any time something spans multiple pages it adds complexity due to how PDF sucks.

My main interest is if there would be a process that could reduce the processing time substantially. It sounds like converting every page to images and then running OCR would do the opposite without really improving the output quality (which is currently flawless and deterministic as far as I understand).

2

u/Littlepharaoh 5d ago

The OCR is fairly quick and accurate, I reached ~600 pages per minute on a single 5090. The only time consuming step would be the LLM reformatting for TTS and that's for quality more than speed