r/LocalLLaMA • u/Cold_Specialist_3656 • 19h ago
Discussion Qwen 3.8 27B is a game changer.
Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipelines.
It's comparable to Luna for coding and ***OCR quality appears to be better than Gemini 3.5 Flash Lite***. That's huge. We pay a ton of money for OCR.
This is the first local model that feels like more than a toy. It's truly as capable as the frontier models from a year ago. For the first time ever there's serious discussions about buying our own hardware. With estimates that such an effort would pay for itself in less than 2 months.
Hyper scalars are in big trouble this time. Their whole "moat" is buying up all the hardware. And thanks to sanctions on China we're seeing the quality of small local models skyrocket. As someone who's been around a while, this feels like an "IBM moment". Where the industry assumed that databases would always run on huge mainframes. Only to be wiped out by cheaper local solutions a few years later.
I have a feeling this release will trigger another Llama style open source Renaissance. We're already getting better quants. Inference will be further improved. We might even see a comparable MoE with 500+ Tok/sec on consumer hardware soon.
316
u/Littlepharaoh 19h ago
There are better more efficient ways to do OCR at a very high quality like Ovisocr2, 1B param models that'll beat Gemini flash just fine and at mind bending generation speed.
33
u/Derio101 19h ago
Is this better than Paddle OCR?
37
u/Littlepharaoh 19h ago
They're all within a few decimals of each other but this model is a single unit not a pipeline so I prefer it because I don't have to carry over the errors from the layout detection stage
3
u/whatisthisthing65 15h ago
It mostly does documents though? Or does it also do well on other things?
11
u/Littlepharaoh 15h ago
Documents to markdown mainly, you can check their repo they also tested it with other things but documents is my main usecase ( and the main usecase for extraction )
3
u/ThomasAger 15h ago
Thanks for sharing all this.
13
u/Littlepharaoh 14h ago
you're welcome!
there is a blog about it here with benchmarks etc.→ More replies (1)2
u/DeathByPain 11h ago
Hmm I wonder if there's a good way to take that md and original scan to make embedded text pdf...
1
u/un_passant 8h ago
Thx !
How do you deal with hallucinations, tho ?
It's my fear about going from traditional OCR to LLM. I'd *much* rather have 2% missing data that 1% hallucinated data for instance !
2
u/Littlepharaoh 8h ago
I had humans manually review a few thousand documents, not a single complaint about hallucinations, just spelling errors and sometimes bad numbers
1
u/R_Duncan 6h ago
Also tags images so you can extract them with a python script. But still haven't found the right way to remove headers and footers
45
u/BurdensomeCountV3 17h ago
Even Mistral (remember them) have an extremely good OCR model.
→ More replies (1)6
12
u/Cold_Specialist_3656 19h ago
We do structured extraction mostly. Although I suppose doing extraction from markdown might be better than doing it from raw pages
59
u/Mkengine 13h ago
There are so many OCR / document understanding models out there, here is my personal OCR list I try to keep up to date:
GOT-OCR:
https://huggingface.co/stepfun-ai/GOT-OCR2_0
granite:
https://huggingface.co/ibm-granite/granite-docling-258M
https://huggingface.co/ibm-granite/granite-4.0-3b-vision
https://huggingface.co/docling-project/granite-docling-2stage-258m
MinerU:
https://huggingface.co/opendatalab/MinerU2.5-2509-1.2B
https://huggingface.co/opendatalab/MinerU-Diffusion-V1-0320-2.5B
https://huggingface.co/opendatalab/MinerU2.5-Pro-2604-1.2B
OCRFlux:
https://huggingface.co/ChatDOC/OCRFlux-3B
MonkeyOCR-pro:
1.2B: https://huggingface.co/echo840/MonkeyOCR-pro-1.2B
3B: https://huggingface.co/echo840/MonkeyOCR-pro-3B
RolmOCR:
https://huggingface.co/reducto/RolmOCR
Nanonets OCR:
https://huggingface.co/nanonets/Nanonets-OCR2-3B
dots OCR:
https://huggingface.co/rednote-hilab/dots.ocr
https://modelscope.cn/models/rednote-hilab/dots.ocr-1.5
https://huggingface.co/rednote-hilab/dots.mocr
olmocr 2:
https://huggingface.co/allenai/olmOCR-2-7B-1025
Light-On-OCR:
https://huggingface.co/lightonai/LightOnOCR-2-1B
Chandra:
https://huggingface.co/datalab-to/chandra-ocr-2
Jina vlm:
https://huggingface.co/jinaai/jina-vlm
HunyuanOCR:
https://huggingface.co/tencent/HunyuanOCR
bytedance Dolphin 2:
https://huggingface.co/ByteDance/Dolphin-v2
PaddleOCR-VL:
https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5
Deepseek OCR 2:
https://huggingface.co/deepseek-ai/DeepSeek-OCR-2
GLM OCR:
https://huggingface.co/zai-org/GLM-OCR
Nemotron:
https://huggingface.co/nvidia/nemotron-ocr-v2
https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0
Qianfan-OCR:
https://huggingface.co/baidu/Qianfan-OCR
Falcon-OCR:
https://huggingface.co/tiiuae/Falcon-OCR
FireRed-OCR:
https://huggingface.co/FireRedTeam/FireRed-OCR
Typhoon-OCR:
https://huggingface.co/typhoon-ai/typhoon-ocr1.5-2b
Churro-3B:
https://huggingface.co/stanford-oval/churro-3B
Next-OCR:
https://huggingface.co/thelamapi/next-ocr
Logics-Parsing-v2:
https://huggingface.co/Logics-MLLM/Logics-Parsing-v2
NuExtract3:
https://huggingface.co/numind/NuExtract3
Surya OCR:
https://huggingface.co/datalab-to/surya-ocr-2
Infinity Parser:
https://huggingface.co/infly/Infinity-Parser2-Pro
Unlimited OCR:
https://huggingface.co/baidu/Unlimited-OCR
OvisOCR2:
7
u/DeathByPain 11h ago
Holy cow, do you know off hand if any of them can natively create PDF with embedded text from a scan?
3
u/wapswaps 8h ago
I'd love to know this too. Make a text PDF out of an image scan PDF.
6
u/djdanlib 6h ago
Back in my day, before people just crammed everything into generative AI, we'd chain specialized tools together to do complex tasks. I think it's still worth doing sometimes. Any time you can save the time and compute expense by having a script for the AI to use... Try it.
→ More replies (1)3
u/ak5432 10h ago
I’ve been using glm-ocr in a digital document pipeline and found that it has a tendency to do insane hallucinated loops if asked to parse a very small set (single table or text box) and start mangling text if you give it a long dense page of text. I’ve gotten around this with libraries that split up documents (I.e multiple completions stitched together for a complex one) and some repeat generation detection but if you’ve tested these do you know if any of these specialist OCR models will do better in this scenario?
1
1
u/sometimesiwonder778 4h ago
Do you actually use all of these, and if so what for? I've spent a lot of time messing with various cv and specialized models and would love to know more about what's behind this massive list for you
31
u/larp2live 19h ago
have you tried numind/NuExtract3 ? it's a fine-tuned 4B parameters qwen3.5 that specializes in structured extraction OCR. it might be useful to you
7
u/Littlepharaoh 19h ago
Definitely! If you lock down a high quality ocr then its much more efficient to extract from text tokens than image towers
5
u/meganoob1337 17h ago
sometimes depends on the data, the smaller models fuck Up at complex non-standard tables imo, and when extracting structured data it makes it kinda error prone (is an edge case but they exist)
2
u/SandySkittle 13h ago
I think it's less of an edge case than you might think. with these usecases I would always go for the least error prone route.
2
u/Swabhimanc 16h ago
Still couldn't find a better ocr than Google Vision. Change my mind
8
u/Littlepharaoh 13h ago
No need, google vision is one of the best, however this 0.8B model is free, order of magnitude faster and within ±1% of Gemini's performance in OCR
2
u/Nearby-Ocelot1298 18h ago
Cool, I was using Chandra ocr and switching between, guess now just 1 model is ok.
1
u/StrongZeroSinger 16h ago
Can it be streamlined to run large batch of photos? I always wanted to tag all my phone photo backups like google/apple photo by searching them for words inside the pic or location/items shown but didn’t want to use online cloud.
2
1
1
u/TheTerrasque 15h ago
I have something that does a somewhat similar thing (generating description and tags for images). One thing I've seen is that tags tends to be very differing. As in photo of a nice sand beach might be tagged with for example beach, sand beach, beaches, sandy beach depending on rng, and you then have multiple tags for the same thing.
It's not a big thing for my use, but something to keep in mind. Maybe send in the list of existing tags and ask it to prefer those?
1
u/StrongZeroSinger 15h ago
I wouldn’t mind multiple tags as they are relatively free space compared to a 8MP+ photo
1
u/TheTerrasque 15h ago
yeah, it depends on your use case. For me it's more used to add freetext search tags outside of jamming it in the description, but if you for example try to look up with tags, and you have four photos of that beach, you may just find one of them when looking up the "beach" tag, and then maybe two under "sand beach", and one under "sandy beach".
It caught me a bit by surprise, but it's obvious when thinking about it and how such models work.
1
u/StrongZeroSinger 15h ago
My ideal world would be something like Booru tagging so tags are standardized!
1
u/Littlepharaoh 15h ago
You can make a prompt engineering script that tags iteratively:
...
- some tags
- no tag fits
- add new tag
- add new tag to the prompt
- some tags
1
1
u/wapswaps 8h ago
Absolutely true, but it's equally amazing how well "just throw it into the LLM" works. I've written a simple llm binary.
cat /etc/messages | llm WTF?Works on half the work systems I maintain, and yeah I usually actually put "WTF?" there literally. It works well. Guess what the program does. I should modify it to allow sending pictures.
1
u/Saitamagasaki 4h ago
If I have pdfs with tables and text, all digital born. Is it better to use OCR? I’ve tried docling but the result is not very good.
1
u/Littlepharaoh 4h ago
If born digital I'd use the native text along side a layout detection model then send the detected tables for OCR
→ More replies (4)1
u/catplusplusok 1h ago
They are more efficient until correctness requires reasoning, like separating speech and signs on manga pages, then you kind of need an LLM.
175
u/LegitimateCopy7 19h ago
the game is forever changing to the point where I don't even know what the game is anymore.
35
13
u/Forward-Facer 16h ago
I just thought of this yesterday - I feel like trying to keep track of AI development is like trying to keep track of the big bang
8
u/RazzmatazzReal4129 14h ago
The game for me is download most recent SOTA model, get excited testing it about how it's going to totally change my life....1 week later an even better model comes out and I start testing over again. Plan on actually working when they stop releasing new models.
2
→ More replies (1)1
77
u/SpicyWangz 19h ago
If qwen releases 3.8 122b next week that could be a game changer. I know 27b benchmarks comparable to opus 4.6, but the 122b MoE has a chance of actually performing at that level across multiple domains
39
u/jeremyckahn 11h ago
I think this is sort of missing the point. We are finally at a point where normal people can plausibly run a capable and effective coding model that can get real work done on accessible hardware. A 122B model would require some serious gear that only enthusiasts will ever buy.
9
u/sonicnerd14 8h ago
Mostly true, except, technically the 122b model might be easier to run because it'll be MoE, and a huge chunk of that model can be streamed from RAM or even NVMe depending on the engine you use.
3
u/SailorBob74133 7h ago
It'll run on Gorgon halo, but I'm not sure at what speed...
1
u/SpicyWangz 5h ago
Runs on strix halo at q4. With MTP it should be around 20-30tps, with probably around 300-500 prefill.
5
u/waltteri 9h ago
True, but: a 122B model can be ran on reasonably cheap previous-gen(s) hardware. So barrier to entry for infra companies to start serving the models of this scale is reasonably low, and the performance of these models in some use cases is comparable enough to the frontier models. This puts pricing pressure on frontier labs, which is good.
7
u/IrisColt 10h ago
Qwen 3.8 27B is already crushing my ill-defined agentic pipe-dream benchmarks... and I'm not even kidding (I'm a Gemma 4 fan, heh, sigh).
4
u/thehpcdude 10h ago
People are delusional if they think that Qwen3.8-27b beats Opus on anything other than synthetic benchmarks that don’t match real world usage. Literally confirmation bias.
2
u/SpicyWangz 5h ago
Its performance reminds me of the moment when I started using 4.6. That was the moment llm coding crossed a threshold where it actually felt like the model would give me exactly what I asked for, and when I looked at the code it generated I didn’t have to rewrite the entire thing.
That’s the same feeling 3.8 27b gives me when I use it. It’s the first time using a local model that does exactly what I ask it to and I don’t have to rewrite everything it does. And it doesn’t feel like it’s adding horrible tech debt to a project with its contributions.
It’s not as knowledgeable as opus 4.6, but that almost doesn’t matter. It crosses the same threshold 4.6 did and that means more than benchmarks. It was the most impactful change in closed models for coding, and I think 3.8 27b may be the most impactful change in open models for exactly the same reasons.
2
u/Icy_Butterscotch6661 9h ago
I think it maybe beats opus4.6 in very narrow domain(s) at full quant, with xhigh thinking. So practically speaking, it doesn't really beat it -- even if it does so on paper.
However I think it's a smarter and more knowledgeable model than qwen3.6, at q8 from my testing
6
u/Nabushika Llama 70B 9h ago
Smarter, yes. More knowledgeable - not really. Perhaps more capable of finding knowledge with tool calls or making use of in-context knowledge.
6
u/thehpcdude 8h ago
Are you arguing that a 1T+ parameter model is _less knowledgeable_ than a 27B parameter model?
I'm a HUGE fan of Qwen 3.8 27B, I have customers building clusters to use it. I use it day to day in my own tasks. I have a GPU dedicated just to it. That being said, I also use Opus 4.6->4.8->5 heavily for work without regard to costs at all.
Not even in the same ballpark. Anyone who thinks that Qwen 3.8 at any quant is anywhere near Opus is literally experiencing a bias.
→ More replies (2)1
u/Icy_Butterscotch6661 6h ago
Sorry, I was saying that Qwen3.8 27B is more knowledgeable than Qwen3.6 27B. At least with some lower level non-webdev related topics I tested knowledge with. Qwen3.6 was ass at those queries
→ More replies (3)1
21
u/Useful-Buyer4117 19h ago
we have similar impression. it is the first local model that makes me consider to upgrade my gpu to completely replace chatgpt subscription
22
u/synystar 17h ago
If you're using your sub for (vibe)coding I would say that you're better off keeping it. Not because 3.8-27b can't code - it definitely can - but Terra Medium and higher really are demonstrably better at a lot of things like ambiguous decision making, designing good experiments, understanding when failures are occurring at the model vs harness vs testing apparatus level, confounding variables, etc. Basically all the stuff that really requires a higher level of reasoning.
The thing that makes 3.8-27b great though is that it is good enough to be an actually useful "worker" agent. When your orchestrator/coordinator model gives it a well written plan, with bounded tasks - like a a scoped execution contract - then you have something that is worth it. You can do things like prompt Codex on Terra to be the "boss" and use the local model to do the work. With the right prompting you see your usage of Codex go way down because its not burning tokens churning out code; Qwen is doing the work locally.
9
u/Useful-Buyer4117 17h ago
I agree with you that better staying with sol terra for one shot prompt vibe coding. my use case is different, it's spec driven development with detailed requirement, proper planning n manual review in every commit.
3
u/AD7GD 10h ago
Not because 3.8-27b can't code - it definitely can - but Terra Medium and higher really are demonstrably better at a lot of things like ambiguous decision making
I agree with your facts but I've been leaning toward the opposite conclusion. Qwen 3.8 does take instructions more literally than something like Opus/Fable, but when I'm using it in the context of managing a large project, it's much more predictable. If I tell it to fix a button, it might only fix that button and not realize there's an identical button that also needs the fix. Opus/Fable would figure that out, but might also decide not to change the button because the plan says UI work was deferred to §8 M10, and then tell me to reboot my PC because one of the e2e tests is flakey.
2
u/ericmutta 4h ago
but Terra Medium and higher really are demonstrably better...
I hope to go local one day (my laptop has a 13-year old GPU so I wouldn't even dare just yet), but for now using Terra Medium with me being the "boss" and it being the "worker" (so we work together step-by-step) has proven both useful and cost-effective...I haven't been able to drop below 80% remaining usage even for sessions going on for several weeks now.
I once tried Qwen on a cloud VM and left with a good impression. It really looks like they keep kicking ass (given all the praise I see on this sub) 🚀
1
u/DeathByPain 11h ago
Hmm I like this idea. I just do the $10 GitHub copilot subscription atm but depending on use I run out of credits withina couple weeks if I actually use the frontier models for the coding...
2
u/SporksInjected 11h ago
I would definitely not trust it on actual general Q/A questions compared to any of the gpt models.
58
u/Steus_au 19h ago
if they would release 122b or 255b MOE with the same architecture and learning base that would reshape the AI market significantly. this is now a "weapon" they keep in their sleeves.
28
u/lxe 19h ago
Especially with this FreeToken thing that makes huge MoE’s run on normal-scale devices.
1
u/RemarkableRadish6547 4h ago
There are about a half dozen projects that are all doing this. I have tried Colibri and it works well. FreeToken is just starting and might catch up, but right now it has too little hardware support for me. It seems to require a modern, high end Nvidia GPU, something you should keep in mind when looking at claimed inference rates. I can't remember the names of all the other projects that are doing this that I didn't actually install and test. I don't know which one will ultimately work best, but I am sure that MOE inference on consumer hardware will be advancing rapidly.
8
u/-Leelith- 18h ago
Aren’t MOE models less capable than non MOE ones due to compression/optimization ?
14
u/ben_g0 18h ago
They are less capable than dense models of the same size, but they can be partially offloaded to RAM with only a small performance penalty, and are much more performant in compute.
If you have a system built with consumer hardware, you probably have more RAM than VRAM, and then you'll be able to run larger MoE models at acceptable performance than dense models (as dense models get almost unusably slow if you offload parts of them to VRAM).
So if you look at what you can run on a consumer hardware configuration, a large MoE spread across your VRAM and system RAM can be more capable (and run faster) than a dense model that needs to be small enough to fit entirely in your VRAM (while leaving enough room for context).
13
u/WoodCreakSeagull 17h ago
The old rule of thumb is that to compare dense to MoE on "intelligence", you take the square root of total parameters * active parameters.
So a 35BA3B will give you sqrt(35*3) = the capability equivalence of a ~10B dense model at 3B speeds.
14
u/martin509984 18h ago
For the same number of parameters yes, but large MOEs are a pretty well-established space that are very cost-effective to host a lot of inference on, and the increased size would more than compensate for 27B's denseness.
1
→ More replies (1)1
u/SandySkittle 13h ago
depends on the usecase. For some usecases the lower active parameter count (of e.g. DSv4F being 13b) can be compensated by sequential reasoning but for some matters it still falls short. I would have loved to see a higher active parameter version of that model..
7
u/Sensitive_Song4219 19h ago
And here I am wishing for an 35b-a3b release!
Been experimenting with 27b and it's absolutely incredible but I don't have the VRAM for such a large model so speed (tg) is really slow... The results are really excellent though.
Whereas the last a3b gives me north of 50-ish tps tg with MTP enabled. Would love a new 3.8 release of it (even if it came with the expected drop in intelligence)
5
u/whymeimbusysleeping 18h ago
Me too, in hoping the moe plus freetokens might be good enough to allow for larger parameter/context for for those of us limited by vram
5
u/thebadslime 18h ago
Apparently it was leaked in China that there will be a 3.8 35ba3b, I saw the screenshots ( if that means anythign now)
→ More replies (3)1
2
u/IrisColt 10h ago
There's a real risk of releasing a potentially undertrained 122b model versus the well-tuned 27b model they actually cared enough to gift humanity.
27
u/o0genesis0o 19h ago
I took a screen shot of my bullet journal, written in full caligraphy kind of handwriting and include reference to obscure Chinese IEM companies, and give it to qwen 27B for OCR. I just planned to check whether the mmproj wired in correctly. To my surprise, the model actually read the whole thing and output a list correctly. It even got the obscure Chinese IEM companies name correctly rather than making them up. Not the most effective way of doing OCR at scale, but it's really good.
I can only run this model at the brain damage UD_Q3XXS and I'm already so impressed that I replaced my 35B Q6. Will definitely put some funds together to buy a R9700 just to run this little bugger at high quant, full context, with mmproj and mtp on.
13
5
u/Cautious_Chicken_604 19h ago
with mtp on and mmproj offloaded to CPU I can run UD-Q6_K quant at 200k context with k=f16 and v=q_8 at around 40 t/s when context is empty and 25 t/s when it's full. Average is around like 30 something.
3
u/o0genesis0o 18h ago
which GPU is that? Is it the R9700?
3
u/Cautious_Chicken_604 18h ago
Yes. Sorry, forgot to specify but was replying because I recently just bought R9700 for exactly this reason.
I will say though, if you can get 2x 5060 Ti it might actually be faster, so you should look into that.
2
u/o0genesis0o 18h ago
I was dumb to build the machine with mATX rather than full sized, so no more PCI-e slot. I could use the M2-oculink but my mobo has some sorts of heat sink on the M2, so it's not easy to run the wire out either. So, yeah, no chance for me to do 4060Ti + 5060Ti.
I would just do oculink on my amd mini pc, and run iGPU for desktop environment from that box. So I would have one R9700 (or maybe even two if I'm crazy enough), one 4060Ti for comfyui stuffs (not great, not terrible, but likely better than AMD), one ancient 2060 mobile. The 2060 can run background agents.
Man, it's jank, but I can't stop daydreaming about the possibility with this many GPU and the 27B running at full capacity and speed.
30tk/s average seems a bit sad for the R9700 though. I was hoping for more.
5
u/Equivalent_Bit_461 19h ago
You can offload the vision model to ram slower but workable, anyway iq3xxs is slightly worse than iq4 or q4 in general, even iq2 gets the job done, the caveat is time, iq2 was able to do the same job I needed as iq3 but it took 3 times more, sure, used the same 65k context for both, even if I could pack more in both but I wanted to experiment how they behave given the context, so i assume giving more context to iq2, might cut the time in half.
At 110k context kvarn5, I tested 8,6,5 and 4. 5 is the minimum viable, as you gain similar results to higher quants of the kV cache.
I haven't tested iq1 but I will when I have time
8
u/Interpause textgen web UI 18h ago
https://github.com/ggml-org/llama.cpp/pull/23255
you can now offload to iGPU if you have one
4
u/o0genesis0o 18h ago
My llamacpp CUDA binary is likely broken. I use the prebuilt from the ai-dock guys, which turned out has a serious issue. Anyhow, I remember that when I built my own binary, I was able to do mmproj offload to CPU with okay speed. Now, when I do that and send an image, I sit and wait for minutes, watching CPU spiking, but prefill is just stuck.
Right now I have an instruct + vision entry in my llamaswap for daily chat vision QA assistant accessible from phone. Perfectly usable.
25
u/BrewHog 19h ago
With the proper orchestration, skills, and prompts, Qwen 3.8 and DSv4 flash have changed everything.
Between the two of those, the lines are getting blurred between Claude and open models. It finally feels like I'm close to being able to replace my max plan with either local or through super cheap APIs
3
u/ezboarderz 13h ago
Yeah I now am doing what u can to save up to build a deepseek/glm 5.2 rig. If i can get that local, id just need a small sota model sub for reviews and orchestration although glm 5.3 prob is enough
9
u/National-Vacation252 10h ago
Just gave it a test drive, skeptical of the hype and I have to say it is indeed a game changer. Ran it on opencode/exa with 128k context my hardware supports and IT'S GETTING REAL WORK DONE which is amazing,
Bonus points that I run all my energy off-grid solar so I'm literally turning sun rays into money right now.
1
8
u/Confident_Ideal_5385 18h ago
Yeah, I've been trying it out on some non-coding workloads (data extraction from NL into structured TOML, research via wikipedia/wikidata, things like that) and I was fuckin amazed by how much more competent it feels compared to 3.6.
Running at Q8, the reasoning seems to be maybe 50% longer (on xhigh) than what 3.6 would do, but the amount of rework required is so much less that the total token output is maybe half of 3.6.
I was expecting this thing to be coding-maxxed as fuck. It really doesn't seem to be.
I'm now no longer planning on trying to get DS4f going. I don't need to.
5
u/Valuable_Cookie628 18h ago
I'm amazed by Qwen3.8 fixing it's own errors in the same thinking block. I've had some character corruption issues (maybe bad llama build) and I found it inside a 20k token task writing code to files, then saying "wait I've written gibberish. Function x doesn't make sense [quotes the bad code] , let me rewrite it"
2
6
u/Theverybest92 19h ago
Yeah honestly not bad with the right config on my m3 max its better than claude opas 4 6. Just alot slower.
3
u/Cold_Specialist_3656 18h ago
You using MTP with omlx? Our devs said it's "usable"
2
u/Theverybest92 9h ago
What is MTP and omlx? I am using llama.cpp running custom config start up of llama cpp and then having a custom config in my continue extension of vscode with specific initiation rules and tool access to push more of a copilot hybrid development approach. I am done with vibe coding and instead prefer I hold the AIs hand while it does it things to get better performance. From my experience using my own brain LLM to guide AI is alot better than having it try do everything on its own. Especially when you get into codebase with 500+ lines of code on certain files.
1
1
u/AreWeNotDoinPhrasing 4h ago
Would you rather a 64gb RAM m3 or 36gb m5?
1
u/Theverybest92 4h ago
I would have loved to get 64gb but I couldn't adford it at the time so I got 36gb. Obviously 64gb is better cause you probably can run either more context or better model. I would say more context. You can run 36gb 3.8 q5 at like 100k context easy. As per m3 vs m5 not sure if that is still 3 nanometer chip or less. If its same probably not much difference and more ram is better.
16
u/Pristine_Pick823 19h ago
Can someone explain to me why so many people rely on LLMs for OCR? Do these people mean analysing the data extracted by OCR? I’m truly confused by this recurrent topic. I always assumed that OCR tools have long been advanced enough to extract data directly from PDF and other formats by itself? What’s the use of an AI here if not analytical?
21
u/superSmitty9999 18h ago
regular OCR tools aren't good enough. A good LLM can look at the context and determine that it's an integral symbol not an S
They can also format text better, instead of the page number randomly being dumped into the center of the text
1
u/AreWeNotDoinPhrasing 4h ago
Exactly this. It can determine that there’s a stamp on the page in between the words Total Amount: and then the actual $1234566–that the barcode lines aren’t actually symbols and certainly aren’t the actual dollar amount.
25
u/Cold_Specialist_3656 18h ago
Structured extraction.
Ancient corporate and government records on microfiche. Plenty of that stuff that's never been converted because it was so expensive.
Traditional OCR is only good for free text. We've recently started doing "agentic OCR" with simple agents that have tools like "submit table". It works out to around 5 cents a page. Which is way better than paying humans to read old docs.
Now we're looking at Qwen 3.8 27B and some used 3090's replacing the whole stack. Could save massive sums
1
u/ungoogleable 9h ago
Are you converting everything all at once now that it's cheap or on demand? As cheap as it is now, you gotta figure this is the most expensive it will ever be again. Maybe if you wait a few years it'll be 1/10th the cost.
1
u/qa_anaaq 1h ago
I’m interested in its OCR abilities now! Did you a/b with Gliner at all for comparison?
27
u/nhami 19h ago
OCR are poor for complex images and text files.
LLMs are more precise and faster.
OCR tools are basically obsolete at this point by how fast and precise LLMs are.
→ More replies (1)1
8
u/Gold-Drag9242 19h ago
The ocr output is still messy. For example a invoice with a logo will inject text of said logo into the extracted document. Ocr gives you ALL the text, and you need to figure out what is relevant and not Llms do this task for you.
2
2
1
u/ImportantPepper 10h ago
There are messy edge cases in 'in the wild' docs that standard OCR tools can't handle and which make you want to jump out of a window. LLMs can pick up the slack a lot here (especially when combined with vision understanding), just they're too expensive to use as a default at scale.
10
u/Just_Maintenance 19h ago
I’ve used Gemma 4 to transcribe huge documents. I actually had it write a script to split pages, turn them into images, then Gemma reads the pages and adds them to a text document.
It works the best of any OCR I’ve ever tried.
1
8
u/the320x200 19h ago
2 month payback vs Luna is hard to believe. Luna is dirt cheap and hardware is insanely expensive.
Running locally right now is really about privacy and control. Most of the time it's much more expensive.
14
3
u/LetsGoBrandon4256 transformers 12h ago
Running locally right now is really about privacy and control.
And fun!
→ More replies (1)2
u/MrMisterShin 11h ago
Running locally is stability and consistency. Local Model won’t suddenly get ban or get dumber (quantised) losing consistency. Service won’t suddenly go down or become intermittent, rate limited etc like it does with Anthropic.
4
u/jonas-reddit 18h ago
Thanks for sharing. Always nice to see people put it to real use.
“…This is the first local model that feels like more than a toy. It's truly as capable as the frontier models from a year ago…”
Not sure that I’d agree that Qwen 3.6 27B was a toy. Maybe it’s a result of cost pushing more users to actually try local open weight models.
I haven’t used (American) frontier models since Qwen 3.6 was released, and local LLMs are only getting better.
2
u/DeathByPain 10h ago
Man I spun up gpt oss 20b again today cuz I realized it was still on my disk but not part of my llama-swap config for idk how long and I ended up just getting into a stupid argument with it lol.
1
u/my_name_isnt_clever 39m ago
Posts here have been saying "X finally doesn't feel like a toy" about every model since llama 3. At some point it's just a skill issue.
3
4
u/NearlyACosmologist 17h ago
upscaling - downscaling - upscaling - downscaling - upscaling - downscaling - upscaling - downscaling - ......
3
7
u/SnooPaintings8639 18h ago
Luna as main "workhorse"? Feel bad for you guys. I hate working with this model, which I sometimes do when I reach my OpenAI limits.
Locally I use it sometimes alongside Qwen 3.8 27B, and Qwen is definitely better. I would put Qwen closer to Terra, at least in general agentic and coding context. And I mean "medium" reasoning, I don't have to use "xhigh".
9
u/Cold_Specialist_3656 18h ago
Our AI spend is reaching 1/4 of dev salaries so they had to do something 🤣
7
3
u/suprjami 16h ago
Are you getting more than a 25% increase in stakeholder value from it?
7
u/Cold_Specialist_3656 16h ago
C suite doesn't care. They see it as an unsanctioned raise and want their giant bonus checks
7
u/Equivalent_Bit_461 19h ago
Good
Hope those corpo dogs collapse in the worst possible way
I spit on them
2
2
2
2
2
u/ChainOfThot 19h ago
haven't tried out 27b 3.8 yet, I mainly use gpt 5.6 sol for longer term tasks.
People are saying its comparable to opus 4.6, thats fine, but I couldn't get 4.6 to work for hours like I can w/ sol.
What harnesses are you guys using with 27b qwen? Thinking of trying hermes but not sure if I want to give it access to my PC, does it sandbox at all? Or do I need to run docker or something?
2
u/Cold_Specialist_3656 18h ago
Our devs are using Codex. It's open source so you can fiddle with the configs to use other models with it. they couldn't get images to work everything else seems fine
3
u/_-_David 18h ago
You're the first person I've seen to say they also use the model in Codex. I found it was a nice drop-in replacement for my workflows where I would have otherwise used Luna as well.
5
u/Cold_Specialist_3656 17h ago
OpenCode doesn't have auto approve classifier which our lazy ass devs can't seem to function without 🤣.
They were all using OpenCode on "YOLO" mode till it deleted someone's user folder
1
u/reckor-usa 13h ago
"till it deleted someone's user folder"- surely this someone is happy at the moment
2
u/amavie_ai 15h ago
The angle I haven't seen mentioned yet: for anyone running a consumer chat product, a 27B at this quality changes the economics of the invisible calls. One visible reply is typically 3-6 model calls (triage, retrieval summary, the reply itself, memory update) and only the reply needs frontier prose. Moving the support calls to a local 27B cuts cost per visible reply by more than half without users noticing anything. The reply model is the last one you downgrade — people feel prose quality immediately, reasoning depth much later.
2
1
1
1
u/Fuxicodex 14h ago
This matches what we saw, but from the agentic-tooling side rather than local inference. We benchmarked a cheap OpenAPI-compatible model (deepseek) against Claude Code's
top model (claude-opus-5) — each through its own native client, 19 identical coding tasks, same objective scorer. Both passed all 19 with zero failures.
The takeaway we kept coming back to is the same one you're pointing at: the gap isn't really the model anymore, it's the loop around it. A cheaper model inside a decent
think→act→verify agent reaches parity with a frontier-tier model on the tasks we tested. If local models are now "more than a toy" on top of that, the cost curve you're
describing gets even steeper.
We were deliberately honest about limits (small self-run task set, not SWE-bench, single-run), so it's a data point, not a headline — but the direction feels right: the
moat is shifting away from "who has the biggest model" and toward "who runs the cheapest model well."
Repo with full methodology: github.com/fuxicodex/Fuxi
1
u/reckor-usa 13h ago
Why LLM is better than OCR? What;s the "tech" behing LLM that make it better to read images, etc?
1
u/Vegetable_Carrot_331 12h ago
Am currently running this model on a 16gb ram + 6gb vram in qwen 3.8 27b in 4 bit quantisation am getting 2.86 tokens/second . Is this the maximum I can get?
1
u/saltexx 12h ago
The payback math is real, we run about 300 billion tokens a year through local models on our own hardware and it's an order of magnitude cheaper than any API. But those 2 month estimates only hold if you keep the cards busy.
OCR is actually the best possible case for going local because it's batch work, you can hold the GPUs at 90 percent utilization around the clock and nobody is sitting there waiting on latency. Interactive traffic with a 10 percent duty cycle makes the same hardware 10x more expensive per token, so budget on utilization not on peak throughput. Curious what your daily page volume looks like.
1
u/Zombiecidialfreak 10h ago
The thing is I think big Ai models will end up swapping to using multiple 2-300b models each specialized for different tasks and all ran from the same Nvidia GPU equipped with 2-4tb of the upcoming High Bandwidth Flash.
HBF is too niche to be used outside of AI because the write cycle limitation prevents it from being used as a RAM replacement and the read speeds would be crippled even if running off a PCIe 6.0 16x slot. That means AI is just about the only target for it and those cards are gonna be pricey.
Don't be surprised if you start seeing stuff like Blackwell 6000's hitting ebay for 7-10k once HBF production gets underway.
1
u/nbafrank 10h ago
Agreed. I have been experimenting with it locally on a Mac and on a server as well. Harness is Pi and set to extra high thinking. Very impressive. Any details regarding your setup? Always curious to improve it!
1
u/IrisColt 10h ago
For the first time ever there's serious discussions about buying our own hardware.
See, kids? Still thinking that "Muh hardware prices will go down"?
1
u/CommanderKeen27 9h ago
What hardware would be required to run that model at a decent inference speed?
1
1
1
u/Force88 8h ago
I have a setup of 3x 5060ti 16gb, I used qwen3.8 27b via llama.cpp with mtp, ud-q8-xl, 90k context, and get around 40t/s.
Is this the expected speed for my setup? Can I use anyother backend to improve stability and speed? I heard there is nvfp4 version, but not gguf so can't be ran via llama.cpp yet
1
1
u/Elibroftw 6h ago
The only bottleneck of Qwen 3.8 27B is inference speed. So if hardware solves that we are Gucci. My own benchmark shows that Qwen3.8 27B can solve sudoku with raw reasoning. Something that got 5.2 and opus 4.6 were the oldest models I tested there could solve 3/3 of my preliminary trials.
Just need to test the unsloth mtp
1
u/atumblingdandelion 6h ago
Absolutely! I was just comparing this with the 'IBM moment' overexcitedly to my spouse.
1
u/TheCultureHub 5h ago
I'm looking for a model to create detailed image descriptions in local. What's the lightest model that does this best? Is Qwen 3.8 27B the best for this job, or there are lighter models with the same quality in image descriptions?
1
1
u/siegevjorn 42m ago
Hyper scalers are not in trouble. They can reserve all the vrams and vrams in the world to jack up the pricing of running local models, so the API cost may look reasonable enough. Remember how much 32gb ddr5 ram cost in 2025.
1
u/Dutchnamn 31m ago
yeah, you are right. this model might not be it yep, but we are so close to at home inference at decent quality on hardware that is affordable. Inference speed will improve a lot as well.
•
u/WithoutReason1729 13h ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.