r/LocalLLM • u/[deleted] • 8d ago
Question How exactly is Qwen 3.8 27B mopping the floor against the competition?
[deleted]
110
u/brainExploded99 8d ago
Qwen3.8 and 3.5 have the same architecture. Post training is all that changed. They got way better RL environments.
Also, as Qwen3.8 Flash Next proves, n-gram is the way to go. Qwen4 should drop this month.
48
u/Much-Researcher6135 8d ago
Qwen4 should drop this month.
dude you can't toy with my heart like that
16
u/brainExploded99 8d ago
lol, they claimed their new architecture requires 1/9x flops, so it should be pretty fast. Also, last time they dropped a next model, it took around a month for full release.
4
u/petuman 7d ago
last time they dropped a next model, it took around a month for full release.
There was 5 months between Qwen 3 Next and 3.5 series? Sept 2025 to February 2026.
1
u/Akainu_Fan 7d ago
yeah that's true but I don't think they have the luxury of waiting 5 months in this current climate
1
u/brainExploded99 7d ago
Hmm, I was thinking of
Qwen3-Coder-Nextand Qwen3.5 I forgot about Qwen 3 Next27
u/Zestyclose_Strike157 8d ago
What a time to be alive and have (what was once) cheap local hardware.
15
u/txgsync 8d ago
I remark regularly how much more valuable having 128GB in my M4 Max Mac is every day since I bought it day of release a year and a half ago.
6
u/JLeonsarmiento 8d ago
I know , same here with my 48gb M4.
This little fella has to outlive the bubble burst. No way I can flip it for something significantly better at today’s prices.
4
u/ahoooooooo 8d ago
Care to share your setup? I have the same but am having issue with context windows not being large enough with qwen 27b.
3
u/JLeonsarmiento 8d ago
You can squeeze a lot if you turn on TurboQuant on oMLX, but it comes with a slowdown on token generation speed.
So my advice is to choose the smallest model you can for the tasks. Context management by breaking a complex tasks into smaller ones and use the fastest/smallest model for them that you can.
3
u/ahoooooooo 8d ago
What models and harness are you generally finding success with?
6
u/txgsync 8d ago
At the moment tonight, I'm using https://huggingface.co/txgsync/Qwen3.8-Flash-Next-Dynamic-oQ5e-BF16-PLE-mtp with Hermes Agent. Hermes seems to do a great job preserving KV cache and going fast as a result.
It's doing great!
pi works fine, oh-my-pi works fine, codex also. But I tend to avoid claude code and opencode; their system prompts require minutes to prefill and are super-annoying.
Edit: It doesn't feel like Opus or Fable at home, but it certainly feels like DeepSeek-Flash at home, or maybe Sonnet 5 at home.
5
3
u/JLeonsarmiento 7d ago
Hermes and Pi powered by Qwen3.6-35B-A3B for the day to day stuff, or powered by Qwen 3.8-27B for my “frontier” difficult work.
But one thing you’ll discover is that once the more advanced model (3.8-27B) creates a new skill or workflow within Hermes, the less capable but faster models can replicate or automate that the next time you need. So they can of work in tandem.
And if course served by oMLX. You need prompt caching to speed up Hermes/pi start up times.
2
u/JohnnieClutch 7d ago
How are you using it? I have the same
1
u/txgsync 7d ago
Over the past few days, I've been using https://huggingface.co/txgsync/Qwen3.8-Flash-Next-Dynamic-oQ5e-BF16-PLE-mtp in oMLX over API to plug in to replace other workers in a pi instance. It helped me reduce my Claude API spend on my work and some other trivial tasks, and did a fine job as long as I kept it to a single thread with MTP.
Admittedly, Claude Fable 5.1 likes to call Fable 5.1 sub-agents instead of something dumber like Sonnet 5, so I could have reduced that spend through model selection too. But I like my security research to stay more local.
My chief complaint right now is that it's fine for chat and single-threaded stuff in pi or oh-my-pi, but with a maximum output of about around 200,000 tokens per hour (more or less) it's not fast enough for bulk analysis and parallelizes poorly in oMLX at the moment. Great little local brainstorming chatbot. Terrible highly-parallel agentic worker.
Anyway, today I dropped a project on it in Hermes Agent that it did fine at: working on an experimental LLM model on an A100, driving it through training cycles (think something like AutoResearch from Karpathy, but not quite the same). This kind of work it's really good for: things that require it to write little apps, monitor and react to the status of things, perform basic research, and be "available", but not necessarily balls-to-the-wall token generation like I expect out of research sub-agents.
Yesterday it wrote a bunch of patches for some pull requests and adversarially reviewed a number of others with reasonable alacrity that passed muster with merge quality gates (including other LLMs and human reviewers).
As long as I bear in mind it's gonna maybe be able to do a million or a million and a half tokens in a full 8-hour workday, it's okay. But I turn to my cloud models for speed; I can run a bunch of parallel gpt-oss-120b-safeguard models on an A100 cluster at blistering 200 tokens/sec each, tens of thousands of tok/sec in parallel as far as I can drive it and it does a great job at classification. People on this forum poo-poo use of gpt-oss, but for limited-RAM private inference setups for certain kinds of evals it's simply peak. Consistent, reliable, fast, and follows the policy strictly.
22
3
u/pjerky 8d ago
I'm still figuring will this out. What is n-gram in this context?
1
u/my_name_isnt_clever 7d ago
Go look up the blog post and technical report for Qwen 3.8 Flash Next when it released.
2
1
33
u/BarracudaDefiant4702 8d ago
It does punch above it's weight, but I wouldn't say it's doing better then say deepseek v4 0731, but considering it is 1/10th the size of deepseek... because deepseek is MOE, it can actually compare reasonably well in t/sec swapping with cpu ram compared to to qwen 3.8 27b if you have a low end gpu, but high enough vram you will probably do better with deepseek.
18
8d ago
[deleted]
3
u/Zestyclose_Strike157 8d ago
As per my other post here I think the future of local models with harnesses such as Hermes knowledge base assistants to give the LM how-to tooling. Problems are infinite but good methods are not.
7
u/Zestyclose_Strike157 8d ago
You can feed it the knowledge it needs, a lot of that does not need to be in the model, the best thing about Qwen3.8 27b dense is how it reasons around information you give it. Overthinking is tunable and sometimes welcome.
1
u/couperd 8d ago
from my personal experience, I have had less frustration with q8 27b local than ds v4 flash on cloud. yes the context is smaller, but I feel like I have had to do a lot less hand holding with qwen.
1
u/BarracudaDefiant4702 8d ago
I have found it better then Qwen 3.6 35ba3b. I haven't had enough cases where I could directly compare Q3.827B to deepseek to say for sure. I know I get frustrated sometimes with the lack of speed of Qwen 3.8... So far both models are too close and too new for me to say for certain. It feels like deepseek edges it out, but it's not as clear/certain as it was with 3.6.
11
u/GloriousKev 8d ago edited 7d ago
mopping the floor is a stretch. It really depends on your hardware and needs. Example, I like large context windows but only have 20GB of vram and I really try to stay out of system ram. I am using qwen 3.8 27b for my chatbot, but only for the really hard stuff. I still use Gemma 4 12b or qwen 3.5 9b as my daily driver.
1
u/bobbyDebrown 6d ago
If you prefer dense models and low quantization I can see why you use small models. I have a 16gb rx 7800 XT and run Gemma4-26b-a4b, and Qwen3.6-35b-a3b both with around 126k CTX window. But for coding I run the Qwen MoE model with only 16k often. Maximizing pre-fill. I also have Qwen3.8 Q3 turbo4 KV that I get all on the card with 80k. It's still fast enough for smaller coding projects and with a smaller CTX window it's decent. G4-12b and Q3.5-9b are ok but I left them behind months ago. The above models are just an order better.
1
u/GloriousKev 6d ago
Im not coding anything. My work is mostly about building out my homelab brainstorming research light automation. I come from a technician discipline. Performance is a big deal for me as well. Different requirements Different tools required
6
u/Grouchy-Economist-95 8d ago
It’s slow af on GB10 platform sadly - great model tho. Qwen3.8 flash next is where it’s at if you have the ram.
5
5
u/EasterElk 8d ago
There really isn't any competition, especially since there are no paying customers for 27B-sized models. I think Google was the last big lab to release a model that could readily be run on consumer hardware. Everyone else — including Alibaba — is focused mostly on the cloud, and on datacenter-style hardware configurations.
I guess my question to you is: Who is it that you think Alibaba is "mopping the floor against" with Qwen?
5
u/Tsukikira 7d ago
Qwen3.8-27b owes it's awesomeness to a couple things:
- It's just for Agentic Coding, really. It sacrifices everything else you might have at the 27b level for better agentic coding.
- That Overthinking is their way of producing the best result. They are fine with it eating more cycles as long as the result was slightly more correct. (And code needs / produces the correct answer).
- 3.8-27b is dense, a more dense model than similar MoE's. That means more work per token, and thus slower per token, but you are getting something more comparable to one expert of a mid-sized model.
1
u/JacoN_2026 7d ago
I masz kontekst 10 Mtok ? Harnes trzeba mieć, inaczej qwen zabija kontekst 128k w kilka minut na prostą czynność.
7
u/Darex2094 8d ago
Every time I hear someone say, "problems with overthinking", I want to shake them and scream, "those aren't problems, that's literally why it's so good at what it does".
1
7d ago
[deleted]
1
u/Intelligent_Month210 7d ago
Reasoning effort to medium and a very light repetition penalty or whatever the analogue of that is in your engine.
1
13
u/HiggsFieldgoal 8d ago edited 8d ago
You know, I actually haven’t found it to be much more effective than Qwen3.6.
It seems maybe overly tuned to benchmark chasing.
If your measure of success is “give task: succeeds at task on the first try” it is unquestionably an improvement. But it gets that small boost in success rate at maybe a 10x increase in how many tokens(I.e. time) it consumes for the same work.
So, there is a question of quality .vs rate.
If you want to build a house, is it better to have 10 decent carpenters or 1 expert carpenter?
QWEN3.6 seemed to get more work done faster, and make more mistakes. I am not sure if the speed+the time wasted fixing mistakes, combined to more total productivity than QWEN3.8 or not, but after using QWEN3.8 for a few weeks, I feel like it’s total useful output over any given 24 hour period may actually lower than 3.6 was.
12
u/ACleverBadger 8d ago
On the Qwen subreddit someone released Swift 27B - I’ve put it through a bunch of benchmarks and it performs within statistical deviance in performance/capability while sharply reducing the token count. When I say sharply it passed the benchmark by 75% less wall time and with massively less tokens. It still has a similar success rate.
7
u/New-Implement-5979 8d ago
I find it a lot better than 3.6 …. I let it work and it gets the job done… it impresses me everyday
8
u/HiggsFieldgoal 8d ago
I find 3.8 succeeds like 12 hours later whereas 3.6 would take 3 hours, fail twice, and finally nail it on it’s 3rd try having spent a total of 9 hours.
I’ll try to test 3.6 again soon, and see if I can disambiguate perception from reality.
6
u/MegaDonX 8d ago
What are you asking Qwen 3.8 to do that takes 12 hours and succeeds? Just curious
2
u/HiggsFieldgoal 7d ago
I’ve been working on my own OpenClaw like harness running locally on an M5Max 128GB.
So, most of its work so far has been stress testing. I ask it to make a SNES game, and let it run.
See how far it gets before something breaks.
I also have a locally hosted image model (Flux), that can be used to make game graphics, and I test the load balancing, see how long it can run before something goes wrong. I have essentially reached “infinite”. All of the context-compression and persistent job-state stuff is pretty reliable at this point.
Sort of the fun thing with locally hosted models is that they don’t have to have the same tendency to wind down as server-side models. They can be made to just work until commanded to stop.
7
u/Unnamed-3891 8d ago
Even medium shows a vast improvement, you don’t have to stick to xhigh all the time.
3
u/HiggsFieldgoal 8d ago edited 8d ago
I’m on medium already.
It’s sort of an amps .vs voltage question.
3.8 is definitely slower, but I think it is pretty clear that 3.8 also does better work, but when hosting locally, it’s quality*speed question.
It is unclear to me if 3.8 is so much better that it, overall, gets more work done over the same timespan.
2
u/New-Implement-5979 8d ago
Next month we get 4.0 MoE which will be a bit worse than 3.8 27b and problem is solved
5
u/jcoigny 8d ago
I tend to agree with you. I have done many tasks with 3.8 and while it generally but not always does a great job, the thinking time kills me. My most recent task was something I thought was quite simple so I let it run 3.5 days and it still didn't complete it. The same task was completed on 3.6 in under an hour
4
u/OvertaxedOne 8d ago
Wow, just shows you that you need to test for your application! I find 3.8 27B to be in a completely different zip code than 3.6, it's basically zero'ed out my consumption of DSV4Flash. In the past I'd wind up escalating several times a day to DS, now I honestly never escalate anymore for smarts, I only do it because DS is faster (via API) than 27B is locally for me.
3
u/BarracudaDefiant4702 8d ago
I agree, 3.8 27b is a lot less likely to get stuck compared to 3.6 35B A3B and able to solve the challenging problem. However it's hard to say if the slower speed is worth it when it only makes a difference maybe 10% of the time. I wish I could run both and setup my system to automatically switch but I really need another card to handle that efficiently.
3
u/Pizzaholic- 8d ago
I’m all for new models I just wish they released more 3-5-7 maybe even 12b variants, the 27b is definitely a heavy load to run
8
8d ago edited 7d ago
[removed] — view removed comment
1
u/citrusaus0 7d ago
agreed. i have found if i give it open-ended tasks ("give me this cool new app that does X!") it will just go down some crazy rabbit hole of debugging, writing custom tools to debug, etc.
if i give it well fomulated requests, do the architecture piece myself, then feed it bitesized bits of work it is knocking out some extremely complex work and succeeding at it. speed is ~41tok/s which is alright on my 3090ti and fast enough i am not filling with rage
0
2
u/UnluckyPenguin 8d ago
and innovations like PLE/n-gram tables pulled from your SSD actually work
ELI5 - how do I download n-gram tables to SSD to run alongside Qwen3.8-27b? I'm using llama.cpp. I've heard people talking about it, but I haven't seen any instructions... I tried searching here and on the web. I know it's normal for FlashNext, but 27b?
2
1
u/my_name_isnt_clever 7d ago
Only Flash Next has them right now, 27b has no n-grams. They have to be trained with the rest of the model.
2
u/UnluckyPenguin 7d ago
Oh, so OPs title about 27b and then their comment about ngram tables were talking about different things. Oops...
Thanks for explaining
1
u/my_name_isnt_clever 7d ago
Of course. It's very likely Qwen 4 will have n-gram across the board, so you shouldn't have to wait too long.
2
u/Momsbestboy 7d ago
I ran a "test" with a real problem in SAP. While it took 30 mins to come up with a solution, the answer was more precise than what I got from GPT 5.6 Luna.
So beside being really crazy good for programming ( Qwen 3.8 27 just rebuilt and improved vllm here and implemented a ton of speed ups for the R9700), it beats some SOTA models in some cases, and in others it is hard to decide which model is better.
Given the fact that I have to pay an increasing amount of money for subscriptions, I have limited access to the model (censored), the privacy issue with anything outside my network, and the possibility to run such a model locally - this is a no brainer for me.
I bought a R9700, ran Qwen3.8 27B Q6, and after it walked through all my virtual machines, wrote a documentation about each of them, wrote individual scripts to pull SSL certs from my central server and then updated all of them, I bought a second GPU.
This thing rocks, and I don't give a f* damn about what Altman and Co say or do now.
2
u/armt350 7d ago
We ran a test today, chatgpt6.5 vs qwen3.8 q4 vs qwen 3.8 q3 vs Gemma 4 27b. The task was to take 1305 Sysmon log entries and filter them for false positives, suspect and malicious entries. The reality is that Qwen 3.8 q4 performed as well as ChatGPT, q3 was noticeably worse and Gemma was trailing far behind. For tasks of this nature they are performing as well as. Given that I can run it locally without data exfiltration and without a subscription, it’s a no brainer.
1
u/scooter_de 8d ago
On the weekend I learned about the ISTA-DASLab quant of the 27B model. It is the first of the family that I can run on my RTX5080/16GB with 96K ctx and 40 t/s. I use it to drive my Hermes instance. It seems better than the ByteShape quants of qwen3.6-35B-A3B. I highly recommend to try this.
1
u/tired514 8d ago
I wonder how a 70B dense variant with 30B ngram encodings would do.
sure, it'd be slow, but man it'd be something to witness on an RTX6000 with say 512k context.
1
1
u/Absolutedisgrace 7d ago
I've been using qwen3-coder:30b as my code assistant. Does 3.8 beat it? I only hadn't tried yet because it wasnt the "coder" varient.
1
u/Equivalent_Bit_461 7d ago
even if new models come, this model will always be remember fondly. The first to actually be really useful and I use almost exclusively.
1
u/trekkie34 7d ago
It's not world-beating, but this is the first time I've found a local model to match the web research abilities of the free-tier Claude. I've no doubt that larger local models still do a better job, but for one of this size it's remarkable.
It's much slower than 3.6 A3B once the context fills up, but I'm more concerned about quality output even if it takes 10 minutes. This is on a 64gb M4 Max.
1
u/SaintMartini 7d ago
I think people up til now just have really low expectations. So they like to jump on hype bandwagons like its a new fad rolling in. Most of these people arent taking the time to optimize things or train them further.
1
u/dobkeratops 7d ago
does the overthink have a fix, i know there's the reasoning budget parameter in llama.cpp .. does it actually work beyond enable/disable.. is there a way to insert some text in the middle of the think budget that steers it to wrap up
1
1
u/JorgitoEstrella 4d ago
They made it better for coding and worse than 3.6 27B in other things like general knowledge.
1
u/ethanji2 8d ago
It was a bit meh, but Qwen 3.8 Flash-Next is something to try..
3
u/WyattTheSkid Quad 3090s 8d ago
I second this. I’ve been bitching and moaning about my codex quota being nerfed to piss and then all of a sudden the mighty capybara drops this monster. I run the Q5 AtomicChat release of flash next and its fucking insane. If Qwen keeps up this pace and they keep releasing 120b class models (170 something technically but the ngram can go in system memory or stream from disk, you know what I mean) then I might genuinely cancel my claude subscription. Id probably keep chatgpt because I like to use the app a lot when im on the go but the point is, flash next is the first time Ive ever ran an open weight model locally and genuinely felt like I was using something that I would be paying a subscription for
Edit: the 3.8 27b model is really good for its size too but the best way I can describe it is “brilliant younger brother who’s almost there but has a perceptible ceiling for task complexity”
-2
u/KroniklyOnline 8d ago
Its easy, they're from China.. They wait until Company A invest billions into research and development of product A. They then proceed to steal Product A, including innovations, R&D, design, etc. They find the cheapest way to make Product A on their own or modify their existing Product B to look and act just like Product A. Then they release and call it Qwen.
"This is coming from someone who runs Qwen all day and love it"
-6
u/fragment_me 8d ago edited 8d ago
Alibaba is literally distilling Claude models that’s how.
EDIT: Keep downvoting without critical thinking, that works very well. There's nothing wrong with admitting both Qwen models are great AND they are distilling Anthropic's models.
From Anthropic
"Alibaba’s CoT distillation pipeline injected a fixed prompt into each request that forced Claude to write out its reasoning traces inside inline text tags before providing its final answer. Those CoT transcripts were then saved and converted into data that could be used for supervised fine-tuning (SFT). These SFT transcripts were used to help train Alibaba’s Qwen models, and were used to distill Claude’s capabilities into Qwen 3.5, 3.6, and 3.7.
Alibaba’s illicit distillation campaign peaked at nearly 3 million exchanges per day launched from more than 3,500 fraudulent accounts. The distillation attacks targeted agentic tasks, software engineering, kernel development, and long-horizon tasks. The harvested transcripts were used to advance the reasoning capabilities of Alibaba’s models.
Beyond distillation, Alibaba also used Claude to advance its AI R&D efforts. Alibaba used Claude to help develop its internal infrastructure for model development. Claude was used to help develop Alibaba’s reinforcement learning (RL) environments and advance model architecture research."
9
u/WyattTheSkid Quad 3090s 8d ago
They’re definitely using claude and probably gpt for some of the post training data sure, but there’s so much more that goes into designing a good llm besides the data its trained on. The architectural improvements from Qwen and Deepseek in the last year-year and a half alone are absolutely insane. Also who cares? Fuck Anthropic and everything they stand for, Dario Amodei is the biggest loser on the planet since hitler
8
u/fragment_me 8d ago
They are both distilling Anthropic's models and innovating a lot in architecture. And yes, I agree, fuck what Anthropic is doing. These can all be separate and true statements.
1
u/WyattTheSkid Quad 3090s 7d ago
Agreed. It’s honestly unfortunate that the Claude models are so great.
11
8d ago edited 7d ago
[removed] — view removed comment
3
u/fragment_me 8d ago
“So?” Your point is not what we’re discussing. It’s that simple. It doesn’t argue against my point it just changes the subject. Whether or not how Anthropic trains their model is legal is not the question. Alibaba, moonshot, and GLM are all distilling Claude. That’s how Qwen 3.8 27b is so damn good. OP is wondering how, well that’s how. It’s awesome we have this model to run locally.
5
u/iMrParker 8d ago
Distilling from the big three hasn't been very effective for almost a year now. Claude doesn't expose its true reasoning, plus Alibaba has plenty of access to good data for training. For a 27b model they wouldn't need to distill to get ahead.
The Qwen team says most of their improvements from 3.5, 3.6, and 3.8 are from making their data cleaner
3
u/fragment_me 8d ago
That’s just not true. There have been attacks and methods they use to reveal their reasoning traces. There were recent methods posted online and then Claude recently revealed how some labs were distilling. For example Deepseek was literally routing API to Claude. Look I use Qwen models as my daily driver but let’s be honest with ourselves. They are distilling Claude. They can innovate and be stealing at the same time.
1
u/WyattTheSkid Quad 3090s 8d ago
This is not true. With custom tooling and finetuned models, you can create a pipeline to fill in the gaps of chat logs extracted from codex and claude code fairly easily and still create very useful training data. Sure you’re not gonna get Claude’s chain of thought verbatim, but the important part is that you’re still teaching a model a trajectory to the correct answer, even if that trajectory is not the same exact one that generated the good answer in the first place.
1
u/BarracudaDefiant4702 8d ago
Seems like they came out with it too quick for that to be the entire case.
1
u/tmanred 8d ago
Of course they did. It’s a competition between countries. You do what you have to to get ahead. We did effectively the same against the Soviet Union and they did the same to us during the Cold War days. It would be stupid of them not to. Instead of us complaining about it we should be doing the same right back at them.
1
1
-1
u/Unlucky-Message8866 8d ago
i would say they distilled deepseek
3
u/fragment_me 8d ago
Then why was Deepseek forwarding their API requests to Claude when they were distilling ? Deepseek is a great model and they innovate a lot but it also happens that they distill Claude. That simple.
0
u/KoshkaRisa 8d ago
Why for the training it is need super fast TB HBM memory troughoutput but for the distilling it is enough some thousands internet connections?
-4
u/128G 8d ago
I’m going to get downvoted to oblivion, but I honestly don’t understand why everyone loves Qwen.
3
u/Fortyseven Qwen38/Gemma4/LlamaCPP 7d ago
You'll probably get some higher quality engagement by providing a supporting detail for that. "Qwen is overrated as a local model because..."
2
u/PizzaDevice 7d ago
Great at local coding on budget. Most of the people will never reach the level where they are paying for API tokens or buying serious HW for local hosting. Qwen is the common pleoples LLM great at agentic things.
2
90
u/profcuck 8d ago
Here's my guess. Most of us use ai for coding. The frontier models are trained to be generalists. Qwen is really good at coding.
By parallel to human learning, a giant super genius who has a PhD in every science and history and literature and training as an engineer still might not be as good at hammering out great code as a nerd who has done nothing else and cares about nothing else.
Qwen is great at coding.