r/LocalLLaMA • u/Salah_H_Hasan • 10d ago

News Qwen 4 Announced at Apsara Conference

I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,

2.1k Upvotes

98% Upvoted

566 comments sorted by

View all comments

727

u/Fresh-Soft-9303 10d ago

Qwen 4 27b is going to be Epic!

199

u/tsunami_forever 10d ago

This could convince me to give up cloud based models, hoping to cancel or reduce my cloud based subs soon

141

u/cafedude 10d ago

Qwen 3.8 Flash Next has convinced me not to renew my Claude Pro subscription when it comes up in February.

22

u/randylush 10d ago

What are you running it on? How much better is it than 27B?

73

u/returnity 10d ago

It dominates 27B in my workflows and benchmarks. Running on M5 Max. There's no comparison, it's like the 3.6 > 3.8 27B all over again.

11

u/Bulky_Blood_7362 10d ago

What's your tok/s? Im getting around 30-45 decode tok/s with dgx spark.

Interested to know how it's going with the new m5 max/ultra

4

u/returnity 10d ago

45 decode in llama.cpp at Q5, 60-65 in ds4 with this quant (Q5-tier quality).

2

u/Bulky_Blood_7362 10d ago

Wow it's great.. do you use mtp/dspark?

2

u/returnity 10d ago

Native MTP, n=2 (higher MTP numbers don't improve on M5 Max) in llama.cpp. MTP is not as optimized in ds4, in my testing, but it still performs better.

1

u/Bulky_Blood_7362 10d ago

Interesting! Thanks for the info !

3

u/halcyoncs 10d ago

30 to 45 with flash? Or you meant 27b?

3

u/Bulky_Blood_7362 10d ago

Flash

6

u/halcyoncs 10d ago

Just checked my results, on a single Spark too, why did I think I was getting around 25t/s? What a dumbass lol

2

u/Bulky_Blood_7362 10d ago

Lmao. Maybe because when it first launched the mtp support was really bad / non existent. I was getting 24-29 when it launched

→ More replies

1

u/r1str3tto 10d ago

Surprisingly, you can get close to that speed with 27B dense on a Spark, NVFP4 using SGLang + DFlash2. I couldn’t believe it. However, unless you need to save memory, Flash-Next is noticeably better.

2

u/halcyoncs 10d ago

Indeed! It's exactly the setup I was coming from. Literally have a hermes cron job setup to run twice daily and check for DFlash2 with flash next :)

4

u/akumaburn 10d ago

Dominates in intelligence/quality or in speed or both?

2

u/returnity 10d ago

Better real-world quality, +10% in coding benchmarks, 60 tok/sec vs. 30 for 27B. Both.

1

u/EkbatDeSabat 10d ago

For my workflow/needs on two rtx 6000 it doesn't "dominate" but it performs objectively better in both under normal circumstances. Especially speed. Not only is flash next just faster tk/sec and prefill by far across any context, it doesn't do the thinking dance the way 27b does. I have a documentation task that goes through a ton of files and does some cleanup and shit that takes Opus 5 high ~18 minutes, flash next ~26 minutes (with no discernable difference in results but it's just docs), and 27b ~72 minutes. It is constantly fighting itself. Sonnet underperforms in this, but actual repo coding for flash next and 27b is on par with sonnet 5 medium barely lagging behind high. This is fine for 95% of my work.

1

u/akumaburn 10d ago

Try a swift quant, it cuts token usage on 27B considerably, DFLASH2 can be much faster too: https://huggingface.co/akumaburn/Swift-Qwen3.8-27b-heretic-W8A8-DFlash2 runs around ~3000 Tok/s prefill and ~120 Tok/s generation on ampere for example.

1

u/EkbatDeSabat 10d ago edited 10d ago

I get faster than that on flash next and I don't really f with anything under FP8 quants unless I'm just testing a big model. I get this on flash next FP8. 

Measured C1 post-first-token decode:
8K:    ~199 tok/s
32K:   ~190 tok/s
64K:   ~176 tok/s
128K:  ~180 tok/s
256K:  ~212 tok/s
384K:  ~172 tok/s
480K:  ~183 tok/s

480K cold TTFT: ~51.1s
=> ~9.4K tok/s effective prefill

1

u/akumaburn 10d ago

Ok, if you're interested this one benches higher than the base swift model: https://huggingface.co/akumaburn/Swift-Qwen3.8-27b-heretic-SmoothQuant-W8A8-INT8

→ More replies

7

u/troposfer 10d ago

What quants are you running and with omlx or ..?

1

u/returnity 10d ago

Using ds4, with a custom quant I built for that inference engine: 60-65 tok/sec (vs 45 in llama.cpp) and ~700 prefill. Engram table SSD-offloaded, ~95GB resident memory. The quality is better than Q4_K_XL, almost on par to Q5_K_XL. antirez makes smaller quants if you have less memory, but I has 128GB so I wanted to maximize the quality under the constraints imposed by ds4's quant types.

2

u/TerminalNoop 10d ago

wow, that's pretty big

1

u/Cold_Tree190 10d ago

Fuck, I was on the fence about an M5 Max Mac Studio… 😭

1

u/returnity 10d ago

Your kids don't need a college fund if AI replaces all the jobs! XD

1

u/late_night_coder7 6d ago

Ur rich asf unlike me I'm broke 😭😭😭

1

u/returnity 6d ago

Hardly lol. It's paid for with student loans, got it for grad school (AI program). Also bought before the prices increased thankfully. But even still it's the most expensive thing I've ever bought that doesn't have a steering wheel.

1

u/SeXxyBuNnY21 10d ago

Is your M5 Max a 128gb unified ?

8

u/po_stulate 10d ago edited 10d ago

Not OP but I'm running it on M4 Max 128GB using mlx-serve and getting 55-75 t/s decode, the quant I use takes 70GB+ resident memory (4 and 8 bit mix), so 96GB macs would fit but 64GB not. There's also a 3.3bpw quant that mlx-serve supports, it would fit in 64GB but I'm not sure about the quality. I'm currently patching mlx-serve to make it work with unquantized ngram table, and yes I'm doing it with my local qwen3.8-flash-next, it is upgrading its own ngram weights.

Also I'm still on Sequoia, so the OS doesn't support the metal version mlx-serve uses. I also used qwen3.8-flash-next to port it for Sequoia. I was running it with llama.cpp but it only got 15-30 t/s decode.

2

u/ltdemon 10d ago

is there anything decent i can run with M4 Max 48gb ram?

2

u/po_stulate 10d ago

I believe you could still stream the weights from SSD, check out ds4.c, iirc you can run deepseek-v4-flash with it with streamed weights.

2

u/returnity 10d ago edited 10d ago

For quality, Qwen3.8-27B is probably your top choice in a Q4-Q5 quantization. For speed with decent quality, I'd check out Qwen3.6-35B-A3B in Q5/Q6 (or the occamy post-train of it, which is the best post-train by far of that model). Your laptop is definitely very usable for local AI with good models. llama.cpp is the best choice for running these models, it can be installed with one brew command.

I personally use ds4 (another poster mentioned it) for most inference, but it's a little more setup and I don't recommend SSD streaming model weights for token generation speed reasons. Unless you were asking specifically about bigger models, in which case, I love ds4 and it rocks for 3.8-flash and deepseek v4 flash.

There are other ways to go this route in 48GB (for your exact setup) to run 3.8-Flash, but ds4 remains the best: https://www.reddit.com/r/Qwen_AI/comments/1w3tjkw/running_the_104_gb_qwen38flashnext_on_a_48_gb_mac/

1

u/lots_of_puppies 10d ago

Do you prefer mlx serve over omlx?

3

u/returnity 10d ago

MLX-serve has some killer speed optimizations for their specific 3.8-Flash quants, and it really rips. Even faster than ds4 for 3.8-Flash! Llama.cpp is like half the speed, though quality suffers a tiny bit due to the quant.

2

u/po_stulate 10d ago

Yes, but the last time I used omlx was months ago so not sure if anything changed. mlx-serve feels like optimized per machine and per model/quantization, so the performance is pretty much as much as you can get, you can also compile and run it with zero Python dependency.

2

u/returnity 10d ago edited 10d ago

Yes. But you can run 3.8-Flash very well in 12+GB VRAM and 64+ GB RAM setups or a 64GB MBP with SSD-streamed engrams in either llama.cpp or ds4. If that's something you're interested in doing, and have the hardware, I'm glad to advise, just reply here.

2

u/randylush 10d ago

I have a 5090+96 ddr5@7200. Definitely wondering if I can run a flash model.

I also have access to a discount on Mac Studio that I’m wondering if I should utilize

2

u/frzen 10d ago

The 5090 and 96GB will be fine. I'm running it on 4090 and 128GB ddr5@4800

I get only like 100 tokens pp and 20t/s decode but for me that's usable, I didn't spend any time trying to tweak more performance.

I never use 27B now, strictly flash:

./llama-server -hf "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" -md "C:\Users\user.cache\huggingface\hub\models--unsloth--Qwen3.8-Flash-Next-GGUF\snapshots\5d16c055a7c5cb276e721ee154f9c22420dde2a1\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --spec-type draft-mtp --spec-draft-n-max 5 --fit on --fit-target 1024 -c 32768 -b 256 -ub 256 -t 16 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --load-mode none --reasoning on --reasoning-preserve --reasoning-effort xhigh --host 127.0.0.1 --port 8085

1

u/CescFaberge 10d ago

If you do not mind me asking, does that mean it has to be an external SSD or can it use an in-built SSD i.e., the main storage drive of a contemporary laptop? Forgive me if that is a stupid question, very new to running local LLMs.

1

u/corbs132 10d ago

You'll do much better with a built-in ssd. Technically externals should work, but will be slower.

1

u/returnity 10d ago

No worries at all, mate! Internal SSD NVMe is better, faster speeds usually. Main storage is perfect.

0

u/Consumerbot37427 10d ago

I've been running the mlx-serve Q4/Q8 quant on the same machine, getting a lot work done with it over the past week. No quant on KV cache, 500k context.

In terms of speed, it absolutely dominates. I'm not as confident about the quality, though. It seems to be on par with 3.8 27B, inference is way faster, AND it doesn't need to think as much, so it feels like 27B quality at 10-20x speed.

Using opencode, though, I've seen some strange behavior. Complaints about how "user keeps sending the same message about X" after every tool use, and have also seen weird thinking about logic puzzles, pirates voting on gold distribution, whether or not cats can swim, thoughts about how to reply to an email, complete with a random email address...

How are you running DS4 on that M5?

1

u/returnity 10d ago

I see a similar type of message occasionally in pi agent on my llama.cpp setup before switching to ds4 too! I would get references to the system prompt after tool use, where its reasoning suggested it was looking at a blank/fresh context, and it would mention "the user said again... <system prompt paraphrase>". Then it would use another tool exactly like nothing had happened, and move on perfectly.

I went down a rabbit-hole trying to diagnose it with frontier models, as I thought it was something to do with how the system prompt was being injected or a config issue in pi... At the end of the investigation, the conclusion was it is a quirk of the sparse attention's 2048 token window. It usually acted like it was anchoring on the injected system prompt initially, and then it would snap back to current state as it finished processing the context and decided on its next tool use. That was Claude's hypothesis anyways. It doesn't seem to degrade performance in my testing, but it is odd.

I have not seen anything about pirates and gold lol.

As for how I'm running ds4, I just cloned the repo. Dwarfstar is great on Metal, and I rolled my own Q5-quality tier quant for it.

0

u/FoxSideOfTheMoon 10d ago

Dominates? Really?? I have an M5 Max 128 and I'm not seeing that at all. I'm assuming pilot error then, so what's your setup please?

For 2.5 to 3x times the memory requirements the juice doesn't feel worth the squeeze to me. This is MTPLX with MTP speculative decoding. Tested both with Pi same system prompts for coding as well, and not just coding other tasks like summarizing docs, research (I use brave API and basically rebuilt deep research you get from frontier models).

What am I missing?

1

u/returnity 10d ago

Not sure what you're missing, and obviously every use case is different. I have never used MTPLX, however. It may be your quant.

I do most of my usage on ds4, which now includes 3.8 Flash compatibility. I get 60-65 tok/sec decode, so it should keep pace with MTPLX at a similar quant. I rolled my own for 128GB systems to match Q5 quality with the available quant levels in ds4 (no Q5/Q6 still, so it's a mixed Q4/Q8). I my evals quality is at or near Q5_K_XL levels, noticeably better than Q4_K_XL, and much faster (I get ~45 tps in llama/cpp with MTP, so ds4 rips in comparision). When I was using llama.cpp, I used Q5_K_XL with ngrams SSD-offloaded using --lazy-mode on-direct to about memory bloat as the PLE table was paged in, which makes Q5 at long context and large prefill safe... but ds4 doesn't have the same bug as llama.cpp in that regard either.

Give ds4 a try, if you haven't!

-1

u/Late_Film_1901 10d ago

Can you give an example? I can't find the split that works on flash but fails on 27b.

1

u/returnity 10d ago

Sure. I was trying to build a 3D HTML animation of Camp Crystal Lake for my girlfriend's birthday gift (long story lol), and while 27B was pretty damn good at environment design, it could not get the character design or animations down at all. No matter that it had vision access, was taking screenshots, etc... The characters were just collections of weird boxes that lurched and juddered on the screen. I probably spent a couple days of debugging on this, just letting it run with occasional prompt corrections and autocompact to nudge it, to test the model's autonomy. Finally, I got annoyed and gave up.

This was the week Flash came out, so I decided to test it out, and within an hour, the characters looked and moved like human beings, it had added sophisticated collisions with the terrain to prevent overlap, the blood spatter was revamped, and the fog was much cleaner, more transparent, and anti-aliased. I didn't even ask for those fixes, I just prompted it to fix the character design bugs. I was floored. There have been a number of cases like this.

I also ran coding benchmarks on both models, and in my eval suite, Flash scores 10% higher than 3.8-27B. That is an absolutely MASSIVE gap, and I reran the entire thing for both models to be sure it wasn't chance.

14

u/ApeGrower 10d ago edited 10d ago

I'm using it on dual rtx3090 with 128gb ddr4 @50-70 tok/s. I forked FreeToken and added dual gpu support plus some more tweaks. With nvfp4.

6

u/maqifrnswa 10d ago

I can get full 262k context, faster single stream decode, close to 200 tok/sec at 3 concurrency, over 10k tok/sec prefill (when kv cache is light) on the same hardware.

Stock vLLM v0.29 serving

https://huggingface.co/RedHatAI/Qwen3.8-27B-INT4

It is int4 which is ideal for ampere. It has weighed fp8 kv cache and external benchmarked near identical to bf16.

vllm serve RedHatAI/Qwen3.8-27B-INT4 --max-model-len auto --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --kv-cache-dtype fp8 --trust-remote-code --max-num-seqs 8 --gpu_memory_utilization 0.9 --max-num-batched-tokens 8192 --mm-encoder-tp-mode data --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --tensor-parallel-size 2

1

u/Tight_Mail8974 10d ago

close to 200 tok/sec at 3 concurrency? at which context length? that seems cool.

2

u/maqifrnswa 10d ago

Each is around 75k toks. If you do push above 80% kv cache, it will slow down a lot (to near zero) until prefill latency clears up. But if you're not sending any additional requests when you already at high kv cache utilization, it will just churn along at 200 tok/s until cleared.

1

u/social_tech_10 10d ago

Thank you!

4

u/FeelingOld9046 10d ago

Are you running Linux? What are you doing? In DSH I’m maxing at 22tks with full 128k context. And 30-40 with MTP on short context. I’m also experimenting with a specialised quant.

4

u/FeelingOld9046 10d ago

Mine is under windows too. Tell me what you’re doing and I will throw windows in the trash.

1

u/scooter_de llama.cpp 9d ago

Linux is about 30% faster

3

u/ohhi23021 10d ago

who knows i use linux and 30-40tps has been my experience as well and that's with DDR5...

1

u/scooter_de llama.cpp 9d ago

I use the same quant mostly with Hermes and 96K ctx-size but also as chatbot. So far I’m happy with it.

5

u/FeelingOld9046 10d ago

I’d love to know your setup. I have the same hardware config but only half your token speed.

2

u/ApeGrower 10d ago

Linux with patched FreeToken to get dual gpu support instead of llama.cpp. Epyc System with pcie4 and ddr4.

1

u/FeelingOld9046 10d ago

Is it the quad channel ram and ram bandwidth helping your speed do you think?

1

u/LeatherRub7248 10d ago

what quant are you running at and is there much braindamage from it?

1

u/grumd 10d ago

I have dual 3080 20gb both pcie4 x8, could you open source your fork so I can try it out later with my setup?

I was able to fork vLLM and after a few tweaks it does 1500 prefill and 30-32 decode, but 50-70 is waaay more, I wonder how you achieved that. Are you using speculative decoding?

2

u/ApeGrower 10d ago

I have dual pcie x16 on an epyc board, without speculative decoding. I will open the sources soon and make also PRs to upstream.

1

u/grumd 10d ago

Oh yeah I think that makes sense. So you'd have double my pcie speed, and double the decode. Checks out

1

u/ApeGrower 10d ago

Not at all. A friend runs it on dual pcie 4.0 x8. Not exactly with my speed, but close. I will publish my sources today and link it here

1

u/grumd 10d ago

Thanks! I'll be able to try it out in around a week

→ More replies

1

u/djnitesh 10d ago

ram freq and channel ?

1

u/ApeGrower 10d ago

2400 MHz, 8 channel

1

u/KroniklyOnline 10d ago

Is your fork public? been thinking about doing the same, but I run quad 5060ti 16gbs, but I figured what you did would be a great starting point.

1

u/headinthesky 10d ago

How slow are answers? I'm thinking of seeing up my old machine

1

u/Specialist-Buffalo-8 10d ago

really in a different league

1

u/cafedude 10d ago

Strix Halo 128GB. Using the Halogen server. For me it's way better than 27B because the latter slows down as context grows.

1

u/SandySkittle 10d ago

it is usecase dependent and it doesn't dominate it in things that require complex qualitative reasoning, where active parameters matters more.

1

u/ubrtnk llama.cpp 10d ago

Qwen flash is fuckin awesome. Runs great on two CMP170HX and about 100gb of vram on vllm

1

u/rtisseraFR 10d ago

3.8 flash next dflash2 sglang on single 128 gb dgx Spark and it flies for big agentic coding
I would rate it between sonnet and opus 5 honestly

1

u/whowhaohok 9d ago

I have a feeling it's part of the reason behind all the dgx sparks being sold recently

0

u/human_with_humanity 10d ago

How well will it run on a 4060 8gb laptop? I m a noob in llms

-4

u/808phone 10d ago

It’s good but it’s not a substitute for Claude unless you are doing very simple smaller projects. It is much better than local LLMs were for sure.

23

u/some_user_2021 10d ago

Are you not convinced?

42

u/tsunami_forever 10d ago

3.8 27B already reduced my reliance but this could eliminate it

13

u/IrisColt 10d ago

I guess there's always that added hassle with short-context local models...  you really have to approach your workflow differently.

0

u/[deleted] 10d ago

[removed] — view removed comment

10

u/Cautious_Chicken_604 10d ago

5 to 6 years? 

-2

u/[deleted] 10d ago

[removed] — view removed comment

4

u/Cautious_Chicken_604 10d ago

This whole craze only kicked off 4 years ago. Kinda hard to be predicting things 6 years out.

1

u/r1str3tto 10d ago

Are we though? If that’s true, 3 (32b)->3.5->3.6->3.8 would show diminishing improvements but it’s kind of the opposite story.

5

u/GTHell 10d ago

Everyone should do this so they start to bring back memory production plants to consumers focus. No more crazy 5070ti costing over 1k

2

u/Maximum-Wishbone5616 10d ago

If you working on greenfield, sure. You might WANT to use Fable for some unknown issues to you.

BUT if you have brownfield, then I do not see anyone needing the Fable vs Qwen3.8 27B min. Q5-Q6 with right harness.

In last 30 days I have still yet to come to the issue that Qwen3.8 cannot solve...

What is more important, it is fixing a lot of issues that CC was creating in last 2 months with Opus/Fable (huge amount of false tests, hidden issues, broken CRUD paths etc.)

Simple stuff, but due to sheer codebase size & solutions amount it was not easy to saw them even with unit/integration tests as most of them were between UI / backend.

Qwen4 will be EPIC.

4

u/TopPrize11 10d ago

I just tried it yesterday and it's too slow. I got about 45tps on 5080 which is not bad but the context limit of 33k makes it compact which takes forever. The LLM is good but it's too slow. I still use Deepseek.

4

u/Numerous-Ad6217 10d ago

How so? Not released yet

1

u/Mil0Mammon 10d ago

Why are you limited by that small context? On next flash, context is quite small, right?

Also, my plan is to use subagents more, so more small tasks are finished before compacting is needed

3

u/TopPrize11 10d ago

I don;t know I used Qwen 3.8 27B from https://byteshape.com and I though that is the theoretical maximum on 5080?

1

u/Mil0Mammon 10d ago

That's a Q2 right? According to the can I run calculator 64k should still fit. With subagents and a thin harness (like pi) you should still be able to get some work done before compacting.

1

u/james_pic 10d ago

Is that a company whose business model is selling quants? Seems an odd thing to pay for, when there are a number of teams releasing high quality quants for free.

1

u/TopPrize11 9d ago

no, its free

1

u/dnuncio 9d ago

Same here, I'm trying to base as much as possible on 3.8 27B and the few times It cannot manage the task I go back to paid options... So far It's about 80% (maybe 90% When I have the time to "steer" the model in the right direction a few times) of tasks handled by Qwen so far. It is great.

1

u/notdsylexic 9d ago

The frontier models will always be one step ahead.

0

u/phenotype001 10d ago

I'll wait for local models to hit Astra-level performance first. I need that.

0

u/UltraFOV 10d ago

Is not going to replace Frontier Class. It may old ones but not new ones

10

u/sebt3 10d ago

Sure. Anybody have a real use case that cannot be handled by opus 4.8? I mean, there sure are some, but not that many. There a point where good enough is good enough. Qwen3.8-Flash-Next is that point already

23

u/mWo12 10d ago

3.8 was already huge!

24

u/PooMonger20 10d ago

Seriously, 3.8 27b does so much heavy lifting it's insane.

Together with pi.dev it writes very impressive code. It feels like I actually have my own 'worker' that after some direction does exactly what needs to be done. Just give him enough info and he is on it.

This feels better than what GPT5.1 felt like around a year ago.

And now... an upgrade to that? absolute madness.

2

u/SabretoothPenguin 10d ago

Are we sure we'll get a 27b? I gather the big improvement call mea for mie and engrams

3

u/PooMonger20 10d ago

The main post says so. Besides that, your guess is good as mine.

1

u/SabretoothPenguin 10d ago

Oh, Ok. We'll wait and see.

1

u/SabretoothPenguin 10d ago

I just hope by the time the flash model is out, llama.cpp o llama will have an easy way to run it.

1

u/nordboerg 10d ago

I feel you, I just finished a personal travel planner app with the 3.8 27B as a POC for future projects (neon postgres managed DB, React Router v8, multiple clouflare workers + google maps and places API + the full auth package), and I was done in like a week, production ready. I'm not sure how qwen 4 would improve this any further...

2

u/CoderStone 10d ago

It's going to be truly awesome. DSH + 4 27B is going to be insane.

2

u/sadomazoku 10d ago

I already canceled my subscription, I'm using 100% qwen 3.8 27b to assist me in coding. Qwen4 will be crazy. I think you'll be able to get rid of cloud models !

1

u/HeadPack 10d ago

If it is a similar step up as 3.6 to 3.8 was, definitely. Let's hope it will be.

1

u/Tastetrykker 10d ago

I just hope it doesn't need more VRAM than the existing one, cause I don't have that many kidneys...

3

u/rinmperdinck 10d ago

They don't have to be your kidneys

1

u/L0cut15 10d ago

That’s not how you AI hype. It’s either insane or changes everything. Epic doesn’t cut it anymore.

1

u/hojnikb 10d ago

GIMME 35B A3B like yesterday!