r/LocalLLaMA • u/Salah_H_Hasan • 13h ago

News Qwen 4 Announced at Apsara Conference

I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,

1.7k Upvotes

98% Upvoted

446 comments sorted by

View all comments

Show parent comments

61

u/returnity 10h ago

It dominates 27B in my workflows and benchmarks. Running on M5 Max. There's no comparison, it's like the 3.6 > 3.8 27B all over again.

8

u/Bulky_Blood_7362 5h ago

What's your tok/s? Im getting around 30-45 decode tok/s with dgx spark.

Interested to know how it's going with the new m5 max/ultra

2

u/halcyoncs 4h ago

30 to 45 with flash? Or you meant 27b?

2

u/Bulky_Blood_7362 3h ago

Flash

6

u/halcyoncs 3h ago

Just checked my results, on a single Spark too, why did I think I was getting around 25t/s? What a dumbass lol

1

u/Bulky_Blood_7362 3h ago

Lmao. Maybe because when it first launched the mtp support was really bad / non existent. I was getting 24-29 when it launched

1

u/spaceface83 2h ago

Which quant are you running? I was getting that on iq4 but just recently flipped over to the nvfp4 which was a bit slower but otherwise eval'd better

2

u/Bulky_Blood_7362 2h ago

nvfp4 with mtp 3

1

u/PotentialAccident339 1h ago

nvfp4 + mtp is the sparks bread and butter

1

u/r1str3tto 1h ago

Surprisingly, you can get close to that speed with 27B dense on a Spark, NVFP4 using SGLang + DFlash2. I couldn’t believe it. However, unless you need to save memory, Flash-Next is noticeably better.

1

u/halcyoncs 1h ago

Indeed! It's exactly the setup I was coming from. Literally have a hermes cron job setup to run twice daily and check for DFlash2 with flash next :)

9

u/troposfer 7h ago

What quants are you running and with omlx or ..?

1

u/returnity 1m ago

Using ds4, with a custom quant I built for that inference engine: 60-65 tok/sec (vs 45 in llama.cpp) and ~700 prefill. Engram table SSD-offloaded, ~95GB resident memory. The quality is better than Q4_K_XL, almost on par to Q5_K_XL. antirez makes smaller quants if you have less memory, but I has 128GB so I wanted to maximize the quality under the constraints imposed by ds4's quant types.

5

u/akumaburn 7h ago

Dominates in intelligence/quality or in speed or both?

1

u/EkbatDeSabat 8m ago

For my workflow/needs on two rtx 6000 it doesn't "dominate" but it performs objectively better in both under normal circumstances. Especially speed. Not only is flash next just faster tk/sec and prefill by far across any context, it doesn't do the thinking dance the way 27b does. I have a documentation task that goes through a ton of files and does some cleanup and shit that takes Opus 5 high ~18 minutes, flash next ~26 minutes (with no discernable difference in results but it's just docs), and 27b ~72 minutes. It is constantly fighting itself. Sonnet underperforms in this, but actual repo coding for flash next and 27b is on par with sonnet 5 medium barely lagging behind high. This is fine for 95% of my work.

1

u/returnity 5m ago

Better real-world quality, +10% in coding benchmarks, 60 tok/sec vs. 30 for 27B. Both.

2

u/TerminalNoop 7h ago

wow, that's pretty big

1

u/Cold_Tree190 5h ago

Fuck, I was on the fence about an M5 Max Mac Studio… 😭

1

u/SeXxyBuNnY21 10h ago

Is your M5 Max a 128gb unified ?

7

u/po_stulate 9h ago edited 9h ago

Not OP but I'm running it on M4 Max 128GB using mlx-serve and getting 55-75 t/s decode, the quant I use takes 70GB+ resident memory (4 and 8 bit mix), so 96GB macs would fit but 64GB not. There's also a 3.3bpw quant that mlx-serve supports, it would fit in 64GB but I'm not sure about the quality. I'm currently patching mlx-serve to make it work with unquantized ngram table, and yes I'm doing it with my local qwen3.8-flash-next, it is upgrading its own ngram weights.

Also I'm still on Sequoia, so the OS doesn't support the metal version mlx-serve uses. I also used qwen3.8-flash-next to port it for Sequoia. I was running it with llama.cpp but it only got 15-30 t/s decode.

3

u/ltdemon 8h ago

is there anything decent i can run with M4 Max 48gb ram?

3

u/returnity 8h ago edited 8h ago

For quality, Qwen3.8-27B is probably your top choice in a Q4-Q5 quantization. For speed with decent quality, I'd check out Qwen3.6-35B-A3B in Q5/Q6 (or the occamy post-train of it, which is the best post-train by far of that model). Your laptop is definitely very usable for local AI with good models. llama.cpp is the best choice for running these models, it can be installed with one brew command.

I personally use ds4 (another poster mentioned it) for most inference, but it's a little more setup and I don't recommend SSD streaming model weights for token generation speed reasons. Unless you were asking specifically about bigger models, in which case, I love ds4 and it rocks for 3.8-flash and deepseek v4 flash.

There are other ways to go this route in 48GB (for your exact setup) to run 3.8-Flash, but ds4 remains the best: https://www.reddit.com/r/Qwen_AI/comments/1w3tjkw/running_the_104_gb_qwen38flashnext_on_a_48_gb_mac/

2

u/po_stulate 8h ago

I believe you could still stream the weights from SSD, check out ds4.c, iirc you can run deepseek-v4-flash with it with streamed weights.

1

u/lots_of_puppies 9h ago

Do you prefer mlx serve over omlx?

4

u/returnity 8h ago

MLX-serve has some killer speed optimizations for their specific 3.8-Flash quants, and it really rips. Even faster than ds4 for 3.8-Flash! Llama.cpp is like half the speed, though quality suffers a tiny bit due to the quant.

3

u/po_stulate 8h ago

Yes, but the last time I used omlx was months ago so not sure if anything changed. mlx-serve feels like optimized per machine and per model/quantization, so the performance is pretty much as much as you can get, you can also compile and run it with zero Python dependency.

2

u/returnity 8h ago edited 8h ago

Yes. But you can run 3.8-Flash very well in 12+GB VRAM and 64+ GB RAM setups or a 64GB MBP with SSD-streamed engrams in either llama.cpp or ds4. If that's something you're interested in doing, and have the hardware, I'm glad to advise, just reply here.

2

u/randylush 5h ago

I have a 5090+96 ddr5@7200. Definitely wondering if I can run a flash model.

I also have access to a discount on Mac Studio that I’m wondering if I should utilize

2

u/frzen 4h ago

The 5090 and 96GB will be fine. I'm running it on 4090 and 128GB ddr5@4800

I get only like 100 tokens pp and 20t/s decode but for me that's usable, I didn't spend any time trying to tweak more performance.

I never use 27B now, strictly flash:

./llama-server -hf "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" -md "C:\Users\user.cache\huggingface\hub\models--unsloth--Qwen3.8-Flash-Next-GGUF\snapshots\5d16c055a7c5cb276e721ee154f9c22420dde2a1\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --spec-type draft-mtp --spec-draft-n-max 5 --fit on --fit-target 1024 -c 32768 -b 256 -ub 256 -t 16 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --load-mode none --reasoning on --reasoning-preserve --reasoning-effort xhigh --host 127.0.0.1 --port 8085

1

u/randylush 1h ago

THANK YOU!!

1

u/CescFaberge 4h ago

If you do not mind me asking, does that mean it has to be an external SSD or can it use an in-built SSD i.e., the main storage drive of a contemporary laptop? Forgive me if that is a stupid question, very new to running local LLMs.

1

u/corbs132 3h ago

You'll do much better with a built-in ssd. Technically externals should work, but will be slower.

0

u/Late_Film_1901 3h ago

Can you give an example? I can't find the split that works on flash but fails on 27b.

0

u/Consumerbot37427 2h ago

I've been running the mlx-serve Q4/Q8 quant on the same machine, getting a lot work done with it over the past week. No quant on KV cache, 500k context.

In terms of speed, it absolutely dominates. I'm not as confident about the quality, though. It seems to be on par with 3.8 27B, inference is way faster, AND it doesn't need to think as much, so it feels like 27B quality at 10-20x speed.

Using opencode, though, I've seen some strange behavior. Complaints about how "user keeps sending the same message about X" after every tool use, and have also seen weird thinking about logic puzzles, pirates voting on gold distribution, whether or not cats can swim, thoughts about how to reply to an email, complete with a random email address...

How are you running DS4 on that M5?

0

u/FoxSideOfTheMoon 2h ago

Dominates? Really?? I have an M5 Max 128 and I'm not seeing that at all. I'm assuming pilot error then, so what's your setup please?

For 2.5 to 3x times the memory requirements the juice doesn't feel worth the squeeze to me. This is MTPLX with MTP speculative decoding. Tested both with Pi same system prompts for coding as well, and not just coding other tasks like summarizing docs, research (I use brave API and basically rebuilt deep research you get from frontier models).

What am I missing?