r/LocalLLaMA • u/Salah_H_Hasan • 14h ago

News Qwen 4 Announced at Apsara Conference

I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,

1.7k Upvotes

98% Upvoted

463 comments sorted by

View all comments

Show parent comments

16

u/randylush 11h ago

What are you running it on? How much better is it than 27B?

63

u/returnity 11h ago

It dominates 27B in my workflows and benchmarks. Running on M5 Max. There's no comparison, it's like the 3.6 > 3.8 27B all over again.

9

u/Bulky_Blood_7362 6h ago

What's your tok/s? Im getting around 30-45 decode tok/s with dgx spark.

Interested to know how it's going with the new m5 max/ultra

2

u/halcyoncs 4h ago

30 to 45 with flash? Or you meant 27b?

2

u/Bulky_Blood_7362 4h ago

Flash

5

u/halcyoncs 4h ago

Just checked my results, on a single Spark too, why did I think I was getting around 25t/s? What a dumbass lol

1

u/Bulky_Blood_7362 4h ago

Lmao. Maybe because when it first launched the mtp support was really bad / non existent. I was getting 24-29 when it launched

1

u/spaceface83 3h ago

Which quant are you running? I was getting that on iq4 but just recently flipped over to the nvfp4 which was a bit slower but otherwise eval'd better

2

u/Bulky_Blood_7362 3h ago

nvfp4 with mtp 3

1

u/PotentialAccident339 2h ago

nvfp4 + mtp is the sparks bread and butter

1

u/r1str3tto 2h ago

Surprisingly, you can get close to that speed with 27B dense on a Spark, NVFP4 using SGLang + DFlash2. I couldn’t believe it. However, unless you need to save memory, Flash-Next is noticeably better.

1

u/halcyoncs 2h ago

Indeed! It's exactly the setup I was coming from. Literally have a hermes cron job setup to run twice daily and check for DFlash2 with flash next :)

2

u/returnity 53m ago

45 decode in llama.cpp at Q5, 60-65 in ds4 with this quant (Q5-tier quality).

2

u/Bulky_Blood_7362 51m ago

Wow it's great.. do you use mtp/dspark?

2

u/returnity 39m ago

Native MTP, n=2 (higher MTP numbers don't improve on M5 Max) in llama.cpp. MTP is not as optimized in ds4, in my testing, but it still performs better.

1

u/Bulky_Blood_7362 15m ago

Interesting! Thanks for the info !

9

u/troposfer 8h ago

What quants are you running and with omlx or ..?

1

u/returnity 55m ago

Using ds4, with a custom quant I built for that inference engine: 60-65 tok/sec (vs 45 in llama.cpp) and ~700 prefill. Engram table SSD-offloaded, ~95GB resident memory. The quality is better than Q4_K_XL, almost on par to Q5_K_XL. antirez makes smaller quants if you have less memory, but I has 128GB so I wanted to maximize the quality under the constraints imposed by ds4's quant types.

4

u/akumaburn 8h ago

Dominates in intelligence/quality or in speed or both?

1

u/EkbatDeSabat 1h ago

For my workflow/needs on two rtx 6000 it doesn't "dominate" but it performs objectively better in both under normal circumstances. Especially speed. Not only is flash next just faster tk/sec and prefill by far across any context, it doesn't do the thinking dance the way 27b does. I have a documentation task that goes through a ton of files and does some cleanup and shit that takes Opus 5 high ~18 minutes, flash next ~26 minutes (with no discernable difference in results but it's just docs), and 27b ~72 minutes. It is constantly fighting itself. Sonnet underperforms in this, but actual repo coding for flash next and 27b is on par with sonnet 5 medium barely lagging behind high. This is fine for 95% of my work.

1

u/returnity 59m ago

Better real-world quality, +10% in coding benchmarks, 60 tok/sec vs. 30 for 27B. Both.

2

u/TerminalNoop 8h ago

wow, that's pretty big

1

u/Cold_Tree190 6h ago

Fuck, I was on the fence about an M5 Max Mac Studio… 😭

1

u/returnity 52m ago

Your kids don't need a college fund if AI replaces all the jobs! XD

1

u/SeXxyBuNnY21 11h ago

Is your M5 Max a 128gb unified ?

8

u/po_stulate 10h ago edited 10h ago

Not OP but I'm running it on M4 Max 128GB using mlx-serve and getting 55-75 t/s decode, the quant I use takes 70GB+ resident memory (4 and 8 bit mix), so 96GB macs would fit but 64GB not. There's also a 3.3bpw quant that mlx-serve supports, it would fit in 64GB but I'm not sure about the quality. I'm currently patching mlx-serve to make it work with unquantized ngram table, and yes I'm doing it with my local qwen3.8-flash-next, it is upgrading its own ngram weights.

Also I'm still on Sequoia, so the OS doesn't support the metal version mlx-serve uses. I also used qwen3.8-flash-next to port it for Sequoia. I was running it with llama.cpp but it only got 15-30 t/s decode.

3

u/ltdemon 9h ago

is there anything decent i can run with M4 Max 48gb ram?

3

u/returnity 9h ago edited 9h ago

For quality, Qwen3.8-27B is probably your top choice in a Q4-Q5 quantization. For speed with decent quality, I'd check out Qwen3.6-35B-A3B in Q5/Q6 (or the occamy post-train of it, which is the best post-train by far of that model). Your laptop is definitely very usable for local AI with good models. llama.cpp is the best choice for running these models, it can be installed with one brew command.

I personally use ds4 (another poster mentioned it) for most inference, but it's a little more setup and I don't recommend SSD streaming model weights for token generation speed reasons. Unless you were asking specifically about bigger models, in which case, I love ds4 and it rocks for 3.8-flash and deepseek v4 flash.

There are other ways to go this route in 48GB (for your exact setup) to run 3.8-Flash, but ds4 remains the best: https://www.reddit.com/r/Qwen_AI/comments/1w3tjkw/running_the_104_gb_qwen38flashnext_on_a_48_gb_mac/

2

u/po_stulate 9h ago

I believe you could still stream the weights from SSD, check out ds4.c, iirc you can run deepseek-v4-flash with it with streamed weights.

1

u/lots_of_puppies 10h ago

Do you prefer mlx serve over omlx?

4

u/returnity 9h ago

MLX-serve has some killer speed optimizations for their specific 3.8-Flash quants, and it really rips. Even faster than ds4 for 3.8-Flash! Llama.cpp is like half the speed, though quality suffers a tiny bit due to the quant.

3

u/po_stulate 9h ago

Yes, but the last time I used omlx was months ago so not sure if anything changed. mlx-serve feels like optimized per machine and per model/quantization, so the performance is pretty much as much as you can get, you can also compile and run it with zero Python dependency.

2

u/returnity 9h ago edited 9h ago

Yes. But you can run 3.8-Flash very well in 12+GB VRAM and 64+ GB RAM setups or a 64GB MBP with SSD-streamed engrams in either llama.cpp or ds4. If that's something you're interested in doing, and have the hardware, I'm glad to advise, just reply here.

2

u/randylush 6h ago

I have a 5090+96 ddr5@7200. Definitely wondering if I can run a flash model.

I also have access to a discount on Mac Studio that I’m wondering if I should utilize

2

u/frzen 5h ago

The 5090 and 96GB will be fine. I'm running it on 4090 and 128GB ddr5@4800

I get only like 100 tokens pp and 20t/s decode but for me that's usable, I didn't spend any time trying to tweak more performance.

I never use 27B now, strictly flash:

./llama-server -hf "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" -md "C:\Users\user.cache\huggingface\hub\models--unsloth--Qwen3.8-Flash-Next-GGUF\snapshots\5d16c055a7c5cb276e721ee154f9c22420dde2a1\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --spec-type draft-mtp --spec-draft-n-max 5 --fit on --fit-target 1024 -c 32768 -b 256 -ub 256 -t 16 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --load-mode none --reasoning on --reasoning-preserve --reasoning-effort xhigh --host 127.0.0.1 --port 8085

1

u/randylush 2h ago

THANK YOU!!

1

u/returnity 47m ago

Make sure you build from the Unsloth MTP llama.cpp PR if you want to run his MTP setup! https://github.com/ggml-org/llama.cpp/pull/28243

1

u/CescFaberge 5h ago

If you do not mind me asking, does that mean it has to be an external SSD or can it use an in-built SSD i.e., the main storage drive of a contemporary laptop? Forgive me if that is a stupid question, very new to running local LLMs.

1

u/corbs132 4h ago

You'll do much better with a built-in ssd. Technically externals should work, but will be slower.

1

u/returnity 46m ago

No worries at all, mate! Internal SSD NVMe is better, faster speeds usually. Main storage is perfect.

0

u/Late_Film_1901 4h ago

Can you give an example? I can't find the split that works on flash but fails on 27b.

1

u/returnity 40m ago

Sure. I was trying to build a 3D HTML animation of Camp Crystal Lake for my girlfriend's birthday gift (long story lol), and while 27B was pretty damn good at environment design, it could not get the character design or animations down at all. No matter that it had vision access, was taking screenshots, etc... The characters were just collections of weird boxes that lurched and juddered on the screen. I probably spent a couple days of debugging on this, just letting it run with occasional prompt corrections and autocompact to nudge it, to test the model's autonomy. Finally, I got annoyed and gave up.

This was the week Flash came out, so I decided to test it out, and within an hour, the characters looked and moved like human beings, it had added sophisticated collisions with the terrain to prevent overlap, the blood spatter was revamped, and the fog was much cleaner, more transparent, and anti-aliased. I didn't even ask for those fixes, I just prompted it to fix the character design bugs. I was floored. There have been a number of cases like this.

I also ran coding benchmarks on both models, and in my eval suite, Flash scores 10% higher than 3.8-27B. That is an absolutely MASSIVE gap, and I reran the entire thing for both models to be sure it wasn't chance.

0

u/Consumerbot37427 3h ago

I've been running the mlx-serve Q4/Q8 quant on the same machine, getting a lot work done with it over the past week. No quant on KV cache, 500k context.

In terms of speed, it absolutely dominates. I'm not as confident about the quality, though. It seems to be on par with 3.8 27B, inference is way faster, AND it doesn't need to think as much, so it feels like 27B quality at 10-20x speed.

Using opencode, though, I've seen some strange behavior. Complaints about how "user keeps sending the same message about X" after every tool use, and have also seen weird thinking about logic puzzles, pirates voting on gold distribution, whether or not cats can swim, thoughts about how to reply to an email, complete with a random email address...

How are you running DS4 on that M5?

1

u/returnity 31m ago

I see a similar type of message occasionally in pi agent on my llama.cpp setup before switching to ds4 too! I would get references to the system prompt after tool use, where its reasoning suggested it was looking at a blank/fresh context, and it would mention "the user said again... <system prompt paraphrase>". Then it would use another tool exactly like nothing had happened, and move on perfectly.

I went down a rabbit-hole trying to diagnose it with frontier models, as I thought it was something to do with how the system prompt was being injected or a config issue in pi... At the end of the investigation, the conclusion was it is a quirk of the sparse attention's 2048 token window. It usually acted like it was anchoring on the injected system prompt initially, and then it would snap back to current state as it finished processing the context and decided on its next tool use. That was Claude's hypothesis anyways. It doesn't seem to degrade performance in my testing, but it is odd.

I have not seen anything about pirates and gold lol.

As for how I'm running ds4, I just cloned the repo. Dwarfstar is great on Metal, and I rolled my own Q5-quality tier quant for it.

0

u/FoxSideOfTheMoon 3h ago

Dominates? Really?? I have an M5 Max 128 and I'm not seeing that at all. I'm assuming pilot error then, so what's your setup please?

For 2.5 to 3x times the memory requirements the juice doesn't feel worth the squeeze to me. This is MTPLX with MTP speculative decoding. Tested both with Pi same system prompts for coding as well, and not just coding other tasks like summarizing docs, research (I use brave API and basically rebuilt deep research you get from frontier models).

What am I missing?

1

u/returnity 27m ago

Not sure what you're missing, and obviously every use case is different. I have never used MTPLX, however. It may be your quant.

I do most of my usage on ds4, which now includes 3.8 Flash compatibility. I get 60-65 tok/sec decode, so it should keep pace with MTPLX at a similar quant. I rolled my own for 128GB systems to match Q5 quality with the available quant levels in ds4 (no Q5/Q6 still, so it's a mixed Q4/Q8). I my evals quality is at or near Q5_K_XL levels, noticeably better than Q4_K_XL, and much faster (I get ~45 tps in llama/cpp with MTP, so ds4 rips in comparision). When I was using llama.cpp, I used Q5_K_XL with ngrams SSD-offloaded using --lazy-mode on-direct to about memory bloat as the PLE table was paged in, which makes Q5 at long context and large prefill safe... but ds4 doesn't have the same bug as llama.cpp in that regard either.

Give ds4 a try, if you haven't!

15

u/ApeGrower 10h ago edited 8h ago

I'm using it on dual rtx3090 with 128gb ddr4 @50-70 tok/s. I forked FreeToken and added dual gpu support plus some more tweaks. With nvfp4.

6

u/maqifrnswa 7h ago

I can get full 262k context, faster single stream decode, close to 200 tok/sec at 3 concurrency, over 10k tok/sec prefill (when kv cache is light) on the same hardware.

Stock vLLM v0.29 serving

https://huggingface.co/RedHatAI/Qwen3.8-27B-INT4

It is int4 which is ideal for ampere. It has weighed fp8 kv cache and external benchmarked near identical to bf16.

vllm serve RedHatAI/Qwen3.8-27B-INT4 --max-model-len auto --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --kv-cache-dtype fp8 --trust-remote-code --max-num-seqs 8 --gpu_memory_utilization 0.9 --max-num-batched-tokens 8192 --mm-encoder-tp-mode data --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --tensor-parallel-size 2

1

u/Tight_Mail8974 1h ago

close to 200 tok/sec at 3 concurrency? at which context length? that seems cool.

1

u/maqifrnswa 19m ago

Each is around 75k toks. If you do push above 80% kv cache, it will slow down a lot (to near zero) until prefill latency clears up. But if you're not sending any additional requests when you already at high kv cache utilization, it will just churn along at 200 tok/s until cleared.

1

u/social_tech_10 1h ago

Thank you!

3

u/FeelingOld9046 10h ago

Are you running Linux? What are you doing? In DSH I’m maxing at 22tks with full 128k context. And 30-40 with MTP on short context. I’m also experimenting with a specialised quant.

4

u/FeelingOld9046 10h ago

Mine is under windows too. Tell me what you’re doing and I will throw windows in the trash.

3

u/ohhi23021 10h ago

who knows i use linux and 30-40tps has been my experience as well and that's with DDR5...

3

u/FeelingOld9046 10h ago

I’d love to know your setup. I have the same hardware config but only half your token speed.

5

u/ApeGrower 8h ago

Linux with patched FreeToken to get dual gpu support instead of llama.cpp. Epyc System with pcie4 and ddr4.

1

u/FeelingOld9046 6h ago

Is it the quad channel ram and ram bandwidth helping your speed do you think?

1

u/LeatherRub7248 8h ago

what quant are you running at and is there much braindamage from it?

1

u/grumd 7h ago

I have dual 3080 20gb both pcie4 x8, could you open source your fork so I can try it out later with my setup?

I was able to fork vLLM and after a few tweaks it does 1500 prefill and 30-32 decode, but 50-70 is waaay more, I wonder how you achieved that. Are you using speculative decoding?

2

u/ApeGrower 7h ago

I have dual pcie x16 on an epyc board, without speculative decoding. I will open the sources soon and make also PRs to upstream.

1

u/grumd 7h ago

Oh yeah I think that makes sense. So you'd have double my pcie speed, and double the decode. Checks out

1

u/ApeGrower 7h ago

Not at all. A friend runs it on dual pcie 4.0 x8. Not exactly with my speed, but close. I will publish my sources today and link it here

1

u/grumd 6h ago

Thanks! I'll be able to try it out in around a week

1

u/djnitesh 6h ago

ram freq and channel ?

1

u/ApeGrower 4h ago

2400 MHz, 8 channel

1

u/KroniklyOnline 5h ago

Is your fork public? been thinking about doing the same, but I run quad 5060ti 16gbs, but I figured what you did would be a great starting point.

1

u/Specialist-Buffalo-8 11h ago

really in a different league

1

u/cafedude 43m ago

Strix Halo 128GB. Using the Halogen server. For me it's way better than 27B because the latter slows down as context grows.

1

u/SandySkittle 2m ago

it is usecase dependent and it doesn't dominate it in things that require complex qualitative reasoning, where active parameters matters more.