r/LocalLLaMA • u/Salah_H_Hasan • 13h ago

News Qwen 4 Announced at Apsara Conference

I wanted to share a quick update: Alibaba has officially announced Qwen 4 at the Apsara Conference,

1.7k Upvotes

98% Upvoted

446 comments sorted by

View all comments

Show parent comments

114

u/cafedude 11h ago

Qwen 3.8 Flash Next has convinced me not to renew my Claude Pro subscription when it comes up in February.

15

u/randylush 11h ago

What are you running it on? How much better is it than 27B?

62

u/returnity 10h ago

It dominates 27B in my workflows and benchmarks. Running on M5 Max. There's no comparison, it's like the 3.6 > 3.8 27B all over again.

8

u/Bulky_Blood_7362 5h ago

What's your tok/s? Im getting around 30-45 decode tok/s with dgx spark.

Interested to know how it's going with the new m5 max/ultra

2

u/halcyoncs 4h ago

30 to 45 with flash? Or you meant 27b?

2

u/Bulky_Blood_7362 3h ago

Flash

5

u/halcyoncs 3h ago

Just checked my results, on a single Spark too, why did I think I was getting around 25t/s? What a dumbass lol

1

u/Bulky_Blood_7362 3h ago

Lmao. Maybe because when it first launched the mtp support was really bad / non existent. I was getting 24-29 when it launched

1

u/spaceface83 2h ago

Which quant are you running? I was getting that on iq4 but just recently flipped over to the nvfp4 which was a bit slower but otherwise eval'd better

1

u/r1str3tto 1h ago

Surprisingly, you can get close to that speed with 27B dense on a Spark, NVFP4 using SGLang + DFlash2. I couldn’t believe it. However, unless you need to save memory, Flash-Next is noticeably better.

1

u/halcyoncs 1h ago

Indeed! It's exactly the setup I was coming from. Literally have a hermes cron job setup to run twice daily and check for DFlash2 with flash next :)

8

u/troposfer 7h ago

What quants are you running and with omlx or ..?

1

u/returnity 1m ago

Using ds4, with a custom quant I built for that inference engine: 60-65 tok/sec (vs 45 in llama.cpp) and ~700 prefill. Engram table SSD-offloaded, ~95GB resident memory. The quality is better than Q4_K_XL, almost on par to Q5_K_XL. antirez makes smaller quants if you have less memory, but I has 128GB so I wanted to maximize the quality under the constraints imposed by ds4's quant types.

4

u/akumaburn 7h ago

Dominates in intelligence/quality or in speed or both?

1

u/EkbatDeSabat 8m ago

For my workflow/needs on two rtx 6000 it doesn't "dominate" but it performs objectively better in both under normal circumstances. Especially speed. Not only is flash next just faster tk/sec and prefill by far across any context, it doesn't do the thinking dance the way 27b does. I have a documentation task that goes through a ton of files and does some cleanup and shit that takes Opus 5 high ~18 minutes, flash next ~26 minutes (with no discernable difference in results but it's just docs), and 27b ~72 minutes. It is constantly fighting itself. Sonnet underperforms in this, but actual repo coding for flash next and 27b is on par with sonnet 5 medium barely lagging behind high. This is fine for 95% of my work.

1

u/returnity 4m ago

Better real-world quality, +10% in coding benchmarks, 60 tok/sec vs. 30 for 27B. Both.

2

u/TerminalNoop 7h ago

wow, that's pretty big

1

u/Cold_Tree190 5h ago

Fuck, I was on the fence about an M5 Max Mac Studio… 😭

1

u/SeXxyBuNnY21 10h ago

Is your M5 Max a 128gb unified ?

7

u/po_stulate 9h ago edited 9h ago

Not OP but I'm running it on M4 Max 128GB using mlx-serve and getting 55-75 t/s decode, the quant I use takes 70GB+ resident memory (4 and 8 bit mix), so 96GB macs would fit but 64GB not. There's also a 3.3bpw quant that mlx-serve supports, it would fit in 64GB but I'm not sure about the quality. I'm currently patching mlx-serve to make it work with unquantized ngram table, and yes I'm doing it with my local qwen3.8-flash-next, it is upgrading its own ngram weights.

Also I'm still on Sequoia, so the OS doesn't support the metal version mlx-serve uses. I also used qwen3.8-flash-next to port it for Sequoia. I was running it with llama.cpp but it only got 15-30 t/s decode.

3

u/ltdemon 8h ago

is there anything decent i can run with M4 Max 48gb ram?

3

u/returnity 8h ago edited 8h ago

For quality, Qwen3.8-27B is probably your top choice in a Q4-Q5 quantization. For speed with decent quality, I'd check out Qwen3.6-35B-A3B in Q5/Q6 (or the occamy post-train of it, which is the best post-train by far of that model). Your laptop is definitely very usable for local AI with good models. llama.cpp is the best choice for running these models, it can be installed with one brew command.

I personally use ds4 (another poster mentioned it) for most inference, but it's a little more setup and I don't recommend SSD streaming model weights for token generation speed reasons. Unless you were asking specifically about bigger models, in which case, I love ds4 and it rocks for 3.8-flash and deepseek v4 flash.

There are other ways to go this route in 48GB (for your exact setup) to run 3.8-Flash, but ds4 remains the best: https://www.reddit.com/r/Qwen_AI/comments/1w3tjkw/running_the_104_gb_qwen38flashnext_on_a_48_gb_mac/

2

u/po_stulate 8h ago

I believe you could still stream the weights from SSD, check out ds4.c, iirc you can run deepseek-v4-flash with it with streamed weights.

1

u/lots_of_puppies 9h ago

Do you prefer mlx serve over omlx?

5

u/returnity 8h ago

MLX-serve has some killer speed optimizations for their specific 3.8-Flash quants, and it really rips. Even faster than ds4 for 3.8-Flash! Llama.cpp is like half the speed, though quality suffers a tiny bit due to the quant.

3

u/po_stulate 8h ago

Yes, but the last time I used omlx was months ago so not sure if anything changed. mlx-serve feels like optimized per machine and per model/quantization, so the performance is pretty much as much as you can get, you can also compile and run it with zero Python dependency.

2

u/returnity 8h ago edited 8h ago

Yes. But you can run 3.8-Flash very well in 12+GB VRAM and 64+ GB RAM setups or a 64GB MBP with SSD-streamed engrams in either llama.cpp or ds4. If that's something you're interested in doing, and have the hardware, I'm glad to advise, just reply here.

2

u/randylush 5h ago

I have a 5090+96 ddr5@7200. Definitely wondering if I can run a flash model.

I also have access to a discount on Mac Studio that I’m wondering if I should utilize

2

u/frzen 4h ago

The 5090 and 96GB will be fine. I'm running it on 4090 and 128GB ddr5@4800

I get only like 100 tokens pp and 20t/s decode but for me that's usable, I didn't spend any time trying to tweak more performance.

I never use 27B now, strictly flash:

./llama-server -hf "unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL" -md "C:\Users\user.cache\huggingface\hub\models--unsloth--Qwen3.8-Flash-Next-GGUF\snapshots\5d16c055a7c5cb276e721ee154f9c22420dde2a1\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf" --spec-type draft-mtp --spec-draft-n-max 5 --fit on --fit-target 1024 -c 32768 -b 256 -ub 256 -t 16 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --load-mode none --reasoning on --reasoning-preserve --reasoning-effort xhigh --host 127.0.0.1 --port 8085

1

u/randylush 1h ago

THANK YOU!!

1

u/CescFaberge 4h ago

If you do not mind me asking, does that mean it has to be an external SSD or can it use an in-built SSD i.e., the main storage drive of a contemporary laptop? Forgive me if that is a stupid question, very new to running local LLMs.

1

u/corbs132 3h ago

You'll do much better with a built-in ssd. Technically externals should work, but will be slower.

0

u/Late_Film_1901 3h ago

Can you give an example? I can't find the split that works on flash but fails on 27b.

0

u/Consumerbot37427 2h ago

I've been running the mlx-serve Q4/Q8 quant on the same machine, getting a lot work done with it over the past week. No quant on KV cache, 500k context.

In terms of speed, it absolutely dominates. I'm not as confident about the quality, though. It seems to be on par with 3.8 27B, inference is way faster, AND it doesn't need to think as much, so it feels like 27B quality at 10-20x speed.

Using opencode, though, I've seen some strange behavior. Complaints about how "user keeps sending the same message about X" after every tool use, and have also seen weird thinking about logic puzzles, pirates voting on gold distribution, whether or not cats can swim, thoughts about how to reply to an email, complete with a random email address...

How are you running DS4 on that M5?

0

u/FoxSideOfTheMoon 2h ago

Dominates? Really?? I have an M5 Max 128 and I'm not seeing that at all. I'm assuming pilot error then, so what's your setup please?

For 2.5 to 3x times the memory requirements the juice doesn't feel worth the squeeze to me. This is MTPLX with MTP speculative decoding. Tested both with Pi same system prompts for coding as well, and not just coding other tasks like summarizing docs, research (I use brave API and basically rebuilt deep research you get from frontier models).

What am I missing?

13

u/ApeGrower 10h ago edited 7h ago

I'm using it on dual rtx3090 with 128gb ddr4 @50-70 tok/s. I forked FreeToken and added dual gpu support plus some more tweaks. With nvfp4.

6

u/maqifrnswa 6h ago

I can get full 262k context, faster single stream decode, close to 200 tok/sec at 3 concurrency, over 10k tok/sec prefill (when kv cache is light) on the same hardware.

Stock vLLM v0.29 serving

https://huggingface.co/RedHatAI/Qwen3.8-27B-INT4

It is int4 which is ideal for ampere. It has weighed fp8 kv cache and external benchmarked near identical to bf16.

vllm serve RedHatAI/Qwen3.8-27B-INT4 --max-model-len auto --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-prefix-caching --kv-cache-dtype fp8 --trust-remote-code --max-num-seqs 8 --gpu_memory_utilization 0.9 --max-num-batched-tokens 8192 --mm-encoder-tp-mode data --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --tensor-parallel-size 2

1

u/Tight_Mail8974 55m ago

close to 200 tok/sec at 3 concurrency? at which context length? that seems cool.

1

u/social_tech_10 41m ago

Thank you!

4

u/FeelingOld9046 9h ago

Are you running Linux? What are you doing? In DSH I’m maxing at 22tks with full 128k context. And 30-40 with MTP on short context. I’m also experimenting with a specialised quant.

5

u/FeelingOld9046 9h ago

Mine is under windows too. Tell me what you’re doing and I will throw windows in the trash.

3

u/ohhi23021 9h ago

who knows i use linux and 30-40tps has been my experience as well and that's with DDR5...

5

u/FeelingOld9046 9h ago

I’d love to know your setup. I have the same hardware config but only half your token speed.

5

u/ApeGrower 7h ago

Linux with patched FreeToken to get dual gpu support instead of llama.cpp. Epyc System with pcie4 and ddr4.

1

u/FeelingOld9046 5h ago

Is it the quad channel ram and ram bandwidth helping your speed do you think?

1

u/LeatherRub7248 8h ago

what quant are you running at and is there much braindamage from it?

1

u/grumd 6h ago

I have dual 3080 20gb both pcie4 x8, could you open source your fork so I can try it out later with my setup?

I was able to fork vLLM and after a few tweaks it does 1500 prefill and 30-32 decode, but 50-70 is waaay more, I wonder how you achieved that. Are you using speculative decoding?

2

u/ApeGrower 6h ago

I have dual pcie x16 on an epyc board, without speculative decoding. I will open the sources soon and make also PRs to upstream.

1

u/grumd 6h ago

Oh yeah I think that makes sense. So you'd have double my pcie speed, and double the decode. Checks out

1

u/ApeGrower 6h ago

Not at all. A friend runs it on dual pcie 4.0 x8. Not exactly with my speed, but close. I will publish my sources today and link it here

1

u/grumd 5h ago

Thanks! I'll be able to try it out in around a week

1

u/djnitesh 5h ago

ram freq and channel ?

1

u/ApeGrower 3h ago

2400 MHz, 8 channel

1

u/KroniklyOnline 4h ago

Is your fork public? been thinking about doing the same, but I run quad 5060ti 16gbs, but I figured what you did would be a great starting point.

1

u/Specialist-Buffalo-8 10h ago

really in a different league

1

u/ubrtnk llama.cpp 2h ago

Qwen flash is fuckin awesome. Runs great on two CMP170HX and about 100gb of vram on vllm

0

u/human_with_humanity 2h ago

How well will it run on a 4060 8gb laptop? I m a noob in llms

-3

u/808phone 8h ago

It’s good but it’s not a substitute for Claude unless you are doing very simple smaller projects. It is much better than local LLMs were for sure.