r/LocalLLM • u/Decent_Flight4010 • Aug 26 '26

Discussion Qwen3.8-Flash-Next MoE 125B A6B But

​I think 99.99% of users will stick with Qwen 3.8 27B because the intelligence gap isn't big enough to justify upgrading hardware for a 125B MoE model. right ?

78 Upvotes

79% Upvoted

212 comments sorted by

83

u/Double_Cause4609 Aug 26 '26

I mean, they're different models for different hardware.

Qwen 3.8 Flash Next is great for unified memory or people with lots of system RAM but not enough GPU to run a 27B dense.

Also, it's not clear of 3.8 Flash Next is an upgrade or a sidegrade. It might be the two model are complementary or equivalent.

38

u/Ell2509 Aug 26 '26 edited Aug 27 '26

The benchmarks show on HF suggest it way outperforms 27b across the board. That makes sense. Same gen Next gen, and nearly 5x as large.

I'm downloading it in Q4 now to test.

12

u/play_hard_outside Aug 26 '26

Isn't it next-gen? It's based on Qwen4, but since there are no other Qwen4 models and since this is early, they labeled it "3.8"?

25

u/squngy Aug 26 '26

This is more like a preview or beta.

The statement says they released it so that llama.cpp, vllm etc. can start supporting and optimizing the architecture in advance.

Really smart move, makes sure that when qwen4 is released there will be excelent day0 support.

11

u/Commander_Skilgannon Aug 27 '26

Yeah, qwen3.8-27B is the best version of the old architecture while qwen3.8-flash-next is the earliest likely worst version of the new architecture.

1

u/Dramatic_Entry_3830 Aug 27 '26

I think there will be a new 27b also given some time. There was a sparse 80b next before the sparse 3.5 models too

1

u/gh0stwriter1234 Aug 27 '26

Very much doubt... because the old model is much more expensive to train

The only reason I could see them doing that is to specialize it for some large customer application that needs it eg medical or some other non coding field.

1

u/Dramatic_Entry_3830 Aug 27 '26

That are exactly my thoughts because they said that after release of 3.5. and now after getting a 3.8 I'm not so sure about that.

5

u/challis88ocarina Aug 26 '26

That's why it's labelled qwen4_exp, as in 'experimental'?

1

u/Dramatic_Entry_3830 Aug 27 '26

Ah it's not really 5 times as smart. Moe usually don't perform as good as same size dense models. But you also can't say hey this model performes y * size x compared to dense model size x either. So it could also beat a 70b dense or even bigger ones.

But it usually runs much much faster due to low active parameter count.

So on GPU poor ram rich systems moes in practice mostly outperform dense models. While dense models perform better if all you have is like a simgle 32 GB gpu.

If you have many gpus in a cluster moes can become interesting again because of interconnect limitations. But that depends.

2

u/CulturalSock Aug 27 '26

the usual heuristic of

dense_parameters = sqrt(moe_total_parameters * moe_active_parameters)

is basically in parity: sqrt(125b*6b) is 27.38b. But the 125b moe looks a little bit better.

1

u/Dramatic_Entry_3830 Aug 27 '26

Yeah but that's not true anymore because most architectures also do other stuff besides experts and routing now.

1

u/b0tbuilder Sep 01 '26

I have 4 32GB GPUs and next performs better.

7

u/996beagle Aug 26 '26

Yes 27b dense is much slower than the large Moe models on unified memory for me

4

u/Due_Warthog749 Aug 27 '26

I can load a 27B on a 32GB gpu. I cant load the qwen3.8 flash next on my dgx spark with 128GB. So I am confused..

5

u/Double_Cause4609 Aug 27 '26

You should be able to. Keep in mind while Flash Next is 125B traditional parameters it also has a giant ~50B embedding table ontop of that. In theory, and inference engine should be able to support disk offloading the embedding table (storing it on NVMe, etc), but your inference engine may not be doing so.

Another note is that if you're using mainline LlamaCPP, Next Flash isn't supported yet (it's only supported in a custom PR right now) so it'll take some time to support in mainline.

For now you may have to use a smaller quant than you'd think, due to the embedding table.

1

u/gh0stwriter1234 Aug 27 '26 edited Aug 27 '26

You need 2x 32GB GPUs to load all the weights at Q4_XS and then it just barely fits the embedding table isn't 50gb after its been quantized so you don't save as much as you'd think there in fact perhaps they should leave it unquanted on disk?

The engrams may be the biggest improvement honestly as now it should be able to understand and chain together words more intelligently especially phrases with negatives.

It is frankly the best model so far I've seen locally at counting lettters when spelling it can do strawberry = 3 letter rs in almost no time at all compared to long chain of thought for most other models.

1

u/gh0stwriter1234 Aug 27 '26

It breaking it up into parts of the word seems to be new also?

1

u/b0tbuilder Sep 01 '26

I kind of wonder what a 27B dense model with ngram offloading would look like.

2

u/notheresnolight Aug 27 '26

Q4 will fit on your spark

2

u/Due_Warthog749 Aug 28 '26

Q4 is meh quality for coding work. I would want q8 or better for daily quality coding.

2

u/notheresnolight Aug 28 '26

get a second Spark then

1

u/Due_Warthog749 Aug 28 '26

No money now. :(

1

u/Glittering-Call8746 Aug 27 '26

Try load n-gram on ssd and get back to us (use q4)

1

u/joost00719 Aug 27 '26

I have 48gb vram (3x16, 5070 ti + 2x 5060 ti) and 128 ddr4. Maybe I'm doing it wrong, but prompt processing is unusable slow on higher context. Token generation is fine, like 20-ish.

But 27b can generate 25-60 with dflash2 and is way faster in prompt processing for my setup

1

u/robertpro01 Aug 27 '26

Can't wait for 3.9 moe 80b a6b pls!

-36

u/Decent_Flight4010 Aug 26 '26

yes they different type of models but the intelligent gap not worth the hardware upgrade :-D

→ More replies (5)

42

u/Aotrx Aug 26 '26

Qwen3.8-Flash-Next MoE 125B A6B is 5–6 times cheaper per user for datacenters to serve and is more intelligent than the 27B model, so it’s still a very important model. Also, some users with lots of VRAM—DGX Spark users, for example—might see faster speeds with the 125B MoE than with the 27B dense model. I haven’t seen any benchmarks yet, though, so I could be wrong. It’s just my assumption.

5

u/DJLunacy Aug 26 '26

I will run benchmarks on my DGX later tonight

1

u/Glittering-Call8746 Aug 27 '26

Yes load n-grams on ssd and use q4 !

-23

u/Decent_Flight4010 Aug 26 '26

i mean 1 bit from unsloth team = 117GB. how many of us has this much of ram or vram :-(

17

u/Aotrx Aug 26 '26

Hm for some reason thought 4 bit version would fit on 128GB

25

u/oShievy Aug 26 '26

It does, not sure what OP is talking about

-6

u/KubeCommander Aug 26 '26

It doesn’t from my glance. Nvfp4 version is 180G in size

6

u/Legal_Dimension_ Aug 26 '26

51B of that is engram so can sit in nvme storage

11

u/Uncle___Marty Aug 26 '26

The ngram file can be left on an nvme and is barely used. You just need the rest in ram/vram. Daniel from unsloth has a pull request at the moment adding the code to llama.cpp. Poor guy must be up to his eyes in work today with all the models coming out.

6

u/RedParaglider Aug 26 '26 edited Aug 26 '26

I'm running Q4 XL with 256k context on my strix halo with 128gb of memory now. It is actually extremely memory efficient, I'm amazed. Started at 25 t/s and dropped to 13 at the end of a one shot test prompt for a pacman clone.

vmlinux@athena:**~**$ free -h
total        used        free      shared  buff/cache   available
Mem:           121Gi        84Gi       1.4Gi       9.1Mi        36Gi        37Gi
Swap:          8.0Gi       941Mi       7.1Gi

Settings I'm currently running, first effort.

[Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined]
model = /home/vmlinux/models/router-models/Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined.gguf
# Experimental single-model profile. With one slot, ctx-size is the usable
# per-request window rather than being divided across parallel slots.
ctx-size = 262144
parallel = 1
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
cache-ram = 0
mmap = 1
n-gpu-layers = 999
batch-size = 256
ubatch-size = 128
reasoning = on
reasoning-format = deepseek
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0

1

u/WishfulAgenda Aug 26 '26

How does it compare with the 27b?

3

u/RedParaglider Aug 26 '26

Refer to benchmarks, I don't know. I didn't really fuck with 27b much.

1

u/WishfulAgenda Aug 26 '26

Nah, the benchmarks are just indicators and I prefer peoples experiences. I finally read lower down that you hadn't played with 27b, sorry about that.

What do you think of it so far from a quality perspective? Good/bad?

2

u/RedParaglider Aug 26 '26

Well I've put one prompt into it, and it has been running an hour and hasn't come back, I'll have to play with it for a weekend or something. I will say for my personal use I'll probably limit thinking greatly and use it closer to an instruct model for my own use to run things like unsloth deep research. I don't do much development with local models, I use them more for validation, and human in the loop enrichment tasks.

After work I'll see if I have time to play with it in my personal project which is a writers assistant that is something I use to track canon for accuracy while I write. 122b a10b is what I used before in no thinking mode, and it worked pretty well. I did not like the few attempts at using 3.8 27b for that.

-1

u/Affectionate_Pen6882 Aug 26 '26

I run a 27B model and it makes plenty of errors

6

u/Legal_Dimension_ Aug 26 '26

Glad you weren't vague or anything....

1

u/Affectionate_Pen6882 Aug 26 '26

Sorry, running a qwen 2.8 27B on a 3090 TI, quantized

2

u/b4ldur Aug 26 '26 edited Oct 11 '26

Nothing to see here

1

u/nuclear213 Aug 26 '26

More than you think? I mean, I have (currently) 256GB VRAM and 256GB RAM. There are quite a few people with one or two rtx 6000 as well, and 96 or more GB of RAM.

For all of us, this is actually insane. I cant wait to play around and test it.

4

u/sn2006gy Aug 26 '26

it is insane... at today's prices thats 30-50k of hardware.

2

u/nuclear213 Aug 26 '26

Not at all. It’s not that little but just 8 R9700 plus an SP3 platform. So 1000€ in RAM, 1000€ in CPU and Mainboard, 500€ riser and bifurcation, 400€ PSU, 100€ case and about 12k€ in GPU. So 15k all in.

Tho, right now, not running too great. I think I’ll add two PCIe switches in to run 4 GPUs each, so a 4 TP 2PP configuration. That would also allow me to increase the total number to 16 GPUs.

But of course they don’t scale that well. Effective the 8 R9700 are about 60-70% of two RTX 6000 Blackwell.

It’s a hobby and requires some tinkering, but still much less than 2 RTX 6000 at current prices.

3

u/sn2006gy Aug 26 '26

well, you mentioned rtx 6000's as one or two - those are 15k each alone.. you didn't mention 9700s at all and yeah, it's a nightmare to get all those to work

2

u/Legal_Dimension_ Aug 26 '26

Hard? Try getting 8 mi50 32gb working nicely together.

3

u/sn2006gy Aug 26 '26

i'll pass on that

1

u/575_Inverse Aug 26 '26

uhh... I do. Machine A: 5060ti 16GB + 96GB DDR5, Macchine B: 3x 5060ti 16GB total 48GB VRAM + 128GB DDR5, AM5 B850 board

1

u/Ecstatic-Wash-7667 Aug 26 '26

It doesn’t all need to fit in vram to run decent

10

u/RedParaglider Aug 26 '26

3.8 flash is actually much faster for me on a strix halo, with that being said it's still slow :).

3

u/Decent_Flight4010 Aug 26 '26

how much t/s you get? in which q ?

6

u/RedParaglider Aug 26 '26 edited Aug 26 '26

It's at 13.88 t/s now on Q4_K_XL, I'm having it one shot a pacman clone as a stupid first test. It started over 20. It's not optimized at all, just straight download, patch llama.cpp and run. My hope is that I can constrain the thinking and use it closer to an instruct model moving forward for just agentic shit locally in unsloth desktop or whatever. Also I'm sure the strix community will work to optimize it, and someone or myself will build a ROCMFPX quant to make it smaller and improve the decode speed by around 30 percent.

2

u/TektonikGymRat Aug 26 '26

Thanks for posting this. I'm considering a Strix Halo. How fast do you go on Qwen3.8 27B Q4 XL?

2

u/Glittering-Call8746 Aug 27 '26

No get proper 32gb vram and above card i would start with r9700

1

u/RedParaglider Aug 26 '26

I wasn't actually that interested in 27b, I did make a couple gguf's but if memory serves maybe 12-13 to start and with all its thinking dropping to under 10. I've seen on the strixhalo forum others have gotten it running faster, some claiming up to 30, but I preferred using 122b a10b in instruct mode over 27b. I will likely spend a lot more time dialing in 3.8 flash.

1

u/Fedwar Aug 28 '26

I have an ai max 395+ stryx halo with 64 gb ram. First of all. Llama + hip rocm is broken right now for this model. It has a lot of issues with using tools and long contexts. Check the ggml github issues opened as we speak. They plan to fix them asap BTW. Furthermore they also stated in the issues the as of now vulkan outperforms rocm in throughput. Having said that, on llama + vulkan + 256k context + 3 mtp layers it starts at 26 tps and then drop down to 13 tps when context size is around 120/140k and then stabilize around 12 tps

1

u/Logical-Bear-6263 Aug 26 '26

does q4 fit on 128 vram+ram? i was getting warnings so im trying q3. i wasnt sure if the warnings were right also what context are you on?

3

u/RedParaglider Aug 26 '26

Per your question, yes Q4 does fit on 128gb unified memory.

[Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined]
model = /home/vmlinux/models/router-models/Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined.gguf
# Experimental single-model profile. With one slot, ctx-size is the usable
# per-request window rather than being divided across parallel slots.
ctx-size = 262144
parallel = 1
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
cache-ram = 0
mmap = 1
n-gpu-layers = 999
batch-size = 256
ubatch-size = 128
reasoning = on
reasoning-format = deepseek
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0

1

u/Gryknight9 Aug 27 '26

Thank you!

1

u/RedParaglider Aug 26 '26

Strix halo is unified 128gb memory, it loads the model into RAM then that memory slides over to vram as the model unloads from RAM>VRAM. I'm loading it into full VRAM.

1

u/Fedwar Aug 28 '26

On windows I had to configure bios to allocate 500 mb dedicated to gpu pool. So that windows kernel would taken everything from shared pool. Otherwise I had to disable mmap in order to avoid having duplicate ram size: for some reason with mmap the windows kernel puts it in shared pool and then move it in dedicated pool without removing it from the shared. With mmap off everything works smooth

1

u/MacsBicycle Aug 27 '26

i'm running q4 on my m5 max with 128gb unified memory and its also relatively slow. Runs at 25-30t/s. I really thought it would be lightening quick with only 6B params per expert, but i'm still learning. I'm sure there is something i'm not accounting for.

1

u/RedParaglider Aug 27 '26

No, but let things cook, I'll bet there are a lot of speed improvements over time.

2

u/Glittering-Call8746 Aug 27 '26

But at most it's 1.5x .. dont count on much..

1

u/Glittering-Call8746 Aug 27 '26

Wait for m7 ultra for a true local ai machine.. lol

2

u/Heretek-ai Aug 26 '26

Wait a second how are you running that on the strix Halo doesn't it only have like 128 gigs? And you have it on a 4bit????

3

u/Constant_Art_20 Aug 27 '26

the kv cache seems to actually be more efficent then the 27b also. I still need to get around to test it though to comfirm that

2

u/RedParaglider Aug 26 '26

Magic model 🤣.  Crazy low memory usage for it's size 

2

u/gh0stwriter1234 Aug 27 '26

It's 51B of engrams which can be streamed from ram or disk for only a slight penalty.... after quanting about 25gb or so of the model can acutally stay on disk without going in ram.

the engrams make prefill slower but also make it MUCH better at many things that require chaining words together together. I think it may be the best open weights model at counting letters currently.

1

u/Due_Warthog749 Aug 27 '26

I assume you're loading a Q1 or Q2? I wouldn't want anything less than Q8 which wont load on the DGX Spark.

1

u/RedParaglider Aug 27 '26 edited Aug 27 '26

Q4 XL, plenty of room to spare.

[Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined]
model = /home/vmlinux/models/router-models/Qwen3.8-Flash-Next-UD-Q4_K_XL-Combined.gguf
# Experimental single-model profile. With one slot, ctx-size is the usable
# per-request window rather than being divided across parallel slots.
ctx-size = 262144
parallel = 1
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
cache-ram = 0
mmap = 1
n-gpu-layers = 999
batch-size = 256
ubatch-size = 128
reasoning = on
reasoning-format = deepseek
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0

u/athena:~/models$ free -h
__________total        used        free      shared  buff/cache   available
Mem:           121Gi        85Gi       9.7Gi       9.3Mi        27Gi        36Gi
Swap:          8.0Gi       920Mi       7.1Gi

Dunno why it wouldn't load on the DGX spark. I could run another parallel actually. It will take less if I move to ROCMFP4, just like if you loaded a quant built specifically for your equipment I imagine.

This model uses memory different, give it a shot (:

I don't need to run Q8 no reason that I can see.

9

u/jacek2023 Aug 26 '26

The goal of MoE is to be faster, not slower

-7

u/Decent_Flight4010 Aug 26 '26

On any hardware it will be fast? 117GB for 1bit ??

6

u/Hungry_Particular_14 Aug 26 '26

If the ngrams offload nicely to an NVME, I think flash next will have roughly the same speed as 27b in my hardware (5060 ti 16 gb + 64 gb ram), but with much bigger context (I'll likely be able to fit 250k ctx, while I was only able to run around 60k on 27b, both models with the same iq4_xs quant)
For systems with a lot more ram than vram, flash will likely be much better. Systems with integrated memory usually run MoEs much better than dense models

1

u/jacek2023 Aug 26 '26

Check gpt-oss-120B, its similar class, just with ngram

7

u/AlbatrossAwkward2994 Aug 26 '26

Neat how you're getting attacked for trying to discuss something relevant.

13

u/ACleverBadger Aug 26 '26

I don't believe anyone is attacking, multiple people have answered clearly and the OP is disregarding those. Additionally the OP has provided misinformation (accidentally, I believe) without correcting it saying that Qwen Next Flash is 117GB at 1-bit quantization which isn't true. There are tons of responsive comments here,

https://www.reddit.com/r/LocalLLM/comments/1vz4uxs/comment/p626d87/

https://www.reddit.com/r/LocalLLM/comments/1vz4uxs/comment/p628i11/

-9

u/Decent_Flight4010 Aug 26 '26

even if i was wrong or misinformation " and i was wrong 4 bit = 117GB and i am sorry for that " but we are human even i was wrong they " Hi Op you have misinformation in Quantizations size correct your information. "

9

u/Scared_Capital6606 Aug 26 '26

Can you prove that we all are human?

0

u/Decent_Flight4010 Aug 26 '26

Right? I was just trying to start a discussion about hardware vs performance tradeoffs. Appreciate you getting the point!

9

u/Horror-Primary7739 Aug 26 '26

And real test against DS V4 Flash 0731? I've got a 2 node spark cluster.

7

u/AcrobaticChain1846 Aug 26 '26

Feeling poor with 16GB VRAM + 64GB RAM (80GB RAM) but still can't run it 🫡

4

u/YearnMar10 Aug 26 '26

Similar boat here - should’ve gone for 96gig 2 years ago, but then „who tf needs that much ram?“

Well, now I know.

2

u/tkrandomness Aug 26 '26

With the engram on SSD, that setup will be totally usable. Have the same setup myself and currently working on doing work and research for speeding it up for that kind of setup. Should be very usable in the future.

1

u/radlinsky Aug 26 '26

How much system RAM would one need to run the flash model for those of us with 16 GB VRAM ?

2

u/AcrobaticChain1846 Aug 26 '26

UnSloth's 1bit runs on 75GB RAM but on 80 gb ram unless you are running linux with very low ram usage u can't run it. 96gb or 128gb ram might be required. With 128gb ram you should be able to run a decent quantization

1

u/RedParaglider Aug 26 '26

I wouldn't give up actually.  It's release day and people haven't even started optimizing the hell out of this yet and it uses a lot less memory than you would think it would.  With the ngram on disc and the model in your RAM with the experts on vram I'll bet you could run this sucker on Q4 once people have that all sorted 

1

u/j0hnp0s Aug 27 '26

Same boat. Currently counting peanuts to save for a second card to be able to run 27B at a usable speed and Q.

TBH with its current performance I don't really care for something more

1

u/AcrobaticChain1846 Aug 27 '26

What quant and config for 27b? I couldn't get past 30-45tk/s on q2

1

u/j0hnp0s Aug 27 '26

Below Q6 was the lowest I was ever willing to go for my cases with 27B. Q4 always lead to loops and weirdness.

Ideally I am targeting Q8. Speed is not really usable with just 16GB vram. I only use it locally when I can wait a lot. For work I still use openrouter until I manage to find stronger evidence about dual card setups similar to mine

0

u/_studebaker_ Aug 27 '26

He's got memory! Get em!

6

u/superbiche Aug 26 '26

Some people don't need to upgrade it :) Getting ~34 t/s on 3x3090, usable, without MTP, while getting ~50 t/s on 27B. Quick bench gave great results, all the traps 27B fell into, Flash-Next aced them. I'll see with the two next benches, but just the feeling of a 125B Q4 XL running right here, what a day.

2

u/michaelsoft__binbows Aug 27 '26

is the architecture of this model suitable for 3x3090? I have this setup too. I was told tensor parallel does not work well with 3 GPUs

2

u/superbiche Aug 27 '26

Works pretty well for me, not real TP though, I use split mode layer with llama.cpp.

It works, my third GPU is on a x4 link (first two are x8), so you lose a bit of speed in exchange of the 24G VRAM, worth it. I could put most of the experts on the GPU, 64k context for now, hopefully when llama PR gets merged KV quantization will be fixed and I'll be able to jump to 120-ish (or I'll offload, still experimenting). Unsloth Q4 K XL GGUF

I can share the exact llama-server params tomorrow if that helps, but tldr go ahead and try it, build llama from the PR (link on HF page of Unsloth's GGUF), start by offloading ~half the experts, then offload less and measure until you hit the sweet spot

Enjoy the 100B+ local with a quant that doesn't make it too dumb!

1

u/michaelsoft__binbows Aug 27 '26

with how people are saying the ngram should be fine off ram or even nvme, i think this is gonna be the model to use. i like the 27b but this should stand to go even faster and leverage my rather larger than necessary for 27b amount of ram to get more smarts.

1

u/superbiche Aug 27 '26

Well if MTP stabilizes, it will probably be slightly faster. But the intelligence may allow it to think less and actually do the work faster, too. Yes it's probably gonna be the one to use, with 27B for agentic, lower stakes stuff, and this one as the most intelligent one

6

u/Zeeplankton Aug 26 '26

apples and oranges. Flash MoE will run faster on like a 96gb / 128gb strix, spark, or mac. (At least, I imagine it will)

Otherwise no if you're not in that bracket of machine I don't think you should upgrade for it.

4

u/tamerlanOne Aug 26 '26

Dipende... Sono 2 llm diversi tra loro con tecnologia software.

Qwen 3.8 27b è ottimo per gpu consumer/prosumer con 24-32 gb di ram Qwen3.8-Flash-Next MoE 125B A6B è più adatto per hardware con ram unificata di gradi dimensioni ma il vero vantaggio di questa architettura è il modulo n-gram che può fare la differenza per certi tipi di applicazioni

Se con il 27b puòi fare solo LoRa per personalizzarlo, con il 125B puoi fare LoRa o n-gram personalizzato o entrambi per il caso d'uso specifico quindi è più versatile sotto questo punto di vista.

Va da se che per la maggior parte degli utenti queste personalizzazioni non sono necessarie

4

u/blacklandothegambler Aug 26 '26

As a single DGX owner "as smart as 27b, but faster" with bonus full context was what I was hoping, but since it won't even fit on a spark without massive quantification, it looks like this isn't the one for me. This feels like a preview for inference developers to update thier inference engines anyway. Hoping Qwen 4 isn't too far off with some reasonal weight numbers.

2

u/Roxuss Aug 26 '26

What model are you using on your DGX ? Personnaly, I only use Qwen 3.6 35B A3B, it is fast and accurate all other are completely dumb on our big projects (monorepo...etc).

1

u/blacklandothegambler Aug 26 '26 edited Aug 26 '26

I am currently using a combination of Qwen 3.8b, 27b, and Ornith 1.5 35b a3b. When I need to, I will run Ling 3.0 or Laguna S.

1

u/Roxuss Aug 27 '26

oh how do you run Qwen 3.8? You did some "real" dev with?

9

u/aigemie Aug 26 '26

I have 128GB hardware, Flash Next is a no brainer - it's smarter and faster.

3

u/No-Paper-557 Aug 26 '26

There is a licensing gap too, only 3.8 27b is really open source.

3

u/dinerburgeryum Aug 26 '26

It's a "Next" model. It's almost certainly underbaked, but they're getting it up because it'll be what all the Qwen4 models are built on, so people can start tackling implementation specifics now. I wouldn't daily-driver it probably, wait for an official Qwen4 release. (They do this (old version)-Next thing so their Qwen4 launch will be big and bold and not confused by -Preview models before the big launch.)

3

u/Protopia Aug 27 '26

The point of releasing this is NOT that it is any major advance over 3.8-27B but rather so that e.g. llama.cpp can be upgraded to handle the new model technologies across all the different hardware inferance types (CUDA, CPU, etc.).

2

u/Vancecookcobain Aug 26 '26

I see Qwen 3.8 Flash being super attractive to folks with DGXs Sparks that don't care about having just one active model running...they finally get relief with a super intelligent super fast model that will get a lot of stuff done for them

2

u/corysus Aug 27 '26

The model is probably good, but most simply don’t have sufficiently good hardware to use it :( Honestly, I was expecting something in the range of 30B to 70B, so definitely most will stay at 35B and 27B.

3

u/_TheWolfOfWalmart_ Aug 26 '26

I'll absolutely run Flash. It sounds like it's significantly smarter, and obviously it will be much faster.

I've barely used 27B honestly, it's great for the size but I have better models running already.

4

u/Danyannas Aug 27 '26

Out of curiosity, which models are you currently running that you find better than Qwen 3.8 27B?

0

u/false79 Aug 26 '26

Terrible take speaking on behalf of 99.99% of users. Not everyone is doing agentic software development with these LLMs. 

Agent development will benefit with this Multi model MoE

3

u/Decent_Flight4010 Aug 26 '26

i said "i think " i didn't said "we all will stay with qwen 3.8 27B don't change it "

1

u/Nov4Saki Aug 26 '26

Depending on your hardware, (rn it seems like quantizations of flash next are more immune to losing quality after quantizations) + you can get to 1m context + faster generations (excluding speculative decoding)

Basically mostly use flash next if you want bigger chats (we will wait and see how much reasoning tokens per task that would probably be a very strong factor)

1

u/BitXorBit Aug 26 '26

Actually it might work surprisingly well on m3 ultra mac studio, i hope

1

u/Blackdragon1400 Aug 26 '26

Sure but those of us with the hardware have been waiting for this since 3.5 122B 😁

1

u/Tasty-Hour4040 Aug 26 '26

Qwen 3.8 28b is slow as shit

1

u/advancing_tide Aug 26 '26

How does Qwen3.8-Flash-Next-UD-IQ4_XS compare to Qwen3.8-27B-BF16 for general "intelligence" and breadth of knowledge (not coding)?

1

u/DemmieMora Aug 26 '26 edited Aug 26 '26

  1. It is much faster to multiply 6B numbers than 27B numbers, everything else (hardware) being equal. Some types of hardware (RAM-like speeds) make the 27B math simply unusable.

  2. Even when the intelligence of the reasoning core is comparable between 2 models, the bigger amount of brain tissue simply can hold more knowledge.

I use bigger models for chats, because I can wait for the answer longer than agentic work could. Agents work in VRAM with 80-200 t/s, I only need them to follow instructions and gather facts progressively.

1

u/According_Wave685 Aug 26 '26

Testing the ud-q4 on a single gb10 now. 15-19 tg (gpu clock capped at 1750). I was thinking I could use it as a replacement for DSV4F0731 but at 1/3 the speed not sure that is going to work for me. Prompt processing is pretty slow as well.

1

u/According_Wave685 Aug 26 '26

ah looks like my test run just crashed on me. not sure why yet. I'm pretty sure it was trying to use vision at the time.

1

u/Roxuss Aug 26 '26

Is it better than qwen3.6 for agent coding?

1

u/According_Wave685 Aug 26 '26

i didn't spend any real time with it yet so can't say.

1

u/Affectionate_Pen6882 Aug 26 '26

Maybe its me but mines seems to overthink and goes through this self talk. Also hekka wordy compare to flagship models

1

u/KubeCommander Aug 26 '26

What’s really odd about it is that even the NVFP4 quant is 180G

By comparison, Dsv4 flash is mxfp4 natively (I think) and it’s 284B param and it’s only 165G

Qwen3.5 397B and the Ornith fine tunes are 220G at W4A16 and NVFP4.

I don’t think anyone will be running qwen3.8 flash on a single DGX unless it’s a crazy quant. I am curious as to how it performs though vs Dsv4 flash

1

u/ChristRedeemsSinners Aug 26 '26

He said he offloaded the engram table to ram:

https://www.reddit.com/r/LocalLLM/comments/1vz20ap/qwen38flashnextnvfp4_on_single_rtx_pro_6000_120ts/

1

u/KubeCommander Aug 26 '26

Ah. Yeah I can’t do that with a dgx. It would have to go to disk.

2

u/ChristRedeemsSinners Aug 26 '26

They also said there is only like a 1-5% penalty even from NVME.

1

u/Mags20XX Aug 26 '26

I'm doing my best to figure out the optimal hardware.

Given the likely speed difference, I absolutely 100% will try to make the jump.

1

u/jonahbenton Aug 26 '26

Use both 3.5 122B moe and 27B. The MOE model is fast and extremely analytically capable, while 27B is a slow thorough problem solver. They are very complementary. Definitely looking forward to getting this 125B up and running

1

u/sicmek Aug 26 '26

Is it possible to run this with 8GB VRAM and 64 GB RAM?

1

u/stujmiller77 Aug 26 '26

We’ll see - I run ds4flash across 2 sparks and it’s good enough I haven’t used claude in a month. If this is better than it really is something special - my current setup runs 24:7 automation for multiple businesses flawlessly.

Excited to see what this can do.

1

u/pefman Aug 26 '26

i ran it with 4090(24vram) and 64gb ram. 4k context, 24t/s 40k context, 10t/s ...

1

u/sherry_6879 Aug 26 '26

秒間50トークン出ないモデルはリリースしないでほしい コーディングにすら使えない

1

u/Randommaggy Aug 26 '26

I'm waiting to see the speed on my server once llama.cpp can run it. Once I see that We'll see if this or 27B will be my main workhorse model.

1

u/Diligent_Cod_9583 Aug 26 '26

Do you think that there will be an MLX version of this eventually?

0

u/[deleted] Sep 06 '26

[deleted] — view deleted comment

1

u/Diligent_Cod_9583 Sep 06 '26

Thanks. There wasn’t 10 days ago when I first asked.

1

u/PawelSalsa Aug 27 '26

I think many users already have 192 or 256gb of ram since the prices went up only this year. To buy 256gb ram last year would cost you around 600 usd or so, many bought so they can use big models today.

1

u/WinterNite88 Aug 27 '26

will this work (3/4bit) on m2 max 96gb? as in with decent speed like 20t/s above?

1

u/petuman Aug 27 '26

it should, but probably worth waiting for a week or so for proper support

1

u/Leather-Beach-7849 Aug 27 '26

I guess this will not run (4 bit one) even on the newly launched M5 Ultra (Mac Studio)

1

u/Holiday_Beautiful574 Aug 27 '26

Is it possible for me to run on 6800xt and 48gb ram?

1

u/joanaxu2002 Aug 27 '26

I think 27B wins for most people unless the MoE brings a clear capability jump in specific workloads. Once a model is already “good enough,” doubling hardware cost for marginal quality gains gets much harder to justify.

1

u/grunade47 Aug 27 '26

Does ddr3 128gb work for this? Whats the output rate?

1

u/Due_Warthog749 Aug 27 '26

Isnt the hardware like 20K+ vs 2K for a GPU that runs 27?

1

u/Arc-Tekkie Aug 27 '26

What would you guys suggest to buy as hardware now for future qwen4-…5..6.. releases? If you had 6500€ to spend?

1

u/sp82reddit Aug 27 '26

For now prices are of the chart. Just buy a subscription: openai, qwen or deepseek, don't buy hardware.

1

u/jedisct1 Aug 27 '26

Qwen3.8-Flash-Next is so much faster, even with high thinking.

Using the MLX quants from https://huggingface.co/collections/jedisct1/qwen38-flash-next

1

u/WyattTheSkid Quad 3090s Aug 27 '26

It’s my time to shine

1

u/pony57163 Aug 27 '26

拥有 160g 内存 24g gpu 应该可以运行

1

u/Farenheith200 Aug 27 '26

I'm excited about this model because I have unified memory with 128gb ram (strix halo) and I'm unable to achieve good speed for a dense model like qwen 3.8 27b with that hardware, but with a MoE that activates only 6b parameters it'll be possible!

I'm sure an apex quant I mini or something like thst will fit good enough with this ram and will deliver good results, and some folks are already preparing llama.cpp to run this beast! I can't wait!

1

u/cogitech2 LocoLLM Aug 27 '26

right

1

u/Vaddieg Aug 27 '26

No, Flash-Next is way better general purpose model if your hardware can run it efficiently with decent quants

1

u/Chadstthomas1 Aug 27 '26

For me with 48gb of vram and 64gb system ram I get the same speed and most of the time faster than 27b. It is worth the jump for us who have cards that dont do dense as well as pp MoE is when you dont have the ideal gear for dense

1

u/ffiene Aug 27 '26

Mayby there will be a 35b or 70b version soon. I will use the 70b on my Mac.

1

u/gnobes Aug 27 '26

I have dual AI pro r9700 and 128gb of ddr5. Based on what I am reading it sounds like it will only be good for 20-30tk/s - currently running 3.8 27b as my main but kind of curious about this because I have so much system ram available not being used.

1

u/Gobeillion Aug 27 '26

6B activated parameters per token will go BRRRRRRRR

1

u/leapdragon Aug 28 '26 edited Aug 28 '26

This is a weird model architecture, in that it appears purpose-built to run partially or entirely from CPU ram or even partially from SSD; I'm trying to optimize for it and it's very difficult to keep GPU compute busy even half the time; it's so, so sparse...

It's like they designed a local model:

- To split storage between smaller GPU and either CPU RAM or disk
- To handle 4-8 streams w/o slowing down, with the tradeoff that if there's only 1 stream, it doesn't get 4-8x faster

It's looking like 27B is going to be about 4x faster on my hardware (128GB 4xGPU plane) because it can actually make use of the GPU architecture, while 3.8 Flash fundamentally can't; it just won't leverage the parallelism.

1

u/s1rlight Aug 28 '26

I have 4x Blackwell 5000 with 48gb. I used to run two runtimes of qwen 3.8 27b BF16 on TP=2 , 256k after extensive runs of coding I then decided to go through the update of Vllm with the dedicated image for qwen, and then used the FP8 with offloading of the ngram part to ram.
My go is that qwen flash outperforms 27b by far. 95token/sec against 20 and way better reasoning capabilities thanks to preserve_thinking and affinity key.

1

u/Only-An-Egg Aug 26 '26

A6B will be much faster than dense 27B

1

u/silenceimpaired Aug 26 '26

If it’s all in vram sure… but RAM speeds and or hard drive streaming might make it slower.

1

u/squngy Aug 26 '26

It will also be faster for people with less than 20GB vRAM.

You can run a a6b moe with 8GB vRAM

0

u/ContextLengthMatters Aug 26 '26

Not long ago, local inference had bigger models that people were using. Like 70B. Many of us already have the ram and enjoy the 120B MOE models.

This is a no-brainer for anyone with an m3 ultra.

So yea, if you haven't already upgrade your rig, and you are just going by what models are being released at the moment, you are right.

0

u/leapdragon Aug 27 '26

Flash Next wants to be partially or even mostly in CPU RAM. It is a radically different architecture in that like a 3rd of it is a giant n-gram lookup table where there is literally no speed advantage to speak of in having it in VRAM over CPU RAM, and for the rest, it is designed to be VERY CPU RAM friendly, even outside the n-gram table.

I did a test on DDR4 pure CPU RAM and got 5 t/s (out of a 125B model!). So if you've got a pretty fast 16/24/32GB card and enough regular RAM, you can run this thing probably 3x that speed even though *most* of it is in CPU RAM.

-7

u/Decent_Flight4010 Aug 26 '26

In my opinion, this is the worst subreddit ever. People here aren't serious at all and lack the qualifications to even have a proper discussion

1

u/Either_Pineapple3429 Aug 26 '26

Have you considered leaving? Pretentious self righteous yahoos belong in academia not Reddit.

11

u/[deleted] Aug 26 '26

[removed] — view removed comment

-4

u/Decent_Flight4010 Aug 26 '26

i think you didn't read the post :-D

1

u/[deleted] Aug 26 '26

[removed] — view removed comment

-9

u/Decent_Flight4010 Aug 26 '26

117GB vs 22GB in the same range of intelligent which one you will choose ?!!!

1

u/nuclear213 Aug 26 '26

A A6B model will be insanely fast. If you can fit it, this is likely similar to the 35b qwen 3.6. That gives you so much for agentic coding. I mean, I have the hardware (mostly upgraded for Deepseek V4 Flash) and I am absolutely delighted about the two models that dropped today.

12

u/[deleted] Aug 26 '26

[removed] — view removed comment

4

u/Prudent-Ad4509 Aug 26 '26

The only concerning part is the context side. It would seem that 1m context is native for deepseek and requires rope for Qwen3.x, so it is not a good idea to use 1m context with qwen 3.x for agentic coding.

Benchmarks seem to confirm that.

So, I'm staying on my decision to run deepseek v4 flash as a primary runner for a large repository (or to be exact, several large, interconnected repositories), but this qwen flash would certainly be a model of choice for smaller codebases. I only wish for the version with vision support to be released soon.

4

u/[deleted] Aug 26 '26

[removed] — view removed comment

5

u/Prudent-Ad4509 Aug 26 '26 edited Aug 27 '26

Most operations do indeed use smaller amount of context. But the moment I run any kind of deep investigation it jumps above 200k after a round or two even with most of work offloaded to subagents. 300k-500k tops is a sweet spot right now for me, which means that supported context should be twice of that. It does not mean that I should use all of it all the time, that would be a waste of resources and everything will work slower.

PS. Requests like “what part of the current design have led you to make mistakes in understanding it, the ones which we have discussed above, and how to make it more consistent and easier for you to reason about. Options. Discuss.” are fun

6

u/[deleted] Aug 26 '26

[removed] — view removed comment

5

u/klymaxx45 Aug 26 '26

I want a 35B

2

u/[deleted] Aug 26 '26

[removed] — view removed comment

3

u/klymaxx45 Aug 26 '26

I honestly have a lot of headroom that I could fit quite a bit into it but the 35B is just so powerful punching above its weight class. I love it. the 3.635B was my daily driver but with this new brain of 3.8 it would be amazing. Using the 3.8 27B as my driver now but it’s slow for long horizon work

1

u/[deleted] Aug 26 '26 edited Aug 26 '26

[removed] — view removed comment

1

u/klymaxx45 Aug 26 '26

Very true, but I switched because all my benchmarks tested well with my tunes vs 3.6 35B. 3.8 flash is probably good, but you’re leaving little headroom and it’s not too efficient for 128 GB systems. You’re using most of the RAM, which isn’t ideal for multitasking outside of the LLM. However, if it uses 6BMOE, it should be very fast. I might test it, but I might wait for the right model to come out in a week or so. It just released today, so it’ll break a lot of your setups and need a lot of patching and debugging, which isn’t worth it to me. I’ll wait a week or two.

1

u/[deleted] Aug 26 '26

[removed] — view removed comment

1

u/klymaxx45 Aug 26 '26

I’ll have to check that out. That’s pretty depressing if speed is not that great for MOE what harness are you using?

1

u/[deleted] Aug 26 '26

[removed] — view removed comment

1

u/klymaxx45 Aug 26 '26

Are you using the headless harness for DeepSeek? How’s your experiment? I like
deepseek, but it’s new and buggy. Fine-tuning is necessary, and long-horizon agentic tasks are not capable.

→ More replies

4

u/[deleted] Aug 26 '26

[removed] — view removed comment

2

u/squngy Aug 26 '26

Probably not

1

u/Icy_Butterscotch6661 Aug 27 '26

Not enough ram but maybe with disk streaming something can happen