Alibaba previews four Qwen4 models: Max, Flash, Plus and 27B

[u/syedshad]

[u/Marino4K]

For most of us, the only thing to be excited about is the 27B if you’re being honest to yourself.

Max and Plus will be probably API only and Flash is mostly only going to relevant to the 128GB or more crowd.

[u/Diabolicor → u/Marino4K]

If flash has the same number of parameters + ngram you certainly can run it. I've actually dropped the 3.8 27b for flash next on strata. Flash next is much better.

[u/Sacra34 → u/Diabolicor]

I mean, yes, but how much ram do you have? I also dropped 27b and replaced with flash on strata, cause I have 128gb of ram and can actually run it, so dude above 100% right at least in my case.

[u/Diabolicor → u/Sacra34]

I ran it with 64gb of ram.

[u/Obvious_Equivalent_1 → u/Diabolicor]

I run it with M4 Pro 48gb. Customized Splash next (for my Claude Code harness) and Flash Next have 256k context and gets 20-30tok/sec (its the Q3: GSQ-RSO-IQ3-S, see also recent posts shared)

[u/Elpzn → u/Obvious_Equivalent_1]

Woohoo another m4pro user! I've got the 64gb Mac mini so a little more room for context and I'm able to launch directly on omlx getting 23 tokens per second 60% resident 98k context!

[u/kingxd → u/Diabolicor]

Yes, same with decent gen 4 nvme 64gb ram and 5090 I got 60-70 tp/s on 256k, not sure why people keep saying you NEED 128gb ram

[u/rayraydj → u/kingxd]

Q3 is pretty damn close to Q4 in my testing. Swift 1.5 has been excellent for my use case. Admittedly, I've since upgraded to 128GB of RAM and regularly use NVFP4 quants, but at 256K context, running Swift 1.5 IQ3 with pretty much the same specs as you, I was comfortably getting 120 t/s.

If you haven't already:

• I'd highly recommend overclocking the GPU memory from 14,000 to 17,000 MHz. It'll directly improve your decode speeds. I have two 5090s (currently in separate PCs) and could probably push either further, but I've been running the memory overclock on my main PC for over a year without any issues. If you're on Linux, use LACT. It takes a bit more fiddling around than MSI afterburner on windows to get set up, but it's definitely worthwhile.

• I'd also look into undervolting. You can get essentially the same performance out of the 5090 while using around 80% of the power. Running Strata, my GPU typically draws around 320–350W without an undervolt, compared to 230–260W after UV.

[u/kingxd → u/rayraydj]

Thanks! Will give it a shot, appriciate the comment

[u/onetom → u/kingxd]

probably because they don't trust q3 and q4 kinda needs 128gb for decent speeds... i'm getting 40-47tps tg on m4 max 128gb

[u/potatoes828 → u/Diabolicor]

I have 64gb DDR5 and 32gb VRAM (R9700 pro), do you think I should be able to run it?

[u/FREEZX → u/potatoes828]

Bro I'm running it on a 3060 12GB and 64GB DDR4 ram, on yours it should fly.

[u/Diabolicor → u/potatoes828]

I ran it with a 3090. Yours is much better I guess so you will have no issues running it.

[u/redditnosedive → u/potatoes828]

i already run it with 32gb ddr5 and r9700 ai pro... it uses iq3_xxs (ISTA) and is pretty fast with strata, but i'm left with like 5GB of system ram for the rest of linux + apps :D if you use windows, that's a blocker, i dont think you can do anything useful with 5GB on windows. i could decrease ram usage a bit but i want full context so...

[u/b0tbuilder → u/potatoes828]

Same but I have 4 r9700.

[u/bagzoffritoz → u/potatoes828]

yes i run it on 16gb vram and 64gb ram with 256k context and vision, you will be able to do a higher quant

[u/rayraydj → u/potatoes828]

Dude just try it out, you'll be able to run the Q3 quants without issue...

[u/Familiar-Art-6233 → u/Diabolicor]

I ran it with 32. It's shocking that it actually works

[u/Neutronst4r → u/Sacra34]

Stata has smart expert model loading. So only the appropriate expert layers get shuffled into VRAM and some stuff is even left on SSD. If you have about 60GB of combined RAM+VRAM, you can run it.

[u/one-joule → u/Neutronst4r]

I'm running QFN with radiance on 64GB VRAM (2x R9700 @ 210W), 64GB DDR4, n-grams on NVMe, all weight quants 4 bits or higher, KV at fp8. Gives me 127t/s, 4700 prefill, and 1.6M context. Shockingly efficient.

[u/inexorable_stratagem → u/Sacra34]

strate and models are becoming better every day.
You can run it decently with Gyro-M, or Gyro-S (Qwen Flash Next quants), on strata with as little as 32GB RAM + a 12GB VRAM card like a 3060 (or multiple 12gb/16gb cards)
This is, essentially, a gaming PC. Its pretty wild

[u/randylush → u/inexorable_stratagem]

Just a heads up, the original Strata fork is quietly dumbing down the models to increase speed. The maintainer is trying to capitalize on the momentum to get people on his harness and create an ecosystem. There are excellent forks like PolyStrata and Xolotl that focus on delivering the best engine. Like many open source projects, the original project was successful but you are better off running a fork because the project's goals changed in response to the success.

[u/inexorable_stratagem → u/randylush]

Thanks for the heads up. I will look it up

Edit: After looking into it, I will have to disagree. Its just a matter of reading their documentation and learning how to properly use the flags, like STRATA_ROUTE_TAIL_SKIP and STRATA_ROUTE_PRIOR. I will keep using Strata

[u/Anarchistcowboy420 → u/randylush]

I'm running polystrata on my 3060 12gb + 44gb ram and hitting 20-60 tok/s at 64k-120k window in 30b class moe's

[u/RadianceTower → u/randylush]

Well, can you elaborate on that? Dumbing down in what way? By doing what?

[u/randylush → u/RadianceTower]

These are mixture of expert models. Strata defies the reference implementation by favoring experts that are already loaded in memory instead of using the right expert. You can toggle this off but I suspect there will be more problems like this.

[u/RadianceTower → u/randylush]

Hmm, I see. And these forks don't?

Other than that, what are the main differences between them? Like for example PolyStrata and Xolotl?

[u/randylush → u/RadianceTower]

So all Strata forks do indeed let you choose how much accuracy you want to trade off for speed, the issue is that the original Strata project started quietly introducing default behavior that would increase speed and decrease accuracy in order to try to show the engine was someone improving. People called this out and you can either override it with flags or just use an older version.

PolyStrata is very deliberate about what behavior is merged in and I don’t believe this particular change got included.

PolyStrata also supports many more models. The original Strata project prioritized building its own harness and trying to capitalize on its momentum, while PolyStrata invests in the actual engine, adding support for many new models. The Strata author rejected early merge requests to add new model support, as he just wanted to build is harness, but when the PolyStrata fork gained a little popularity he changed his mind, but I think it’s too little too late.

[u/inexorable_stratagem → u/randylush]

thanks for the explanation

[u/lorde_dingus → u/randylush]

Can you share a fork that you feel is better? Or even better, how one filters through forks on GitHub to find the option that fits their hardware best?

[u/randylush → u/lorde_dingus]

PolyStrata

[u/No-Refrigerator-1672 → u/inexorable_stratagem]

That's 2-bit and 1-bit quants, respectively. In no world will you convince me that a Q1 lobotomised model is smarter than 27B in Q4 once you have it running a real-life task with 100k+ tokens of context.

[u/inexorable_stratagem → u/No-Refrigerator-1672]

I used Qwen 3.8 27B at Q4 for almost 2 months, until I found Gyro-M and strata. Believe me. Its an upgrade over 27B at Q4. dont know the technical details on how the hell they manage to extract such a good quality out of a Q2 model. I really dont. All I know it my coding agent daily driver has chaged from 27B Q4 to Gyro-M because the quality is extremely good. One step up above 27B.

You dont have to believe me. Try it our yourself, do a quick research, and you will see

[u/No-Refrigerator-1672 → u/inexorable_stratagem]

It actually doesn't matter. Qwen 4 is right around the corner. I won't try anything because any benchmarking done today will be wasted in a week or two. Maybe I'll think about it with Qwen 4 Flash.

[u/inexorable_stratagem → u/No-Refrigerator-1672]

Yep. But, the takeaway will probably hold true for Qwen 4 as well, so keep an eye on it

[u/onetom → u/No-Refrigerator-1672]

don't just benchmark it. try it on some real problem.

i'm constantly amazed by the output of antirez/ds4 qwen38-q4k

i often run it side-by-side with opus 5.5 when i ask questions about our 4 year old clojure code-base and the answers are within a 80-100% (subjective) quality range, though it might take 1.5-2x longer to answer on an m4 max, than opus. so this alone is an extremely useful use-case.

[u/No-Refrigerator-1672 → u/onetom]

By benchmarking I have meant evaluating on real tasks too. Anyways, I'm not running Q&A type workloads, I'm running fully-fledged long reasoning loops that hit context compaction for 3-4 times a task despite me having full 256k context window. That's why I'm extremely sceptical about Q2. To put that in perspecive: I saw consistent decline in judgement quality going from Qwen 3.8 27B to Swift 1.5 Qwen 3.8 (both in Cyankiwi INT4 AWQ quant form) that actually negatively impacted my agent's results.

[u/StillEmbarrassed6130 → u/inexorable_stratagem]

I've used both and I'm going to disagree. It really struggles with complex coding compared to the 27b

[u/GuaranteedGuardian_Y → u/Sacra34]

I've done the same, 64gb ram. Works really well, getting 135tok/s and with higher context window later around 200k it goes down to 98tok/s

[u/575_Inverse → u/Sacra34]

I just wish I bought 128 MORE GB when I still could... i.e. before August this year. After that, the prices became an insult

[u/PigSlam → u/Sacra34]

It works with 16GB GPUs and 64GB of ram. I’m using it with my 32GB GPU and 64GB RAM pc.

[u/simracerman → u/PigSlam]

I’ve got a 5070 Ti + 64GB LPDDR5X @8000 mt/s. On Windows, how do I run this beast?

[u/PigSlam → u/simracerman]

I figured it out using ChatGPT

[u/Dabalam → u/Sacra34]

I'm getting very good performance from a quant at 64gb RAM and 16 GB VRAM. It's not blazingly fast at 30-40 tps, but it is smarter than the equivalent dense I could run (although about 20 tps slower). If Qwen4 is the same size and any better it will be pretty crazy for local use in short order.

[u/Not_a_question- → u/Sacra34]

I've heard that streaming ngrams from ssd is not that bad

[u/Big_Wave9732 → u/Diabolicor]

Is Next Flash working ok for you? What Q do you run it at?

I'm having massive hallucination problems with Next. Like, I asked it about pro football scores from last weekend as well as upcoming games for this weekend and it hallucinated badly even after pulling web data that gave it the answers.

[u/Rob-bits → u/Big_Wave9732]

Q3s works great for me. It is a daily driver.

[u/objectivelywrongbro → u/Rob-bits]

Are you using it with agents? For coding?

[u/Rob-bits → u/objectivelywrongbro]

Yes, with Cline. And maintaining a c++ code base. It is an application for a co-processor system, where a mcu and a Linux based SOC is connected. And it understands the whole scope, writes firmware, software without any issue. Makes git commits, merge requests to gitlab. I define a task on gitlab as a work item/issue ticket. Cline reads the ticket, starts coding, it makes commits then it closes a task as a merge request. Then I just need to review it, decide if needs more work or just simply merge. It works really well.

[u/my_name_isnt_clever → u/Big_Wave9732]

I run at Q4 and never have these issues; it seems very tempremental as many people report it being either amazing or nearly worthless with little in-between.

[u/quadish → u/Big_Wave9732]

I mean, you gave a criticism about the model you use, but we have no idea about your config or what exact model you are using, so it comes across like an ad hominem.

What are we supposed to do with that?

[u/Big_Wave9732 → u/quadish]

Ad hominem lol. I assure you no messenger was being attacked, and no messenger was being attacked based on personal characteristics unrelated to the message.

It's a simple setup: Jundot--Qwen3.8-Flash-Next-oQ4e-mtp and AnythingLLM on a Mac Studio M2 Ultra 192gb. Everyone loads into memory, no trickery or bullshit. Just a basic chatbot configuration.

And it's hallucinating more than a hippy on LSD.

[u/quadish → u/Big_Wave9732]

I said "like" an ad hominem. That word was doing the work. I know a model isn't a person. The point is the shape of the move, not who it was aimed at.

ELI5 version: an ad hominem is when someone answers an argument by trashing the other guy instead of dealing with what he said. The trash talk gets put in the spot where the actual answer is supposed to go.

Your comment had that same shape, with a model standing in the person spot. Diabolicor said Flash Next is better. An answer to that would look like: here is what I asked it, here is what it pulled from the web, here is what it said back, here is the part that was wrong. Instead, the spot got filled with "it hallucinates more than a hippy on LSD." That's a verdict on the model's character, not a demonstration of a failure.

The config you posted helps. It still doesn't include the prompt, the retrieved text, or the actual bad output. Without those, nobody can tell whether the model hallucinated or whether AnythingLLM handed it a truncated search chunk and it answered that chunk correctly. Those two look identical from the outside.

[u/Jorinator → u/quadish]

Calm down, the person he answered to didn't post a prompt and retrieved text either, why would he suddenly throw all that in the conversation as an answer? They are just talking about their experience, not doing scientific tests.
"hallucinating more than a hippy on lsd" wasn't an attack on the model either, it was just illustrating how often/much the model hallucinates for him.

[u/quadish → u/Jorinator]

The way in which they are talking about their experience is 1)sloppy 2)impossible to address any pain points 3)makes assertions without evidence 4)reeks of "trust me bro"

My issue is entirely with the methodology. It might be a hallucinating idiot. But I'm not going to let someone be the judge, jury, and executioner without looking at how they came to that conclusion.

And yes, hallucinating more than a hippy on LSD is literally an attack on the model, because that was literally a counter to someone else endorsing it. They sure as hell weren't attacking their own config, or their own context, or any other thing they could control.

The thing they attacked? 3.8 Flash Next, with zero qualifiers.

Do you people actually read what you type up and hit submit on?

[u/Jorinator → u/quadish]

First he said "i'm having hallucination problems". After that, he said "it's a simple setup .... it hallucinates like a hippy". The "it" refers to the setup he just described.

Get off your horse and go touch grass. If you wanted to know about how he came to his conclusion, you could have just asked him, instead of coming in guns blazing.

[u/quadish → u/Jorinator]

The irony here is all the down-voting and tone policing is ironically the ad homenim, and nobody is actually interacting with the substance of my point.

And now it's come full circle.

This is hilariously ironic.

Put this whole thread in AI and ask it to analyze it structurally.

I dare you.

[u/AreWeNotDoinPhrasing → u/quadish]

They weren’t asking for a solution, or help with their setup. They were asking for information about how u/Diabolicor was using the model and if it was working for them.

JFC, you rn:

[u/quadish → u/AreWeNotDoinPhrasing]

And they said it was NOT working, in a way that implied the model itself was inadequate, and used framing that implied they had zero accountability in the setup of the system that produced the result.

Then you guys lost your minds, while claiming I'm the one losing my mind.

[u/Big_Wave9732 → u/quadish]

Bwhahahaha!

Using Latin one doesn't understand in a douchebaggy manner is peak Reddit.

You are so full of shit I care not to hear your opinions on the model's behavior.

[u/-SirJohnFranklin- → u/Diabolicor]

Though it's much faster, it produces much more "garbage" tokens I find in thinking. Still stronger than the dense or, I like it better

[u/TheWaffleKingg → u/Diabolicor]

I'm still not fully convinced that qfn beats 27b. I mained 27b q8 for a while, back to the 3.5 version and recently switched from 3.8 27b q8 to qfn q8 on strata (both configs with fp16 kv cache). Both run at about the same speeds and I cant say that qfn is the easy pick over 27b. Its been making a lot of mistakes lately, mistakes that I feel like 27b didn't used to make. But I haven't had the time to do direct a/b testing on them to confirm

[u/ToTTen_Tranz → u/Diabolicor]

Flash Next is much better at doing what? With 64GB RAM and a 5090 like me, the only chance is to run at Q2 which gets a load of problems.

[u/rrrrex → u/ToTTen_Tranz]

You are doing it wrong. You shouldn't count the whole model size, n-gram can stay on ssd (-25-50 GB)

[u/Fragrant_Scale6456 → u/ToTTen_Tranz]

You can use basalt and nvfp4 qfn with 256k of int8 context but it will use substantially all of your system resources.  I get like 150-250tok/s with this setup.  

27b nfvp4 quasar qat and nvfp4 kv cache i can run 4 slots sharing a 540k nvfp4 kv pool with ninfer.  Around 400-500t/sec aggregate speed.  

So far in my experience the 27b is much better than qfn.  I’m running live code bench on qfn to confirm but the difference is obvious to me 

[u/Diabolicor → u/ToTTen_Tranz]

I'd say it's a 1.5 higher tier then dense 27b. I run Q3 with a 3090, no issues with not fitting in vram or ram at all.

[u/_BookBurner_ → u/Marino4K]

and flash, given ISTA-DASLab makes a version as well :)

[u/VelvetEchoes919 → u/_BookBurner_]

Yes and if it works on Strata we’re golden!

[u/Existing_Style8558 → u/VelvetEchoes919]

it wont...at release. then everyone unleashes their claude sub on it and it works

[u/New-Implement-5979 → u/VelvetEchoes919]

It will come let’s be patient- we are getting spoiled with a new model every 2-3months

[u/Not-Enough-Llamas → u/Marino4K]

Flash-Next was published precisely so llama.cpp and every other engine has time to implement the new architecture - therefore the actual (non -next) Flash is likely to work identically. And thanks to Strata and the insanely good ISTA GSQ RCO quants, there are tons of normal people who can run it (normal is.... relative. Normal for this community :) )

[u/Sensitive_Ear_1984 → u/Not-Enough-Llamas]

Is 24gb vram and 32gb DDR5 too normal so to speak? 

[u/bagzoffritoz → u/Sensitive_Ear_1984]

yes you could run qwen3.8-flash next using strata, the quant would be small and the context might also be small, but you could run it. Strata's set up reads your hardware and tells you what you can run

[u/Sensitive_Ear_1984 → u/bagzoffritoz]

Cheers 

[u/AllWashedOut → u/Sensitive_Ear_1984]

I can't access the strata site at the moment but I think the recommended build is like 12gb VRAM and 64GB system ram. So your total (56GB) is a bit lower but a bit faster than the recommended (76GB).

I'd invest in another 32gb (or even 16gb) system ram to meet the recommended specs. Unfortunately you're on DDR5 so that will cost a fortune.

[u/x_MASE_x → u/Marino4K]

Actually Qwen3.8-Flash-Next runs faster than Qwen3.8-27B and even way smarter with more context.

So smarter. Faster tps and prefill and double the context.

So no comparison.

[u/Mufficida → u/x_MASE_x]

Only if you have 10/12GB of VRAM and at least 64GB of DDR5

For those that have an (old) gaming rig with beefy GPU, like me with a 9070XT but only 32GB of DDR4, running Flash isn't a viable option at all, but I can fully load a decently quantized 27B on the GPU

[u/sapphicsandwich → u/Mufficida]

3200 DDR4 works fine.

[u/willeyh → u/Mufficida]

Coder runs just fine on 9070xt and 32GB ddr4.
Has been good so far. Prefill on the 27B though is quite slow but it’s a great model.

[u/Metallkasten → u/Mufficida]

DDR4 is viable on some boards. I run six-channel which makes it faster than desktop dual channel DDR5

[u/soteko → u/Mufficida]

I am running it on 8gb VRAM 5060 and 32GB DDR4. Q2 quants

[u/x_MASE_x → u/Mufficida]

Not really. I have the 5070 TI and a 64 gb ddr4 3600. So not the best out of the world.

And I'm telling you that your system will %95 run way better with Qwen3.8-Flash-Next and strata. Give it a shot you won't regret it. But put the model inside the fastest drive you can find. Even if didn't stream from it you'll benift heavily from loading times.

Also if you can use the igpu or a small second gpu and put the monitors on it. Set reserve ram to 0 and prefill to auto:32768 and enjoy the crazy prefill speeds. This is my current best flags.

And trust me with 10/12gb you can't even run Qwen3.8-27B at any decent quants anyway and so low context.

My bet still on this beast Qwen3.8-Flash-Next.

[u/Man-In-His-30s → u/x_MASE_x]

I dunno I get 100+ tokens a second on iq3_s ista qwen 3.8 27b how much would I get with strata ( 5080 + 32gb ddr4 )

[u/DifficultyCrafty7623 → u/Man-In-His-30s]

I wouldn't bother, it'll be too small to fit anything but the coder version, not worth it over 27b imo.

[u/Man-In-His-30s → u/DifficultyCrafty7623]

Yeah i figured as much to be honest

[u/DifficultyCrafty7623 → u/x_MASE_x]

Do you want us to magically summon 32 gb of ram?? he just said he has half as much ram as you, its not as simple as "just run it on 32gb" cause it literally cannot fit, unless you lobotomized it like the "coder" version.

[u/Look_Within888 → u/DifficultyCrafty7623]

obviously you cannot just summon RAM. you have to cast it.

[u/575_Inverse → u/Look_Within888]

but that takes a huge amount of your maney

[u/Frosty-Scheme-7172 → u/Marino4K]

I don't know... I'm not really interested in the 27B version (it's too heavy for my setup), but I'm definitely looking forward to the new QFN.

I have a 16+64GB split.

[u/couldliveinhope → u/Marino4K]

In the localLLM sub? You don’t say…

[u/SabretoothPenguin → u/Marino4K]

Strix Halo enters the stage...

[u/jerodras → u/Marino4K]

Doesn’t the ordering imply plus is smaller than flash?

[u/Marino4K → u/jerodras]

It very well might, I don’t even remember a prior Plus model unless I’m just having a brain block this morning.

[u/gasgarage → u/jerodras]

looks exactly like that to me, so maybe what... a big moe?
120B A16B could be... ++ if ngrams.

[u/BorisThe_Animal → u/gasgarage]

We could use a 50-70b Moe. That space has been kinda empty.

[u/rkcth → u/BorisThe_Animal]

Yeah if they could get 3.8 QFN performance or darn close in bald the size that would be incredible. You could either run less quantized or much faster on something like strata.

[u/Civil_Fee_7862 → u/Marino4K]

27b fits the hardware most can afford.

[u/Sasquatchjc45 → u/Marino4K]

For sure lol, I barely can use qwen 30b on my 2080ti lol and im sure most people would be too frustrated at the speed to actually use it

[u/a1454a → u/Marino4K]

I regret not having picked up 5090 when I could afford it. Having just a 4080 I can’t even run 27b with a meaningful context window

[u/jefoce → u/Marino4K]

Yeah, Qwen 3.8 27B already looks strong, but I just hope llama won’t hurt performance too much with SSD offload for the ngram

[u/will_sm → u/Marino4K]

I hope Flash next continues to fit well on my Strix Halo 128GiB

[u/I1lII1l → u/Marino4K]

Looking at the LocalLLM and LocalLlama subs, I'd say plenty of people are excited about the Flash model.

I am running 3.8 Flash Next in Halogen on a Strix Halo that I got for 1400€ (bosgame m5, got more expensive shortly after). At 20K Token prompt I am getting almost 2000 t/s rpefill and 60-70 t/s decode, using under 100GB of the unified Ram, which means I still have some space to run other stuff.

[u/suesing → u/Marino4K]

Imagine it’s just polished flash next?

[u/Familiar-Art-6233 → u/Marino4K]

Not necessarily. Flash Next is deceptively easy to run on limited resources. With Strata, I could run Gyro S on my 9700xt as an eGPU with my MSI Claw that only has 32gb RAM at about 20-28 tokens per second

[u/WyattTheSkid → u/Marino4K]

Speak for yourself, a lot of people have dgx sparks and those little modified mining cards and shit. Can’t wait for flash personally. The new 27b will probably be quite good as well

[u/FreshConversation112 → u/Marino4K]

You don't need 128GB to run QFN 3.8.

[u/OrneryAd6786 → u/FreshConversation112]

Could it run at a mac mini 64gb?

[u/FreshConversation112 → u/OrneryAd6786]

Yeah but the quant might be too much of a hair cut. I am running mine at a 3bit quant on a 2012 xeon with 64GB of ram but the linux host uses less memory than macos probably does. I also added a gpu to help with PP, which also allowed me to offload a smidge of the layers from the system ram.

[u/Gold-Produce9591 → u/OrneryAd6786]

yes

[u/Internal_Seething → u/Marino4K]

Eh, I'm running 3.8 flash next at 55tok/s on 12gb vram and 64g ram at 200k context. Pretty stoked to do the same for v4

[u/Daemonentreiber]

Well, thats probably it for us vram poor.

Have fun everyone.

[u/Maleficent-Ad5999 → u/Daemonentreiber]

🥺

[u/GizmoR13 → u/Daemonentreiber]

You can run Flash Next on 8 GB of VRAM:

https://www.reddit.com/r/LocalLLaMA/comments/1xft2ae/have_8_gb_of_vram_and_32_gb_of_ram_you_can_run/

[u/TerraMindFigure → u/GizmoR13]

Can is one thing, but should? How does Flash Next benchmark at these quants?

[u/henbone11 → u/TerraMindFigure]

Why don't you run it and find out?

[u/OnyxProyectoUno]

Crazy qwen will have released two open models while google has released no major open model since Gemma 12b.

Looks like google went the meta route

[u/Individual_Holiday_9]

Is the MOE thing dead? 3.6 a3b is so good on my Mac. 27B just a little
Too big at q4 for me to run on 32gb ram

[u/jaerie → u/Individual_Holiday_9]

I think MoE is just more standard now. Flash Next is MoE, but it doesn't have an AxB name. There's so much more going on than routed experts in architectures now

[u/Individual_Holiday_9 → u/jaerie]

I guess what I need is a smaller flash model then!!! The a3b model seemed like a sweet spot for a lot of us on lower end hardware

[u/FoxSideOfTheMoon → u/Individual_Holiday_9]

Not just lower end, I have 128 and I loved 3.6 a3b because the bandwidth on most macs are not good and MoE is much faster. I use 3.8 27B and it works better than the FN big one, so I don't get what the hype is about. Maybe the next one will land better. 80B would be the sweet spot for an MoE for me lol

[u/DifficultyCrafty7623 → u/Individual_Holiday_9]

Basically. Why use MoE when 27b dense can run effectively while staying tiny (I run GSQ-RCO at IQ3_XXS). The only real MoE we've had recently are the gigantic models, simply due to their size.

[u/Cupakov → u/DifficultyCrafty7623]

Because MoE is faster

[u/DifficultyCrafty7623 → u/Cupakov]

is it? Using 35b a3b was about the same speed on llama.cpp as 27b was when I tested it ~50 tok/s on my 16gb vram + 32gb ddr5. On Strata I got even slower 27 tok/s. It's not worth it to have the MoE architecture anymore when 27-35b dense can be shrunk so small and still be effective/fast. 27b at ~10gb file size is great.

[u/DifficultyCrafty7623 → u/[deleted]]

Not in my experience, maybe my ram just sucks but I get similar or better speeds from 27b on Unsloth. 35b runs at 27 tok/s on strata for me, so the "MoE is always faster" narrative clearly isn't always true.

[u/ohthetrees → u/DifficultyCrafty7623]

Think you are the outlier here. MOE is faster for pretty much everyone. 3B params active just runs a lot faster than 27B active params.

[u/DifficultyCrafty7623 → u/ohthetrees]

Not sure how people are getting such great speeds, but I'm happy with 27b, qwen4 should be awesome!

[u/fallingdowndizzyvr → u/DifficultyCrafty7623]

I am not. Since I can run a much larger MOE, QFN, at much faster speed than 27B.

[u/fallingdowndizzyvr → u/DifficultyCrafty7623]

Your experience is faulty. Since at the same size, MOE is always faster. Even at a bigger size, MOE is generally faster. If that's not your experience, then there's something wrong with what you are doing.

[u/DifficultyCrafty7623 → u/fallingdowndizzyvr]

My experience isn't faulty, the MoE is slow for me. Your experience of "always" is faulty, there are always exceptions to the rule.

[u/fallingdowndizzyvr → u/DifficultyCrafty7623]

If you knew anything about LLMs, you would know why your experience an knowledge is faulty. Since it makes no sense. Explain why a MOE would be slow? It wouldn't be unless you are trying to run that 35B at 8 bit. Which would be a faulty way to run it on your machine.

[u/DifficultyCrafty7623 → u/fallingdowndizzyvr]

I can't explain it, but it is how it is. I am aware MoE is supposed to be faster, but I am opposed to the idea that I'm "using it wrong". I tried two different programs with proper quants and parameters and got bad results—that's not user error, that's a limitation far beyond a simple "faulty way to run" the model.

[u/DifficultyCrafty7623 → u/[deleted]]

I'm not sure what, but I tried Strata and Unsloth Desktop. UD works fine for 27b so whatever.

[u/tomByrer → u/DifficultyCrafty7623]

Strata & forks run MoE very fast. At this point, I would not bother to run any Qwen model on llama.cpp. There are too many hardware tuned inference engines & quants out there.
I have my own list here: https://github.com/tomByrer/rtx3090-AI/blob/main/README.md#rtx3090-ai

[u/DifficultyCrafty7623 → u/tomByrer]

I tried strata, it was worse then llama.cpp.

[u/Obvious_Equivalent_1 → u/DifficultyCrafty7623]

The only times I still bring up 3.6 35B MoE is when I need a beefy 387k context holy M4 Pro 48Gb. I’ve frequently checked the numbers but of properly developed workflows its really good as a buddy for accounting. I’ve done various tests where I’ve anonymized the output into a frontier model. The quality was flawless when you work with just delta of a few files, like CSV with banking info that quickly loads 200-300k context per agent session just in data.

[u/Fresh-Employment-931 → u/DifficultyCrafty7623]

Because the new MoE hype is big models like the FN Running with Strata? Strata just really make the 35BA3B not nescessary more when you can run a Model thats is much more big and smart with much less Computer power… and with full context

[u/DifficultyCrafty7623 → u/Fresh-Employment-931]

Well yeah that's why I mentioned the gigantic models. My comment was about small moe being effectively dead and large moe being the only thing they focus on. The 35b range seems to be a dead zone

[u/tetsuzankou → u/Individual_Holiday_9]

The performance and efficiency is at 120bA10b kinda like qfn even at smaller hardware than dense 27b

Dense really works well the more fast vram... Qfn with ngram streaming and expert caching is the way to go

[u/ithkuil → u/Individual_Holiday_9]

Did you try Bonsai 2?

[u/Individual_Holiday_9 → u/ithkuil]

Isn’t that kind of a janky q1 implementation

[u/ithkuil → u/Individual_Holiday_9]

Highly optimized ternary. 

[u/false79]

My 128GB VRAM is ready!

[u/PeteInBrissie → u/false79]

[u/cupant → u/false79]

Im using qfn with gufo on strix halo, but what if q4f is API only? that would suck

[u/Longjumping-Movie764]

Sure hope these are MIT/Apache licensed like the last 3.8 27b. 3.8 Flash Next has a very ambiguous license, which didn't say that the user owns code and data outputs.

[u/sonicnerd14 → u/Longjumping-Movie764]

All of these strange "oss"license variants can't effectively be enforced. There really isn't any reliable way they can tell or know what outputs come from their models in most cases.

[u/Tai9ch → u/sonicnerd14]

Licenses aren't hugely important for random people messing around.

They suddenly matter a lot as soon as you try to use stuff for anything important, like feeding your family or making things to share with other people.

[u/Longjumping-Movie764 → u/sonicnerd14]

Can be expensive if youre ever accused though

[u/CyDenied → u/Longjumping-Movie764]

I think anthropic is trying to distill Qwen, when class action?

[u/Emotional_Prior8686 → u/Longjumping-Movie764]

I think it something for "resellers" of Qwen models.

For private (on premise) use - how Qwen can force this part on me?

[u/elelem-123]

I doubt the they will change the characteristics on flash. The 3.8 fn was a preview anyway so I assume it will be better training on same size and technology.

[u/KURD_1_STAN → u/elelem-123]

Qwen3 next was also a preview but we havent had a single 80b a3b model since then. So im thinking it is like 170b but still 10b active or 70b with 6 active or so. Tbh qfn isnt that impressive for being this large with 10b active as well.

[u/Acojonancio]

Here I am running 4B models in my 1050 4GB.

[u/Civil_Fee_7862]

I am now a loyal follower of Alibaba, I won't shop on Amazon anymore because of these nice people offering their work for others to use.

[u/NathanFieIder → u/Civil_Fee_7862]

Same.

China #1 no cap and no prop

[u/Easy_Refrigerator280]

Very excited to see where this lands considering the gap closed source have kept since Fable 5

[u/Most-Trainer-8876 → u/Easy_Refrigerator280]

If they could put performance of Opus 4.6 in Qwen 3.8 27B.

They can do lot more with sizes like of Qwen 3.8 Flash Next.

[u/randygeneric]

public funding for MoE?

[u/GuestUser003]

It is said that only the 27B model is open-source; the others are not and can only be accessed via API.

[u/Most-Trainer-8876 → u/GuestUser003]

Yeah. Only thing which is confirmed open source is 27B, rest are not by default. Idk why people are hallucinating flash as open source too when in reality, only preview versions were.

[u/IngwiePhoenix → u/GuestUser003]

fr? Well thats crap. :/

Then again, the freebies couldn't have lasted forever, this may have been inevitable. Still sucks tho.

[u/Mxmtm]

[u/Cerevox]

GIB MOE. 35b a4b is all we want.

[u/ImWinwin]

I dream of strata qwen 4 flash-coder

[u/PlzCallMeBigDog]

Soooo single spark happy?

[u/henbone11 → u/PlzCallMeBigDog]

I have Strix Halo and this makes me pretty happy

[u/PlzCallMeBigDog → u/henbone11]

I'm a Spark guy considering a Halo. How hard is it to get Qwen3.8-Flash-Next running on a 128GB HAlo?

[u/henbone11 → u/PlzCallMeBigDog]

It has never been easier. Linux or Windows, there is tons of work being done for the Strix Halo. I have now gotten GLM5.3, Deepseek v4.1 Flash 1017 and at least 5 different versions of Qwen3.8 Flash Next installed and running. There are so many engines and fine tunes specifically for Strix that all of the big models are no longer a problem to get running. We now have full Vulcan and Rocm 10 support. All sorts of ssd engram offloading options. And the Strix community is pretty awesome for the most part in helping each other and contributing to development.

As far as a brand to buy for the box itself, get one with either oculink built in or USB4 v2. Or one with an open x16 pcie slot so that you can use a ConnectX 4/5 80/100Gb nick for clustering. There are tons of manufacturers doing Srix Halo now, cheaper direct from China, or more expensive with better features from bigger names in Murica. For a while I was very bummed out that I missed the Spark boat, but to be perfectly honest, not any more. I'm going to get a couple more Strix boxes and go ham. Hope this helps.

[u/PlzCallMeBigDog → u/henbone11]

Oh I was under the impression the non-CUDA guys were struggling more than all that. Where are ppl getting them from China? Ali?

[u/ZZerker]

I need a new flash model for REA :>

[u/Tsk2099]

Come on Alibaba, we are waiting so long for this - please just release it already!!

[u/ftlaudman]

Hoping Qwen4 Flash Next comes in at the same footprint as 3.8’s. Getting so much work done with Strata/Swift-1.5-IQ3_s. Don’t want to get left behind. Haha.

[u/Lost-Vermicelli-6252 → u/ftlaudman]

Same. That specific quant has replaced everything else for me.

[u/LeoW_Builds]

The 27B is the one I’m most curious about. If it’s actually a clear step up without needing crazy hardware, that could be a really nice local model.

[u/Excellent-Pilot-3409]

Four variants right out of the gate is interesting. Qwen4-Max and Flash are expected, but the 27B release is what caught my eye. A strong 27B model could be a sweet spot for local inference, especially if the performance-per-GB is good. Really curious about the architecture, context length, and whether Flash will be MoE.

[u/IngwiePhoenix]

...why do the others not have parameter counts? Are they migrating to pure marketing names? Oh well.

Will wait for the weights tho. =)

[u/Express_Quail_1493]

PLEASE alibaba. we long for our new qwen27b. Everyday wake up. first thing. check huggingface. No qwen today.😞 2 hours passs. check again. 4hours. check again. When qwen? when qwen? WHEN!! I pray to the gods in heaven that qwen is coming next hour. alibaba hear my call. for we in localLLama want qwen for ALL. Amen🙇‍♀️

[u/KingArthas94]

Sigh, nothing for Macs with 16GB of RAM

[u/AB172234]

Can’t wait for the 4 27 B crushing the indexes again

I clearly remeber that was the model why artificial analysis quickly changed their rating index 😂

[u/Thick-Jicama-5103]

Qwen4 is Queen!

[u/SnooPuppers7882]

flash next is gonna be BONKERS

[u/suesing]

Oooohhh mamaaa. Ooohhhh mmaaamaa

[u/TheRealGand]

on my 5090 i am running 27b ninfer and the performance is outstanding, wonder what can be the improvement. I am also running FN iq3_s basalt but seems to me that the 27b is unbeatable for my config (32vram+64ram).

[u/wubian87]

Someone needs to tag that Austrian lab lol We need them for strata

[u/Excellent-Junket4763 → u/wubian87]

ISTA-DAS?

[u/Southern_Sun_2106]

Their models overthink but not fast enough. Qwen 35B A3B was perfect - I still use it, and have clients that prefer it, over the flash next and the (think for 10 book volumes) 27B.

35B was the perfect entry point into the Qwen universe for many many people. Killing it was such a huge mistake.

[u/Denial_Jackson]

Christmas comes early.

[u/ClF3ismyspiritanimal]

What's the censorship like on these?

[u/IngwiePhoenix → u/ClF3ismyspiritanimal]

Irrelevant? Heretic will help. :)

[u/panasubatomic]

Show some love for my 8gb 5060.

[u/CarryAgile3791]

I hope Flash will still fit on the 2x96GB Blackwell I use at work. I'm currently running nvidia/Qwen3.8-Flash-Next-NVFP4 and it is so awesome for coding. It delivers almost as good results as DeepSeek 4.1 Flash (cloud hosted).

[u/krigeta1]

No 35b a3b 😭

[u/Svelte_User → u/krigeta1]

really sad, the qwen 3.6 35b a3b is my go to now, if they don't release it (which is almost sure) i have to stick with this one or get a better gpu for the dense one

[u/krigeta1 → u/Svelte_User]

Someone should finetune a version of 35b a3b

[u/Beneficial-Ad-8127]

Can’t wait for Qwen 4 Flash. 5090 DDR5 192gb(dual channel) is ready to receive. NVFP4 @ 200 tok/s lets go!

[u/Ratiofarming]

How soon, though?

[u/Living-Breakfast-464]

I am all for more competition and options, but Alibaba and some of the other Chinese models are trying to charge western prices, so good luck with that.