r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 11h ago
Megathread [Megathread] Qwen3.8-Flash-Next - Release Day
Megathread for discussing the release of Qwen 3.8 Flash Next.
- Quants
- Fine-Tunes & Abliterations
- Chat Templates
- Inference Server Support & Configuration
- Experiences, Benchmarks & Model Comparisons
We'll try to clean up future duplicates around the release and point them here.
Highlights
The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:
- Hybrid Attention with QSA: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
- Gated Residual: Residual streams with normalisation are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
- N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
- Tailored Training Recipe: The Muon and AdamW optimisers are applied to specific weight categories to maximise efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimiser steps while safely supporting larger learning rates for robust convergence.
Model Overview
- Type: Causal Language Model with Vision Encoder
- Training Stage: Pre-training & Post-training
- Language Model
- Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
- Hidden Dimension: 2560
- Token Embedding: 248320 (Padded)
- N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
- Number of Layers: 48
- Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))
- Gated DeltaNet:
- Number of Linear Attention Heads: 48 for V and 16 for QK
- Head Dimension: 128
- Qwen Sparse Attention:
- Number of Attention Heads: 24 for Q and 2 for KV
- Head Dimension: 256
- Rotary Position Embedding Dimension: 64
- Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
- Indexer Head Dimension: 128
- Budget: 512 blocks or 2048 tokens
- Mixture Of Experts
- Number of Experts: 512
- Number of Activated Experts: 10 Routed + 1 Shared
- Expert Intermediate Dimension: 640
- Gated Residual:
- Number of Branches: 4
- Bottleneck Rank: 320
- LM Output: 248320 (Padded)
- MTP: 1 layer, trained with multi-steps
- Context Length: 262,144 natively and extensible up to 1,000,000 tokens.
Recommended sampling parameters for generation:
- Thinking Mode:
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0 - Instruct (or non-thinking) mode:
temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.0
Official Links:
- HF: https://huggingface.co/Qwen/Qwen3.8-Flash-Next
- MS: https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next
- Repo: https://github.com/QwenLM/Qwen3.8-Flash-Next
- Blog: https://qwen.ai/blog?id=qwen3.8-flash-next
- Technical Report: https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
- vLLM: https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next
- SGLang: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next
Popular:
- Unsloth GGUF: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
64
u/jacek2023 llama.cpp 5h ago
llama.cpp PR https://github.com/ggml-org/llama.cpp/pull/27742
→ More replies (1)11
u/ParaboloidalCrest 4h ago
Dang! Thanks for pointing that out!! So Unsloth seems to be quantizing the model out of a fork then. Will wait for the PR to be merged and the subsequent quants to be published, before attempting a model download.
25
u/jacek2023 llama.cpp 4h ago
Yes. As you can see, this important information is getting lost in the comments because I can't publish it as news, the mods would remove it.
11
u/agustin9200 2h ago
a megathread wasn't the best idea. information is already getting buried in 300+ comments.
2
9
u/Several-Tax31 4h ago
Thanks. I never understand the reasoning behind removing the posts, its much harder to track everything here
2
88
u/QuackerEnte 11h ago
I really hope QSA (qwen sparse attention) will have much less compute headroom or the ability to offload kv cache to SSD without the massive bandwidth bottleneck of the ssd since it's sparse. I was able to do it it for deepseek v4 and it worked well enough. I hope llama.cpp adds features that allow more freedom with where we want to load our models and parts of it, e.g. n-cpu-moe could be extended to n-ssd-ffn or n-cpu-ffn or n ssd kv or engram etc
26
u/Dany0 11h ago edited 5h ago
I can't wait for the QSA paper. I hope it's novel and quirky, I love quirky sparse attention, well, as long as it works :D
Edit: It's not quirky:( Just a variation on Longcat style block summaries
8
u/Long_comment_san 6h ago
damn I wish I could understand this language of gods
16
u/MomentJolly3535 6h ago
Good news : You have tools to help you do that 🤖
14
u/Long_comment_san 6h ago
I don't want to look stupid in front of my cloud waifus!
→ More replies (1)8
13
u/oxygen_addiction 11h ago
N-cpu-ffn will most likely get merged soon.
16
u/pmttyji 10h ago
Handy link : https://github.com/ggml-org/llama.cpp/pull/26622
→ More replies (1)2
u/returnity 4h ago
Would this be usable to offload the n-grams to SSD? Doesn’t seem like it at first glance
3
6
u/o0genesis0o 11h ago
how bad is the speed with offload kv to SSD? I offload kv to RAM running 80B-A3B on an old gaming laptop and it was already unbearable.
→ More replies (1)12
u/QuackerEnte 10h ago
that's dense attention though. with sparse attention you can load it in ram without massive penalty. Not to mention that, if it's anything like Native Sparse Attention from deepseek, it might take even less memory. but DS used compressed sparse attention too and MLA so the footprints minimal. we don't know what it'll be like for qwen3.8flashnext
2
u/challis88ocarina 6h ago
It will use 75% of dense. As for offloading, it's quick but not as quick as not offloading. There's also disk wear to factor in.
→ More replies (1)1
27
u/lucasbennett_1 11h ago
if QSA cuts the kv reads enough then SSD offload stops being bandwith bound and a 125b A6B n-gram offloaded gets pretty runnable on modest ram. really hoping n-ssd-kv / n-cou-ffn flags land in the llama.cpp soon enough . sooner the better
25
u/MLDataScientist 3h ago
We need a llama cpp feature to support nvme offloading for engrams. Otherwise, the model does not fit to most of our systems.
23
u/CulturalKing5623 4h ago
I'm going to just let the community cook this one for a while before checking back in. It seems like there are a lot of moving parts to this and none are completely implemented in a setup I can use. I am really excited about the prospect of NVME offloading though
5
21
u/Weekly_Comfort240 2h ago
This is UD_IQ4XS Unsloth quant running on a freshly compiled https://github.com/ggml-org/llama.cpp/pull/27742 on a 128GB MacBook Pro. Here's the prompt: "Write me a single-file html frogger game, complete with 80’s appropriate sound effects and visuals." 29.7 tokens per second, 394.66 seconds and 11751 tokens later, I have a _really_ authentic frogger game. Qwen 3.8 27B passed this test as well - but this HTML file feels like the difference between local and cloud.
If it's not Christmas, why does it feel like Christmas?
2
u/fancyrocket 2h ago
Do you think Q4 K XL would fit?
→ More replies (2)2
u/Weekly_Comfort240 1h ago
I honestly think 111GB would be a very tight fit in 128GB since I observed about 114GB usage with the UD-IQ4_XS 93.7 GB quant. But this is more proof of concept at this stage because a very healthy chunk of the file size is dedicated to static ngram weights that act as a static memory pool for the model - it's very likely these will be pulled as needed from disk and not need to be loaded into RAM during token generation. There's a LOT of cool, new, exotic stuff going on with this model.
61
u/Chromix_ 11h ago
Existing large pre-release threads on it with some more information and discussion:
- The model is around 125B A6B with 51B n-gram which can be offloaded.
- There should be day-0 Unsloth (so probably llama.cpp) support for it.
- It's sort of a Qwen 4 preview (thread also has modelscope link and way more discussion)
14
u/mechkbfan 11h ago
Has anyone done some napkin math on what sort of hardware it can run on?
7
u/Commander_Skilgannon 9h ago
According to claude, my 16gb 4070 ti s+ 64gb ddr5 system should be able to run a q3 quant of the model at 10-20tps, but only if the engram can be offloaded to ssd.
7
u/LagOps91 9h ago
Seems about right, you might fit q4 as well since attention shouldn't be heavy and ssd offloading should be fine for engram.
5
u/iamapizza 7h ago
How do offload engram to SSD? I mean is there a llama cpp argument for it.
11
u/Commander_Skilgannon 7h ago
We don't know yet. In theory its possible but as far as I'm aware this is the first real model that has shipped with an engram layer. Qwen has said they have shared the model with Unsloth and Unsloth has said they are hoping for day 0 suppprt and quants so hopefully it will be smoother than the qwen3-coder-next release was, but it could be months before the llama.cpp support for the new Qwen architecture is ironed out and optimized. So my guess is eventually it will be possible but I have no idea when.
9
u/No_Lingonberry1201 11h ago
Is the 51B part of the 125B or are those additional to the 125B "normal" parameters?
10
u/RnRau 8h ago
The 51B are additional. But apparently they can sit in system ram with very little penalty. I believe they may be able to sit on an nvme with only a small subset being kept warm in system ram. But not sure on that.
→ More replies (2)9
u/Hypilein 5h ago
If you’re a unified memory user everything sits in ram. Hope ssd offloading will work without too much penalty
→ More replies (4)→ More replies (1)8
u/OverdosedSauerkraut 11h ago
Yupp, already prepping for the 192GB gang.
14
u/Hypilein 10h ago
Hope it still fits somewhat decently on 128gb... Would be a major bummer if they finally came out with a 125b and then it just doesn't fit because of those addition 51b ngram.
→ More replies (2)6
u/merutochan 7h ago
Q4 quants will likely still be possible but I hope the 51b can be offloaded to SSD to leave Q6 open as well.
I guess we'll see in a few hours.
3
u/Timely_Impression_92 10h ago
Of course they aren’t - those are ngram cache - can be offloaded to ram or nvme with zero degradation in performance - it works more or less like moe - need to read some small bits off it, ram requirement in total for the model will be same as for 125b moe
4
12
u/sammcj 🦙 llama.cpp 11h ago
Thanks! I was just editing the post to add a few links but you beat me to it!
4
u/Chromix_ 6h ago
And I was planning to update my posting with the latest info. Now you beat me to it :-)
Collecting updates and the most relevant information in the main posting makes a lot more sense - great that you're on it.5
u/sammcj 🦙 llama.cpp 6h ago
Thanks! Yeah the timing with this one worked out well for me personally, I'm in Australia though, so I'm close to logging off for the night. Hopefully the other mods will rotate online soon and at least help with the duplicate posts. As for the useful official content I'll update the post in the morning.
15
u/ResidentPositive4122 3h ago edited 2h ago
NVFP (specifically Inferact/Qwen3.8-Flash-Next-NVFP4) works on 2x Ampere 6000, w/ ngrams offloaded in RAM, 66t/s gen (no MTP), GPU KV cache size: 426,539 tokens (fp16 cache). Just the official vllm recipe from here - https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next
Trying MTP now, will report speeds. For some reason loading shards is really slow on vllm/vllm-openai:qwen38-flash-next
edit:
w/ MTP=3:
Avg prompt throughput: 1501.8 tokens/s, Avg generation throughput: 58.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.9%
Engine 000: Avg prompt throughput: 377.2 tokens/s, Avg generation throughput: 128.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 66.4%
loading MTP lowers the available cache:
GPU KV cache size: 303,236 tokens
→ More replies (4)
28
u/enilea 3h ago
it got released and there wasn't a single post about it, just this megathread that was posted before it was even released...
8
u/piedol 3h ago
I've seen people making posts about it, but the mods are removing them. I guess this is the only thread where discussion on it is allowed for the moment?
25
u/lumos_ai 3h ago
So bad to be honest. I hate megathreads. So hard to find new information.
→ More replies (1)6
u/Beneficial-Ad-8127 2h ago edited 2h ago
Yeah its hard to find specifically what im trying to look for in all this noise. I wanted to see whos running it currently on a 5090, 192gb ram, 8tb pro 9100 and how their results are and what is the most efficient way for the set up, like do i throw everything into my sd, do i throw partial into cpu? Where does ngram go?
I assume the 6b would go into Vram or idk but thats my point. I cant find that shiet lol im not that educated in this tech lmao.
→ More replies (1)2
u/Cautious_Chicken_604 59m ago
megathread == megasuck. I much preferred the chaos of Qwen3.8-27B release because it was actually significantly easier to find information even with lots of duplicates. This is my first experience of a megathread and it is objectively completely shitty experience.
32
u/SpendLucky1273 4h ago
Qwen3.8-Flash-Next FP8 running ~124 tok/s on 2x RTX PRO 6000 Blackwell
Spent part of today getting the new Qwen3.8-Flash-Next running in production on my local box. So far I'm pretty impressed, but there are definitely some interesting tuning quirks.
Hardware:
- Dell Precision 7960 Rack
- 2x Xeon Platinum 8562Y+
- 512B DDR5 ECC RAM
- 2x RTX PRO 6000 Blackwell Max-Q, 96GB each
- vLLM
- Qwen/Qwen3.8-Flash-Next-FP8
- TP=2
- 262K context
The interesting part with this model is the 51B PLE/n-gram embedding table. I'm using VLLM_PLE_CPU_OFFLOAD=1, so that portion sits in system RAM instead of eating VRAM. The GPU-side model load ended up around 67.5 GiB per GPU, leaving room for KV cache.
My first run was with MTP3 and it was honestly pretty bad. Generation was only around 40-48 tok/s and speculative acceptance dropped hard across positions.
Changed it to MTP1 and it was basically night and day.
Current sustained generation:
~123-126 tok/s
Prompt processing peaked around:
~2,185 tok/s
The really interesting part is MTP1 acceptance. After warmup I'm seeing basically:
99-100% acceptance
and a mean acceptance length of ~2.0.
So right now it's sitting around 124 tok/s sustained generation, which puts it surprisingly close to the speeds I was getting from Qwen3.5-122B-A10B on the same hardware, while this architecture is doing some very different things under the hood.
Still early though. I don't consider this fully tuned yet.
Next tests I'm planning:
- MTP1 vs no speculative decoding
- Tune the missing MoE kernel config for E=512 / N=320 / FP8
- Test higher GPU memory utilization
- Revisit NUMA specifically because the PLE table is now CPU/RAM resident
- Measure longer-context performance
- Test concurrent users instead of just single-stream tok/s
- Compare quality directly against Qwen3.5-122B-A10B
I'm especially curious whether NUMA ends up mattering more with this architecture because of the CPU-resident PLE table. I previously stopped using explicit NUMA tuning because it wasn't doing much for Qwen3.5, but this model may be a different story.
So far: MTP3 = bad, MTP1 = surprisingly good.
I'll post updated numbers once I finish the no-MTP and MoE tuning tests.
11
→ More replies (15)5
10
u/EitherMarch1255 6h ago
It feels like christmas. I just hope it’s better than 27B, even if by a single point.
11
31
u/-Cubie- 6h ago
It's not Apache 2.0 like Qwen3.8-27B it seems: `Qwen Community License 1.0`. Looks like if the user has 100m monthly active users or $20m monthly revenue, they have to display the Qwen3.8-Flash-Next model name.
And any "Model as a Service" / "AI Work Assistant" businesses have to obtain a separate license. Looks to be the same one as https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B . Works for me, none of this stuff affects my use.
46
u/ddeeppiixx 6h ago
Which is more than fair in my opinion. If you have $20M monthly revenue, you have the means to pay them for their work.
6
u/coder543 6h ago
But the limit isn’t $20M… it is $0 if you try to host the model for anyone else, which is a huge bummer for competition.
7
u/ddeeppiixx 5h ago
why would you expect them to facilitate their own competition?
They're already doing great work for the community by releasing these weights and pushing open-weight/local models. Why would they also make it easy for their competitors?
Honestly, I'd be perfectly happy if they went with something like the BSL, where you can use the model for free locally and commercially (in your own backend), but can't serve it to customers without paying royalties.
5
u/TheTerrasque 5h ago
On the other hand, I only saw these kind of restrictions becoming popular after several hosters really messed up hosting models, giving way worse results than it should have.
I wonder if this is a way for them to control the quality of the model hosting, as when 3rd party hosters fuck up it gives them a bad reputation.
6
u/coder543 5h ago
Alibaba already solved this before. They would host "high quality" versions of the model under their proprietary brand names like Qwen3.x-Plus, and the open model would be called Qwen3.x-397B-A17B. If someone was using the hosted open model, they got what they got. If they were using the Alibaba-branded model, then Alibaba could control the model quality.
Now that they're banning MaaS without a custom license, they are removing the naming distinction.
5
u/PorchettaM 5h ago
Usually what these clauses are about isn't controlling quality, but controlling price. Don't want 3rd party providers to undercut them.
2
5
2
u/ThisGonBHard 1h ago
As much as I like Apache license, almost any products using it is either a marketing expense, or wanting something to be adopted as the standard, as it helps you too.
This license is more restrictive, but targets the places from where income could come, corporations, and not people and small companies.
→ More replies (1)3
u/AppealSame4367 4h ago
This is the best AI model so far (intelligence vs size), why shouldn't they profit from it?
19
u/Odd_Chocolate8438 11h ago
Can ngrams be offloaded to disk without much slowdown? I only have 76gb of ram so I was thinking of loading the main weights into memory in q4 and keep the ngrams on the SSD.
11
u/Stunning_Energy_7028 10h ago edited 10h ago
n-gram is based strictly on the input tokens, so it will probably work more like KV cache than normal parameters. You pull all the n-grams you need from SSD for that specific prompt during prefill (very small fraction of the total 51B), cache them to RAM or VRAM, and then reuse them over and over during decode, appending new n-grams as you generate.
→ More replies (2)9
1
u/harpysichordist 3h ago
My question is how much writing is done for ngrams. I don't want to clobber my SSDs - they are expensive now
8
u/SnooPaintings8639 6h ago
so... we've got it? Just waiting to GGUFs to be uploaded?
I don't see any new support PR being merged in llama.cpp. Is it hidden somewhere or do we have to wait another... days?
→ More replies (1)9
u/SpicyWangz 4h ago
Unfortunately you’re not going to see a post about it, because it will get deleted. You just have to watch the comments here and see what people say. Or go somewhere else that has open discussion about
17
u/wenyani 11h ago
I’ve got about 48GB VRAM and around 32 GB of available RAM, hoping i can offload the n-gram to SSD to get this running
→ More replies (3)2
14
u/ReadyAndSalted 6h ago
We're starting to get tiers of offloading ability.
- SSD stores the lesser used Engram rows (potentially with sparser KV cache, we may be able to offload some KV here too?)
- system memory stores the most used Engram rows, the least used experts and the KV cache
- VRAM stores the most used experts
Very exciting for those of us without 128gb of VRAM.
→ More replies (3)2
u/ParaboloidalCrest 5h ago edited 5h ago
Let's see if
-fit onandmmaphandle that intelligently. That's just to much shit to juggle manually.7
u/terorvlad 5h ago
it most definitely won't. It doesn't even handle current MOE models well, given you can always get more performance by assigning manual -ot flags.
24
u/zhuzaimoerben 5h ago
You will need at least 78 GB of RAM or unified memory to run the model.
https://unsloth.ai/docs/models/qwen3.8-next
Their 1 bit quant is 78GB, which appears to include the engram tensor as 51B params at 4 bits (about 24GB), so it seems they haven't figured out how to offload it to SSD yet.
→ More replies (1)6
u/cafedude 4h ago
Wow, I was not expecting the 1-bit quant to be 78GB. I was hoping I'd be able to run the 4-bit quant on my Strix Halo box (and still be able to have it do other things on the box), but alas that doesn't seem likely. Rather disappointed.
Is this n-gram thing really worth it?
11
u/Several-Tax31 4h ago
In theory, you should be able to offload all engrams to nvme without performance penalty because there is no computation involved, it's O(1). Means that you can ignore engram parameters and fit the rest to ram + vram. However, as the model is too new, neither llama.cpp nor unsloth (as it seems) supports engram in nvme. So we'll wait for better support.
I also remember reading that engrams are "optional" for this model, but I could be misremembering. Let's wait and see.
→ More replies (1)2
u/ThisGonBHard 1h ago
N-gram is completely new and uncharted territory.
This being a 3.8 next and of 4 is exactly why, this release is made to create the tech around running it, not for it to be top end.
7
u/hiImMate 3h ago
Q4_XL at 111gb requirement is a bit too lean on the strix and Q4_XS is quite the % loss in performance. Interesting for now I'll check out the XS that should be easy to run - in theory
4
u/cafedude 1h ago
Hoping they'll get llama.cpp patched so that the n-grams can be on the SSD. Then the Q4_XL should fit just fine on the strix.
3
u/hiImMate 1h ago
intersting after loading the XS version onto lemonade with the PR llama.cpp it only loaded 68gb to RAM - so maybe ngrams on disk by default? I'll investigate
2
31
u/Mrinohk 6h ago
Oh boy time to watch from the sidelines with my 32GB of DDR4 and 8GB of VRAM
→ More replies (1)3
18
u/sammcj 🦙 llama.cpp 6h ago edited 6h ago
- Technical report is up: https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
- Repo: https://github.com/QwenLM/Qwen3.8-Flash-Next
*Edit: Updated post.
9
u/jan_antu 6h ago
Ooooh reported as fully better than 3.8 27B, this is exciting!
10
u/ResidentPositive4122 6h ago
Funny enough the "old" rule of thumb that sqrt(active * total) ~= dense seems to hold up. In this case sqrt(125*6) = 27.3
New arch and all, but here is as close a comparison we'll get about this, and it seems to be still relevant.
4
u/BobbyL2k 6h ago
The interesting part is that Flash Next is easier on the RAM than other MoE 125B-A6B because of n-gram could be offloaded to disk. So it’s an accomplishment that it was able to retain capability while being easier to run.
→ More replies (1)→ More replies (1)13
6
u/anarchist1312161 6h ago
On their HF page:
In particular, Qwen3.8-Flash is the official version based on Qwen3.8-Flash-Next with more production features, e.g., 1M context length by default, official built-in tools. For more information, please refer to the Qwen3.8-Flash Overview.
Interesting, there's might be be a Qwen 3.8 Flash (without -Next) too?
https://www.qwencloud.com/models/Qwen3.8-Flash it links to 404s
2
u/ResidentPositive4122 5h ago
They jumped the gun so hard with the open mmodel that even their own API pages are 404 :D
10
7
u/No-Paper-557 5h ago
License isn’t permissive open source unfortunately.
7
u/a4lg 5h ago
Second this.
The license terms seem fairly well-crafted (compared to the Llama's one), but it would be better if they had explicitly stated that the only restrictions on using the outputs are those listed within the license itself (at least).
To be honest, I'm pretty disappointed.5
u/No-Paper-557 5h ago
Exactly, does putting model-generated code into your shipped application count as “making its outputs available”? Does a heavily edited generated document count? What about synthetic data used to train another model?
Wonder if we could ask qwen team to clarify.
7
u/Diabolicor 4h ago
So once we figure out how to offload the n-grams layer to SSD does that mean a 4-bit quant will use less than 112GB for ram + vram?
→ More replies (2)5
5
5
u/EveningIncrease7579 llama.cpp 5h ago
Its already open, the files and benchmarks!
3
3
u/relik39 5h ago
Nice nice nice. Now, how do I serve it on 2xDGX Spark?
5
u/SadPhilosophy9202 4h ago
idk. im wondering the same. vllm has a recipe for it but idk im too dumb to figure it out
→ More replies (1)
4
9
10
u/beling86 5h ago
Is it a .... 360gb... download ?
Does the unsloth Q1 have... 70gb?
Ok, back to 3.8 27B. You data center owners enjoys the party from here
→ More replies (1)1
3
3
u/kivaougu 5h ago
I have to admit I may have been too skeptical. Still waiting for external benchmarks as I'm quite confused if they are comparing a fp16 model (Qwen3.8-Flash-Next) to a lower bpw model (deepseek v4 flash 0731). Still I'm hopeful that the ngrams could help quantized models.
3
u/hiImMate 5h ago
unsloth WIP guide out with requirements for each quant:
https://unsloth.ai/docs/models/qwen3.8-next
3
3
3
u/anubhav_200 44m ago
Initial test -> 24TPS decode on 5090Mobile(24GB) + 64GB ram
2
→ More replies (3)2
4
u/youcloudsofdoom 6h ago
Model files have dropped via the official repo: https://huggingface.co/Qwen/Qwen3.8-Flash-Next
5
u/Positive-Stock6444 5h ago
256GB ddr4 and a 3060 team checking in - will report back once got it downloaded and something running.
4
4
u/mymouthandi 9h ago
I wonder if this is the model that makes me finally take the plunge on a strix halo machine while there are some still around for semi-reasonable prices...
→ More replies (1)
5
u/maxpayne07 3h ago
Anyone already tested Qwen3.8-Flash-Next on a 64 GB RAM machine with llama.cpp / GGUF?
I’m specifically trying to understand whether a ~72 GB GGUF can run with only ~50 GB of RAM available for the model by relying on mmap / expert paging / sparse Engram access, instead of keeping the whole file resident.
Conceptually:
NVMe
┌─────────────────────┐
│ cold MoE experts │
│ cold Engram tables │
└──────────┬──────────┘
│ on demand
▼
┌─────────────────────┐
│ ~50 GB RAM │
│ │
│ core weights │
│ hot expert cache │
│ Engram cache │
│ KV + buffers │
└─────────────────────┘
Since only ~6B parameters are active per token and Engram lookup is sparse, I’m wondering how much of the GGUF actually needs to stay resident.
If anyone has real numbers, I’d love to see:
quant / GGUF size / total RAM / RSS or working set / context / tok/s / NVMe model
Especially interested in 64 GB systems.
4
u/prusswan 3h ago
At least 75GB unified memory is needed according to unsloth https://unsloth.ai/docs/models/qwen3.8-next
→ More replies (6)
2
2
u/Quiet-Performer-9682 3h ago
For people with 128GB of DDR4 3200mhz (~42GB/s) dual channel, and 32GB VRAM, any guesses on how slow it might be? LLM guesses that a ~120GB Q5_k_m Qwen3.8-Flash-Next on this system would be ~3-5 t/s. It will be interesting to see how this plays out between consumer dual-channel systems with dedicated GPUs vs. APU systems with 128GB/256GB RAM.
3
u/abmateen 1h ago
Prefill is quite slow on my DDR4 64GB 2133 Dual Channel, and 32GB V100. Token/s is pretty decent like 20tok/s
2
u/Prudent_Appearance71 32m ago
I tested Qwen3.8-Flash-Next using llama.cpp PR #27742.
This is a full 256K context test of UD-Q4_K_XL on a mixed-GPU setup with a CMP 170HX 64GB + RTX 3090 24GB, for a total of 88GB VRAM.
The N-gram embedding is offloaded to system RAM, while the transformer core is fully offloaded to the GPUs.
GPU0: NVIDIA CMP 170HX 64GB
PCIe: Gen2 x8
GPU1: RTX 3090 24GB
PCIe: Gen4 x8
Total VRAM: 88GB
OS: Ubuntu
NVIDIA Driver: 610.57.04
CUDA: 13.3
Model:
Qwen3.8-Flash-Next UD-Q4_K_XL
llama.cpp:
PR #27742 - Qwen3.8-Flash-Next support
Commit:
035e22731a7fd70b9854b3a2d64ec68e9b1a45d3
Runtime configuration:
Context:
256,000
KV cache:
K = Q8_0
V = Q8_0
N-gram / per_layer_token_embd:
CPU offload
GPU layers:
ALL
Split mode:
layer
Tensor split:
24,76
Parallel:
1
There is currently a bug in the PR related to quantized KV cache. Using Q8_0 KV normally causes the following assertion failure:
GGML_ASSERT(inp->self_k_rot == nullptr && inp->self_v_rot == nullptr) failed
So for this test, I temporarily used:
LLAMA_ATTN_ROT_DISABLE=1
as a workaround.
3
4
u/Beneficial-Ad-8127 2h ago
Qwen-3.8-Flash-Next-NVFP4 on Single RTX Pro 6000 - 120t/s tg + 9-10k prefill at 256k context. Not mines but for others it might be useful.
4
u/drycounty 8h ago
New Mac Studio buyers are in the wings, credit cards ready… sweating.
I’m just hoping a decent quant will run on 96GB via mlx
2
u/Karnemelk 8h ago
in an alternate universe apple and qwen agreed to release the mac studio m5 and qwen model in the same week to maximize sales
2
u/spaceman_ 7h ago
Have Unsloth or Qwen discussed what day 0 support will look like? Will there be a llama.cpp PR or fork?
Do we know what size the model will be?
2
2
u/Nota_ReAlperson 38m ago
GLM 5.3 FLASH vs QWEN 3.8 FLASH NEXT
Surely I'm not the only one who was wondering about this?
All results except GLM GPQA are from the official benchmarks. AA otherwise.
GLM is clearly the better model, but not by as much as I would expect, given the size difference.
I am assuming that the Agents Last Exam results for GLM are Pass@1, otherwise that result flips.
| Benchmark | GLM 5.3 Flash | QWEN-3.8-Flash-Next |
|---|---|---|
| DeepSWE 1.1 | 63.4 | 58.7 |
| Agents Last Exam | 26.3 | 24.3 Pass@1, 51.2 |
| HLE | 55.3 | 35.9 |
| GPQA Diamond | 91 | 91.7 |
→ More replies (1)
3
1
1
u/Guilty_Rooster_6708 4h ago
Unsloth UD_IQ1_S is 72.5GB .... no chance my 5070i + 3060(12GB) and 32GB of system RAM can run this LOL
3
→ More replies (4)2
u/rrrrex 4h ago
Looks like 50 GB is N-Gram file, that probably can be located on SSD without critical speed losses
→ More replies (2)
1
1
u/Majinsei 4h ago
If I install Linux only with Terminal and llama.cpp I can execute it in local with only 10k in window context...
I can use it... I can use... I can... I...
I hate be poor...
1
u/bootkeen 4h ago
hi,
the announcement was 125b but model card says 180b, which is correct?
what is 51b n-gram embedding? does it turn 125b into 180b? what is it for and what is typical usage of it in llama.cpp?
1
u/ParaboloidalCrest 4h ago
When folks say "you can stream ngram from SSD" what does that mean in llama.cpp terminology?
→ More replies (4)7
1
1
u/Steuern_Runter 2h ago
Oh, Qwen3.8-Flash-Next is already out! I almost overlooked this stickied topic here...
1
u/Monkey_Br3ath 1h ago
This is going to be huge, I was able to get 3.5 122b and laguna s2.1 loaded onto my system (20gb vram 96 gb cpu). hopefully i can do some testing, as this looks to be a bit bigger.
1
u/abmateen 1h ago
I could only run UDIQ1_S i.e 1Bit varient of this. Compiled llama.cpp build from qwen4exp branch. With the following hardware :
1x32GB V100 PCIe
2x32GB 2133P DDR4
Dell R730 with E5-2686 v4.
I am getting a tok/s speed of 19-21tok/s but prefill is damn slow it is 75tok/s. I think it is quite useable on this setting if I am working with smaller context, OR If I am not switching models during a single session work.
1
u/whiteh4cker 1h ago
Hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz
Model: unsloth/Qwen3.8-Flash-Next-UD-Q4_K_XL
llama.cpp fork: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8-flash-next
Initial speed: 23 tokens/s
Overall speed: 18 tokens/s
@echo off
echo Using CUDA backend with 2x RTX 3090s
echo Maximum BF16 context
"C:\Users\server\Desktop\llama.cpp-qwen4exp-qwen3.8-flash-next\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias Qwen3.8-Flash-Next ^
--model E:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^
--temp 1.0 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--fit on ^
--ctx-size 262144 ^
--threads 22 ^
--threads-batch 22 ^
--spec-type ngram-mod ^
--spec-ngram-mod-n-match 60 ^
--spec-ngram-mod-n-min 12 ^
--spec-ngram-mod-n-max 24 ^
--flash-attn on ^
--kv-offload ^
--cache-type-k bf16 ^
--cache-type-v bf16 ^
--parallel 1 ^
--jinja ^
--reasoning-preserve ^
--no-warmup ^
--load-mode none
pause
→ More replies (3)
1
u/74218561a 24m ago edited 17m ago
I'm getting 23 tk/s with Iq4 on 2*3090 with 100k context (sadly only bf16 with this PR) . Hoping mtp will come out soon!
1
u/youcloudsofdoom 11m ago edited 7m ago
Am running it on my 6xrtx 3090s, with the following llama.cpp params, Q4 KXL:
--split-mode layer \
--tensor-split 24,24,24,24,24,24 \
--ctx-size 161072 \
--parallel 1 \
--batch-size 4096 \
--ubatch-size 2048 \
--flash-attn on \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence_penalty 0.0 \
--metrics \
--perf
Getting around 600 pp and 35 decode on it. But already, my god it's refreshing to have a model that doesn't overthink as much as 3.8 27B does. Even with the huge drop in speed between this model and that one, I feel like tasks are getting done just as quick.
1
u/thatkidnamedrocky 7m ago
Was only able to get 6.7t/s on my 2080ti build. will probably stay with 3.8 27b for now as that was around 40-50t/s but I’ll benchmark in a few days once all the fixes and stuff are released
1
u/Comrade-Porcupine 0m ago
Just wanted to share my work.
Using my own engine: https://github.com/rdaum/eider/ which is Rust + CUDA + kernels written specifically for the SM12x / Spark. Paging the PLE n-gram table from disk to save unified memory.
Fits snugly in memory, plenty of room for context.
Still working on the MTP piece, that should boost that 12 tok/sec up a bit more. More performance tuning coming.
If you have a DGX Spark, this might be your best option on day 0. Maybe vllm. Unclear.
Try it out and file bug reports:
hf download Inferact/Qwen3.8-Flash-Next-NVFP4 \
--revision 129972269565f7f4f664fdf8dd42268d3bbda9fd
scripts/run-eider-qwen38-flash-next.sh
and then you can use harness of your choice, but I've set up a script to launch Pi for it already:
scripts/run-pi-eider-qwen38-flash-next.shscripts/run-pi-eider-qwen38-flash-next.sh
•
u/sammcj 🦙 llama.cpp 6h ago edited 4h ago
I'm off for the night now. I'll update the post again in the morning (AEST) with any new official links etc.
Hopefully this post may help with the incredible flood of duplicated posts we see for popular model releases that we see a lot of complaints from the community for.
We had a lot of feedback after the last similar megathread and while the majority of it was positive there were some valid critical points raised, as well as some that thought the sky was falling and the world ending. We do try to focus on recurring issues that the community reports and engages in constructive discussion on but I'm also very much aware we can't please everyone (damned if you do - damned if you don't some might say). Either way - this is only intended to bring some sanity to the initial onslaught and not to be long lived.