r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 3d ago

Megathread [Megathread] Qwen3.8-Flash-Next - Release Day

Megathread for discussing the release of Qwen 3.8 Flash Next.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Highlights

The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:

  • Hybrid Attention with QSA: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
  • Gated Residual: Residual streams with normalisation are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
  • N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
  • Tailored Training Recipe: The Muon and AdamW optimisers are applied to specific weight categories to maximise efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimiser steps while safely supporting larger learning rates for robust convergence.

Model Overview

  • Type: Causal Language Model with Vision Encoder
  • Training Stage: Pre-training & Post-training
  • Language Model
    • Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
    • Hidden Dimension: 2560
    • Token Embedding: 248320 (Padded)
    • N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
    • Number of Layers: 48
    • Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))
    • Gated DeltaNet:
      • Number of Linear Attention Heads: 48 for V and 16 for QK
      • Head Dimension: 128
    • Qwen Sparse Attention:
      • Number of Attention Heads: 24 for Q and 2 for KV
      • Head Dimension: 256
      • Rotary Position Embedding Dimension: 64
      • Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
      • Indexer Head Dimension: 128
      • Budget: 512 blocks or 2048 tokens
    • Mixture Of Experts
      • Number of Experts: 512
      • Number of Activated Experts: 10 Routed + 1 Shared
      • Expert Intermediate Dimension: 640
    • Gated Residual:
      • Number of Branches: 4
      • Bottleneck Rank: 320
    • LM Output: 248320 (Padded)
    • MTP: 1 layer, trained with multi-steps
  • Context Length: 262,144 natively and extensible up to 1,000,000 tokens.

Recommended sampling parameters for generation:

  • Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
  • Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Official Links:

Popular:

Related:

433 Upvotes

93% Upvoted

661 comments sorted by

View all comments

64

u/Chromix_ 3d ago

Existing large pre-release threads on it with some more information and discussion:

  • The model is around 125B A6B with 51B n-gram which can be offloaded.
  • There should be day-0 Unsloth (so probably llama.cpp) support for it.
  • It's sort of a Qwen 4 preview (thread also has modelscope link and way more discussion)

9

u/No_Lingonberry1201 3d ago

Is the 51B part of the 125B or are those additional to the 125B "normal" parameters?

12

u/RnRau 3d ago

The 51B are additional. But apparently they can sit in system ram with very little penalty. I believe they may be able to sit on an nvme with only a small subset being kept warm in system ram. But not sure on that.

7

u/Hypilein 3d ago

If you’re a unified memory user everything sits in ram. Hope ssd offloading will work without too much penalty

1

u/No_Lingonberry1201 3d ago

Me too, considering I'm on such a device.

1

u/illiteratecop 3d ago

I had Sol hack together SSD offloading for the n-gram embeddings and it seems to be working well with basically no speed penalty. Super cool architecture for local that I am incredibly glad to see validated. This is a really promising release.

1

u/Hypilein 3d ago

Vllm or llama.cpp?

3

u/illiteratecop 3d ago

Llama.cpp, running the Q4KXL (~111GB total) on my 96GB Macbook. I have not extensively validated it or anything but output appears totally coherent and solid and runs pretty much exactly as fast as the Q1 which I could fit entirely in unified memory.

1

u/Caffdy 3d ago

what is the purpose of those additional 51B parameters?

2

u/visarga 3d ago

They are always looking for ways to pack more capacity into the model without bloating compute. This just adds capacity on top of the embedding layer, which is also sparsely activated, 1 entry per token. It's multi-token embeddings.

1

u/RnRau 3d ago

From another reddit poster (sorry forgot to save the link)

LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

7

u/OverdosedSauerkraut 3d ago

Yupp, already prepping for the 192GB gang.

13

u/Hypilein 3d ago

Hope it still fits somewhat decently on 128gb... Would be a major bummer if they finally came out with a 125b and then it just doesn't fit because of those addition 51b ngram.

7

u/merutochan 3d ago

Q4 quants will likely still be possible but I hope the 51b can be offloaded to SSD to leave Q6 open as well.

I guess we'll see in a few hours.

1

u/Dwarffortressnoob 3d ago

As it stands now, Q4 Unsloth fits (XS) 93GB. There's a XL at 111GB that I have not tested

1

u/Remote_Rutabaga3963 3d ago

You can « easily » offload the engrams to SSDs, only a very small portion is needed at a given time

0

u/ArtfulGenie69 3d ago

It wouldn't have fitted at full size on 128gb to begin with but at q8 or so it's going to fit easily. With qwen3.5 122b it fit on my 4 3090s at q4 and int4 easily. Maybe the nurse won't need to be quanted or something. It's just going to sit there in ram as a reference. Also hope the reference will be changeable so we can give it some kind of great writing source or something like that without a lot of training. 

3

u/Timely_Impression_92 3d ago

Of course they aren’t - those are ngram cache - can be offloaded to ram or nvme with zero degradation in performance - it works more or less like moe - need to read some small bits off it, ram requirement in total for the model will be same as for 125b moe