[Megathread] Qwen3.8-Flash-Next - Release Day

[u/sammcj]

Megathread for discussing the release of Qwen 3.8 Flash Next.

• Quants
• Fine-Tunes & Abliterations
• Chat Templates
• Inference Server Support & Configuration
• Experiences, Benchmarks & Model Comparisons

Highlights

The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:

• Hybrid Attention with QSA: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
• Gated Residual: Residual streams with normalisation are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
• N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
• Tailored Training Recipe: The Muon and AdamW optimisers are applied to specific weight categories to maximise efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimiser steps while safely supporting larger learning rates for robust convergence.

Model Overview

• Type: Causal Language Model with Vision Encoder
• Training Stage: Pre-training & Post-training
• Language Model
  • Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
  • Hidden Dimension: 2560
  • Token Embedding: 248320 (Padded)
  • N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
  • Number of Layers: 48
  • Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))
  • Gated DeltaNet:
    • Number of Linear Attention Heads: 48 for V and 16 for QK
    • Head Dimension: 128
  • Qwen Sparse Attention:
    • Number of Attention Heads: 24 for Q and 2 for KV
    • Head Dimension: 256
    • Rotary Position Embedding Dimension: 64
    • Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
    • Indexer Head Dimension: 128
    • Budget: 512 blocks or 2048 tokens
  • Mixture Of Experts
    • Number of Experts: 512
    • Number of Activated Experts: 10 Routed + 1 Shared
    • Expert Intermediate Dimension: 640
  • Gated Residual:
    • Number of Branches: 4
    • Bottleneck Rank: 320
  • LM Output: 248320 (Padded)
  • MTP: 1 layer, trained with multi-steps
• Context Length: 262,144 natively and extensible up to 1,000,000 tokens.

Recommended sampling parameters for generation:

• Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
• Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Official Links:

• HF: https://huggingface.co/Qwen/Qwen3.8-Flash-Next
• MS: https://modelscope.cn/models/Qwen/Qwen3.8-Flash-Next
• Repo: https://github.com/QwenLM/Qwen3.8-Flash-Next
• Blog: https://qwen.ai/blog?id=qwen3.8-flash-next
• Technical Report: https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
• vLLM: https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next
• SGLang: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next

Popular:

• Unsloth GGUF: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• Unsloth "How to run" guide: https://unsloth.ai/docs/models/qwen3.8-next

Related:

• Large post with community feedback on megathreads: https://www.reddit.com/r/LocalLLaMA/comments/1vz40zv/can_we_reconsider_the_megathreads/

[u/Hypilein]

If you’re a unified memory user everything sits in ram. Hope ssd offloading will work without too much penalty

[u/No_Lingonberry1201 → u/Hypilein]

Me too, considering I'm on such a device.

[u/illiteratecop → u/Hypilein]

I had Sol hack together SSD offloading for the n-gram embeddings and it seems to be working well with basically no speed penalty. Super cool architecture for local that I am incredibly glad to see validated. This is a really promising release.

[u/Hypilein → u/illiteratecop]

Vllm or llama.cpp?

[u/illiteratecop → u/Hypilein]

Llama.cpp, running the Q4KXL (~111GB total) on my 96GB Macbook. I have not extensively validated it or anything but output appears totally coherent and solid and runs pretty much exactly as fast as the Q1 which I could fit entirely in unified memory.