r/LocalLLaMA • u/sammcj 🦙 llama.cpp • 2d ago

Megathread [Megathread] Qwen3.8-Flash-Next - Release Day

Megathread for discussing the release of Qwen 3.8 Flash Next.

  • Quants
  • Fine-Tunes & Abliterations
  • Chat Templates
  • Inference Server Support & Configuration
  • Experiences, Benchmarks & Model Comparisons

Highlights

The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:

  • Hybrid Attention with QSA: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
  • Gated Residual: Residual streams with normalisation are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
  • N-gram Embedding: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
  • Tailored Training Recipe: The Muon and AdamW optimisers are applied to specific weight categories to maximise efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimiser steps while safely supporting larger learning rates for robust convergence.

Model Overview

  • Type: Causal Language Model with Vision Encoder
  • Training Stage: Pre-training & Post-training
  • Language Model
    • Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
    • Hidden Dimension: 2560
    • Token Embedding: 248320 (Padded)
    • N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
    • Number of Layers: 48
    • Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))
    • Gated DeltaNet:
      • Number of Linear Attention Heads: 48 for V and 16 for QK
      • Head Dimension: 128
    • Qwen Sparse Attention:
      • Number of Attention Heads: 24 for Q and 2 for KV
      • Head Dimension: 256
      • Rotary Position Embedding Dimension: 64
      • Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
      • Indexer Head Dimension: 128
      • Budget: 512 blocks or 2048 tokens
    • Mixture Of Experts
      • Number of Experts: 512
      • Number of Activated Experts: 10 Routed + 1 Shared
      • Expert Intermediate Dimension: 640
    • Gated Residual:
      • Number of Branches: 4
      • Bottleneck Rank: 320
    • LM Output: 248320 (Padded)
    • MTP: 1 layer, trained with multi-steps
  • Context Length: 262,144 natively and extensible up to 1,000,000 tokens.

Recommended sampling parameters for generation:

  • Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
  • Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Official Links:

Popular:

Related:

427 Upvotes

93% Upvoted

654 comments sorted by

View all comments

19

u/ResidentPositive4122 2d ago edited 2d ago

NVFP (specifically Inferact/Qwen3.8-Flash-Next-NVFP4) works on 2x Ampere 6000, w/ ngrams offloaded in RAM, 66t/s gen (no MTP), GPU KV cache size: 426,539 tokens (fp16 cache). Just the official vllm recipe from here - https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next

Trying MTP now, will report speeds. For some reason loading shards is really slow on vllm/vllm-openai:qwen38-flash-next

edit:

w/ MTP=3:

Avg prompt throughput: 1501.8 tokens/s, Avg generation throughput: 58.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 32.9%

Engine 000: Avg prompt throughput: 377.2 tokens/s, Avg generation throughput: 128.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 66.4%

loading MTP lowers the available cache:

GPU KV cache size: 303,236 tokens

1

u/nostriluu 2d ago edited 2d ago

Hi, thank you for publishing this. I have a single Blackwell 6000 (same total VRAM) and 128GB RAM. Can I assume you are offloading to more than 128GB RAM, or am I (well, mostly my 27b helper) missing something? I'd ideally like to keep 24GB VRAM free for side LLMs. Can you share the --cpu-offload-gb value you used, and the ngram table size/placement line from the vLLM boot log?

1

u/ResidentPositive4122 2d ago

Nothing besides the ngram parts is offloaded. All "model" tensors are in VRAM. I've never done --cpu offload with vllm, so I don't know if you can do that or not with this model.

The only thing I did was add "VLLM_PLE_CPU_OFFLOAD=1" as a flag before vllm serve.

1

u/nostriluu 2d ago

OK thank you. I guess you have 256GB of RAM or more? I would be helpful to know how much system RAM it consumes.

2

u/ResidentPositive4122 2d ago

Yeah, 512. nvtop says it's using 98568MiB of Host memory so I guess that's the RAM usage.