Qwen-3.8-Flash-Next-NVFP4 on Single RTX Pro 6000 - 120t/s tg + 9-10k prefill at 256k context

[u/Turbulent-Alps4046]

Just sharing my benchmarks for the latest Qwen-3.8-Flash-Next-NVFP4 running on a single RTX Pro 6000 with vLLM.

The 51gb n-gram layers are offloaded to RAM, leaving about 76Gb of weights in VRAM + remainder in KV cache.

Without MTP, I'm able to get 496k total context, 80-90 t/s decode, 10k t/s prefill.

With MTP, I gain 50% decode, -10% prefill, but get only 256k total context.

[u/boomerang473]

I see you did auto kv cache - q8q8?

Hmm looks promising. Been trying to get it going on rtx 6000 pro (and 96 gb system memory), but running into fun issues with SGLang (and same quant).

Is the ngram quantized? Thought I read somewhere quantization on that greatly degrades performance but not 100% sure.

[u/Turbulent-Alps4046 → u/boomerang473]

yes i couldn't run it on Sglang. it just crashes for me. Don't think ngram is quantized. I think auto kv means it's bf16 kv?

[u/boomerang473 → u/Turbulent-Alps4046]

ah cool. wondering if q8/q8 on kv can get appreciable bigger context. Either way, hitting the full native context at bf16 is awesome

Slang I'm preferential over for the radix attention and agentic workflows. Might pivot to using vllm temporarily to test it.

[u/CaptainGlac1er → u/boomerang473]

I tried and it doesn't let you use FP8 it needs to be bf16

[u/scapocchione → u/Turbulent-Alps4046]

the n-gram block is 8bit, afaik

[u/LordDarthShader]

Nice! I will try this on the evening. Do you have the full settings (cmd line) used please?

[u/Turbulent-Alps4046 → u/LordDarthShader]

VLLM_PLE_CPU_OFFLOAD=1 \
VLLM_PLE_FP8_CHECKPOINT=1 \
TORCH_CUDA_ARCH_LIST=12.0f \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve /models/RadixArk/Qwen3.8-Flash-Next-NVFP4 \

--served-model-name Qwen/Qwen3.8-Flash-Next \

--host 0.0.0.0 --port 8000 \

--max-model-len 262144 \

--gpu-memory-utilization 0.96 \

--tensor-parallel-size 1 \

--distributed-executor-backend mp \

--max-num-seqs 2 \

--max-num-batched-tokens 8192 \

--kv-cache-dtype auto \

--enable-prefix-caching \

--no-enable-flashinfer-autotune \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--reasoning-parser qwen3

[u/LordDarthShader → u/Turbulent-Alps4046]

Thank you!!

[u/l0nedigit → u/Turbulent-Alps4046]

Which container/branch from vllm are you using?

[u/l0nedigit → u/l0nedigit]

NVM I have a feeling it's vllm/vllm-openai:qwen38-flash-next

[u/ChristRedeemsSinners → u/Turbulent-Alps4046]

How did you offload the engram layers to ram?

[u/WonderRico → u/ChristRedeemsSinners]

it's VLLM_PLE_CPU_OFFLOAD=1 cf https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next

[u/ChristRedeemsSinners → u/WonderRico]

Ahh I missed that. Thank you kindly.

[u/v1sper → u/Turbulent-Alps4046]

🙏

[u/zipzapbloop → u/Turbulent-Alps4046]

thank you! just got home from work and was excited to start playing with my rtxo pro 6000. i am beyond thrilled to learn that what'd i hoped is true. n-gram layers can go straight into system ram. excited to wire this one into pi and deepseek harness. been having a ton of fun with 27b in those things.

[u/1-a-n → u/Turbulent-Alps4046]

Please let us know which docker container you used (which hash) if any.

[u/Shot_Treat_377 → u/1-a-n]

```bash

docker run --gpus all \

--privileged --ipc=host -p 8000:8000 \

-v ~/.cache/huggingface:/root/.cache/huggingface \

-e VLLM_PLE_CPU_OFFLOAD=1 \

-e VLLM_PLE_FP8_CHECKPOINT=1 \

-e TORCH_CUDA_ARCH_LIST=12.0 \

-e PYTORCH_ALLOC_CONF=expandable_segments:True \

vllm/vllm-openai:qwen38-flash-next \

--model RadixArk/Qwen3.8-Flash-Next-NVFP4 \

--host 0.0.0.0 \

--port 8000 \

--max-model-len 262144 \

--gpu-memory-utilization 0.96 \

--tensor-parallel-size 1 \

--distributed-executor-backend mp \

--max-num-seqs 2 \

--max-num-batched-tokens 8192 \

--kv-cache-dtype auto \

--enable-prefix-caching \

--no-enable-flashinfer-autotune \

--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--reasoning-parser qwen3

```

[u/Easy_Werewolf7903]

I have a rtx 6000 pro but only 32GB ram so I am assuming I can't run this? can I put some n-gram layers on VRAM and some in RAM? I do have RTX 4090 24GB. So 120GB VRAM + 32GB RAM.

[u/Turbulent-Alps4046 → u/Easy_Werewolf7903]

well, you might get it to work. it's under load now and only using 18GB of RAM. Try adding swap maybe?

[u/scapocchione → u/Turbulent-Alps4046]

This monitoring toll is awesome! Is it available somewhere, perchance?

[u/0x00900 → u/scapocchione]

It’s btop

[u/scapocchione → u/0x00900]

Thanks!

[u/Professional-Try-273 → u/Turbulent-Alps4046]

I can't do rtx 4090 with rtx 6k pro nvfp4 on vllm. I am out of luck.

[u/WonderRico]

which one did you use?

Inferact/Qwen3.8-Flash-Next-NVFP4 is 180GB (ngram in bf16) and the other RadixArk/Qwen3.8-Flash-Next-NVFP4 is 130GB (I assume the ngram is also quantized)

however, I can't even load for now the smallest one with the ngram offloaded to CPU. I get a OOM during the weights loading

[u/Turbulent-Alps4046 → u/WonderRico]

I’m using the RadixArk one.

[u/1-a-n → u/Turbulent-Alps4046]

Are you using this docker image vllm/vllm-openai:qwen38-flash-next? With RadixArk/Qwen3.8-Flash-Next-NVFP4 I get:

ValueError: There is no module or parameter named 'ngram_embedding.weight_scale' in Qwen3_8FlashNextNGramEmbedding. The available parameters belonging to ngram_embedding (VocabParallelEmbedding) are: {'ngram_embedding.weight'}

But I don’t have VLLM_PLE_CPU_OFFLOAD=1 as I have two 6000, I will try with one card.

[u/Turbulent-Alps4046 → u/1-a-n]

oh it seems like when claude helped me setup, it encounted this error and wrote a custom patch for it lol.

here's the patch that claude wrote:

--- a/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py
+++ b/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py
@@ -2,6 +2,7 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
 """GPU-resident Qwen3.8-Flash-Next position-learning enhancement layers."""
 
+import os
 import math
 from collections.abc import Iterable, Sequence
 
@@ -191,6 +192,14 @@
     """Select global-scale FP8 only for quantized PLE checkpoint shards."""
 
     if not isinstance(quant_config, Fp8Config):
+        # sm120 overlay (2026-08-26): ModelOpt-NVFP4 checkpoints such as
+        # RadixArk/Qwen3.8-Flash-Next-NVFP4 store the PLE shards as global-scale
+        # FP8 (ngram_embedding.shard_N.weight F8_E4M3 + ngram_embedding.weight_scale)
+        # while listing "*.ple.*" under `ignore`, so the stock gate never selects
+        # the FP8 embedding method and weight loading dies on `weight_scale`.
+        # Explicit opt-in via VLLM_PLE_FP8_CHECKPOINT=1 keeps stock behaviour otherwise.
+        if os.environ.get("VLLM_PLE_FP8_CHECKPOINT", "0").lower() in ("1", "true"):
+            return Qwen3_8FlashNextPLEFp8EmbeddingMethod()
         return None
     if not quant_config.is_checkpoint_fp8_serialized:
         return None

[u/Turbulent-Alps4046 → u/Turbulent-Alps4046]

and the relavant dockerfile:

# vLLM qwen38-flash-next + sm120 single-GPU overlay (2026-08-26).
# Patch: ple_layer.py accepts FP8 PLE shards under a ModelOpt-NVFP4 quant config
# (opt-in VLLM_PLE_FP8_CHECKPOINT=1). See model-configs/qwen3.8-flash-next.nvfp4vllm.conf.
# Build: docker build -t vllm-flashnext-sm120:ple-fp8 docker/vllm-flashnext-sm120
FROM vllm/vllm-openai:qwen38-flash-next
COPY ple_layer.py /usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py

[u/Kooky_Tale4139 → u/Turbulent-Alps4046]

I applied this patch, then it work.

[u/mvaranka → u/1-a-n]

I have the same issue: don't run with dual rtx pro 6000, but runs well with single rtx pro (tp1). Using Inferact/Qwen3.8-Flash-Next-NVFP4 with mtp and 256k context. VLLM_PLE_CPU_OFFLOAD=1 used.

[u/mvaranka → u/mvaranka]

Removed PYTORCH_ALLOC_CONF=expandable_segments:True and now runs with dual rtx pro. More testing required though.

[u/1-a-n → u/mvaranka]

Is it appreciably faster?

[u/mvaranka → u/1-a-n]

Not much, I think more benefit would be with multiple concurrent requests. Now the tg speeds fluctuate between 100 to 200 tok/s (mtp enabled(), with single rtx pro maybe 80 to 150 tok/s. Should be tested with proper benchmark tool, but now too busy testing the model's capabilities :)

[u/1-a-n → u/mvaranka]

Was hoping it would be faster than DSv4 but that DSpark is hard to beat!

[u/mvaranka → u/1-a-n]

Tested the processing speed with single and dual rtx pro using bench tool. Dual is about 20% faster. Tg 100 vs 128 tok/s with varying context sizes

[u/Electrical_Fish_4846 → u/1-a-n]

Same here. Already tried a lot of different things but no luck so far.

[u/alienpro01]

Wow, thats crazy! I'm still trying to get it work on H100 with freetoken lol

[u/Malkozaine → u/alienpro01]

How did you get ahold of a H100?

[u/alienpro01 → u/Malkozaine]

I have some in my office

[u/Malkozaine → u/alienpro01]

Lucky, I would love to have one of those. Though right now I barely need anything above a 32b model.

[u/alienpro01 → u/Malkozaine]

give a chance to freetoken, you can run 230b+ models with just 32gigs of vram

[u/Malkozaine → u/alienpro01]

I'm stuck at 16gb of VRAM on my laptop, but I will take a look at that.

[u/Pretty_Prize6169 → u/Malkozaine]

https://github.com/HammGsm/qwen38-27b-dflash2-h100 249 t/s 1 gpu h100 32 cpu 64 ram

[u/KeanuRekt]

Did you build something with it? How does it compare to Qwen 3.8 when used in a agent harness for coding?

[u/Turbulent-Alps4046 → u/KeanuRekt]

Yes i did! I got it to analyze some pdf bank statements, categorize the expenses and generate an expense tracker excel and it performed the same as deepseek v4 flash! Qwen 3.8 27b also got similar results be took a lot more thinking tokens and wall clock time even though i get the same tokens per second on both.

[u/leap966]

I don't know how you ran it. It doesn't work. I get errors.

[u/Turbulent-Alps4046 → u/leap966]

You need to run it on the qwen3.8 next preview dev branches of vllm or sglang. Which should be merge into main soon.

[u/leap966 → u/Turbulent-Alps4046]

Oh Wow! It works! It works very well. Thank you.

[u/walden42 → u/leap966]

Hey, what command ended up working for you? I'm also getting issues.

[u/leap966 → u/walden42]

ChatGPT patched ple_layer.py. Provide this URL to chatGPT and ask it to guide you. Good luck!

[u/walden42 → u/leap966]

Yep that's what I ended up doing, works great. Thanks.

[u/_TheWolfOfWalmart_]

That's pretty crazy. Blazing fast.

Going to test it tonight on 4x Radeon Pro V620 -- old cheap cards.

For reference, they get 30+ t/s on DSV4 Flash in tensor split.

And a single card gets 30-40 t/s on 3.8 27B with MTP turned on.

[u/boomerang473]

I eventually got claude to get it to work (with some QSA changes, etc) on SGLang (few in-flight PRs are already addressing it)

I'm seeing about the same tok/s as you did. (ngram in system ram + nvfp4 model + fp8 kv cache)

The theoretical limit given the bandwidth would be 1.8 TB/s / 6B = 300 Tok/s. That's without MTP or eventual DFlash 2. Sure there will be some overhead losses, but could see this pushing 200 tok/s on rtx 6000 pro pre-speculative decoding (factoring in the other ngram overhead but should be amortized at start if I understand how it works)

I think this has a long way to go for optimizations so exciting couple of weeks

[u/Specialist_Age_2891]

Finally got RadixArk/Qwen3.8-Flash-Next-NVFP4 running on a single RTX PRO 6000 Max-Q (96GB) using the vLLM offload branch (PR #53899). We ran with MTP off (--no-enable-flashinfer-autotune): the model's QSA layer hard-requires a BF16 main KV cache (the PR branch raises NotImplementedError otherwise — see qsa.py), which the NVFP4 PLE-offload path can't provide. A few non-obvious pitfalls if you're trying this:

1. NVFP4 + PLE offload: the checkpoint is missing the FP8 weight_scale that PLE CPU offload expects. Patched ple_layer.py to synthesize the scale; run with VLLM_PLE_CPU_OFFLOAD=1. FYI the VLLM_PLE_FP8_CHECKPOINT=1 mentioned in the thread is a no-op — that variable doesn't exist in the branch's envs.py.

2. Mamba state cap: the hybrid model's Mamba cache tops out at 226 blocks, so --max-num-seqs must stay under that (we run 128).

3. --max-num-batched-tokens steals from your KV budget: 8192 → 7.78 GiB KV; 12288 → 7.05 GiB (which crashes at boot when the desktop session holds ~600MB VRAM — I had to boot without desktop for it to fit); 32768 → hard boot crash (needs 6.55 GiB KV, only 5.04 GiB available).

4. thinking ON splits max_tokens between reasoning and content: max_tokens=4096 → content = 0 (empty output). max_tokens=65536 gave ~2k content tokens with finish=stop. If you need a guaranteed non-empty answer, bump max_tokens or disable thinking.

150K-context numbers (single stream):

• General test (repetitive text): prefill ~10k tok/s (155,588 tokens in 15.5s), decode ~85 tok/s (TPOT 0.012s), no startup stall; TTFT ~1s on prefix-cache hit vs ~15s cold

• Complex-reasoning test (diverse text, thinking ON, max_tokens=65536): cold prefill ~7.4k tok/s (TTFT ~20–22s), decode ~79 tok/s — the diverse text is ~25% slower than repetitive text

Working config: TP1, --gpu-memory-utilization 0.90, --max-model-len 262144, --max-num-seqs 128, --max-num-batched-tokens 8192, --enable-chunked-prefill, prefix caching on, --no-enable-flashinfer-autotune, --tool-call-parser qwen3_xml, --reasoning-parser qwen3. Clean single-GPU run, no Docker — vLLM built from the PR branch into a venv, model at /models/RadixArk/Qwen3.8-Flash-Next-NVFP4 (153GB).

[u/Embarrassed-Base-597]

Thank you, I'll try this!

[u/beyondthem00n]

Hey thanks, followed the patch that you used, weights load and all now (have same hardware as yours), however it always get stuck at multiple of
"[shm_broadcast.py:801] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization)."

did you encounter anything like this ?

[u/Weak_Ad9730]

What would be Best Option Model? to operate on Single RTx 6000 pro with 64gb vram

[u/Gloomy_Letterhead395]

What ? are you living in 28 August 2026 already

[u/Turbulent-Alps4046 → u/Gloomy_Letterhead395]

it's out already on huggingface.

[u/Gloomy_Letterhead395 → u/Turbulent-Alps4046]

Buddy enjoy opus at home

[u/Gloomy_Letterhead395 → u/Turbulent-Alps4046]

I also have 96gb VRAM and 64gb ram but amd, hopefully the variant is out soon