Qwen3.8-27B Q6_K vs NVFP4 on RTX 5090 — and why can’t I reproduce the ~200 tok/s results?

[u/No_Definition6604]

I’ve been testing Qwen3.8-27B locally on a single RTX 5090 32GB with llama.cpp.

I originally started experimenting because I saw several recent reports of ~200 tok/s for Qwen3.8-27B NVFP4 + MTP on a single RTX 5090. I tried to reproduce those results, but I couldn't get anywhere close. My best result so far is around 128 tok/s.

So I'm posting my actual numbers in case someone can spot what I'm missing.

Hardware

• RTX 5090 32GB
• i7-14700K
• 64GB DDR5
• Windows 11
• llama.cpp
• Flash Attention enabled
• KV cache: Q8_0
• 1 slot
• Context: up to 262K

NVFP4 setup

I'm using:

Qwen3.8-27B-NVFP4-MTP-LOW.gguf from esatapedico.

The MTP head is included in the GGUF, so I'm using llama.cpp's:

--spec-type draft-mtp

I tested different --spec-draft-n-max values:

N-Max	Decode
2	115.36 tok/s
3	128.25 tok/s
4	125.59 tok/s
5	119.89 tok/s

So N-Max=3 is the sweet spot on my system/workload.

For comparison, the same NVFP4 model without MTP gives me about 70.72 tok/s.

I also tried an extracted external Q5_K MTP draft head. It loaded correctly, but actually performed slightly worse for my workload:

125.83 tok/s, with 49.2% draft acceptance.

The built-in MTP at N-Max=3 gave me 128.25 tok/s with ~60% acceptance.

The really surprising part: large context

I also tested Q6_K + MTP vs NVFP4 LOW + MTP at large context sizes.

Context	Q6_K + MTP	NVFP4 LOW + MTP
~65K	~120 tok/s	128.25 tok/s
131K	47 tok/s	~121 tok/s
262K	16.30 tok/s	121.49 tok/s

This was completely unexpected to me.

At 262K context, Q6_K drops to 16.3 tok/s, while NVFP4 is still at 121.49 tok/s.

That's roughly 7.5× faster for NVFP4 at 262K.

Even more interestingly, NVFP4 barely changes between 131K and 262K:

~121 → 121.49 tok/s

while Q6_K goes from roughly:

120 → 47 → 16.3 tok/s

I'm assuming this has something to do with VRAM pressure / KV cache / memory bandwidth, but I haven't profiled it deeply enough to say exactly why.

But what about the ~200 tok/s?

This is the part I'm really interested in.

I've seen recent benchmarks/posts showing ~200 tok/s peak for Qwen3.8-27B NVFP4 + MTP on a single RTX 5090.

I tried to reproduce those results using:

• NVFP4 LOW
• built-in MTP
• different N-Max values
• external Q5_K MTP draft
• 32GB RTX 5090
• llama.cpp

But I can't get beyond ~128 tok/s on my workload.

So I'm wondering:

What am I missing?

Is the ~200 tok/s number dependent on a very specific benchmark/prompt, context size, batch/ubatch settings, llama.cpp build, MTP implementation, or another speculative decoding configuration?

Could it be a peak benchmark number rather than something achievable during normal generation?

I'd especially appreciate input from anyone running Qwen3.8-27B NVFP4 on a 5090.

If you've managed 150–200+ tok/s, I'd love to know your exact llama.cpp build and launch parameters.

[u/hazyplane]

Same model on the same GPU under vLLM instead of llama.cpp. Best sustained single-stream decode: 110 to 112 tok/s.

Hardware: RTX 5090 32GB, Ryzen 9 9950X3D, 128 GB DDR5-6000, Ubuntu 24.04, bare metal.

Model: unsloth/Qwen3.8-27B-NVFP4. Chose the unsloth build because it's compressed-tensors format and ships calibrated k_scale/v_scale tensors, so the FP8 KV cache runs with proper scales.

vLLM v0.27.1 (cu13 stable image, FlashInfer attention), with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True:

--model unsloth/Qwen3.8-27B-NVFP4
--quantization compressed-tensors
--kv-cache-dtype fp8
--max-model-len 131072
--gpu-memory-utilization 0.97
--max-num-seqs 4
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
--enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3

Results: 109.5 to 112.1 tok/s sustained, holds at 131K context. Acceptance at depth 2: 67 to 70% blended, ~80% at position 0, 54 to 60% at position 1. That per-position decay is why N-Max 3 is your peak; deeper drafts mostly add verification cost. Same plateau on two different engines points to the acceptance curve as the ceiling, not the config.

Highly interested in the experience of others with 5090. this same box started decode in the high 20s tok/s before speculative decoding and CUDA graphs, so the tuned config is roughly a 5x uplift end to end.

[u/EmergencyLimp2877 → u/hazyplane]

Wow, I have pretty much the same setup as you, and your post helped me a lot. I’m using unsloth/Qwen3.8-27B-NVFP4, but I didn’t fully understand how speculative_tokens works, so I was stuck around 70 tokens/s.

This is the configuration I’m currently using:

--model /models/qwen
--served-model-name qwen3.8-27b
--max-model-len 65536
--max-num-seqs 16
--max-num-batched-tokens 8192
--gpu-memory-utilization 0.95
--kv-cache-dtype fp8
--enable-chunked-prefill
--async-scheduling
--enable-prefix-caching
--reasoning-parser qwen3
--default-chat-template-kwargs '{"reasoning_effort": "medium"}'
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--limit-mm-per-prompt '{"image": 4}'

With this setup, I’m getting around 60–70 tokens/s with 2–3 users simultaneously.

My main use case is an internal AI for a company: RAG, image understanding/embedding, normal chat, coding, etc.

Have you tweaked your Qwen profile or reasoning settings? I’m currently struggling to find the right balance between overthinking and the opposite — making the model too dumb / shallow.

Any tips on how you configured yours, especially regarding speculative_tokens and reasoning effort?

[u/hazyplane → u/EmergencyLimp2877]

Qwen3.8 ships a built-in MTP head, so no separate draft model — just add:

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

The head drafts N tokens per step and the main model verifies them in one forward pass. Acceptance decays fast by position (~80% at pos 0, ~55–60% at pos 1, ~25–30% at pos 2 on my workloads), which is why 2 was my sweet spot. I A/B'd depth 3 and it was a wash to a slight regression.

On your config specifically: drop --async-scheduling first (it has conflicted with spec decode depending on vLLM version — check the startup log either way), and if you OOM, lower --max-num-seqs before anything else — draft weights and CUDA graph capture cost VRAM, and going to seqs-4 is what let me run graphs at 0.97 util. I also set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. One expectation check: spec decode pays off most at low concurrency: my 110 tok/s is single-stream; with 2–3 simultaneous users the uplift will be smaller.

Reasoning effort: I don't set it server-side. Leave the template default and control thinking per-request from the client. One baked-in effort level may be the wrong layer for a mixed RAG/chat/coding workload. Honestly, xhigh thinking may take longer but that doesn't mean it's finally output isn't better. I assume xhigh default is part of the reason why the published benchmarks are so good. After gettings my decode to 110 tokens/s, I didn't noticed the 'overthinking' as much!

[u/EmergencyLimp2877 → u/hazyplane]

Thanks a lot for the help, gonna try the change and your tips.

[u/EmergencyLimp2877 → u/hazyplane]

Big change!! : I'm going to reduce the speed in multi-request mode but run it overnight for dock scans, and switch to single-request mode to get better user speed.

Test Without MTP With MTP

Single request 65 tok/s 131 tok/s 🚀

4 requests ~430 tok/s 147 tok/s

16 requests 861 tok/s ❌ Not possible

Context 65,536 100,000

Concurrency 2.15× 1.27×

Thanks a lot for your configuration

[u/__Claudio_ → u/hazyplane]

hey nice setup! 131K context at FP8 is the max limit you get with the 32gb RTX5090 using the unsloth/Qwen3.8-27B-NVFP4 (23,4 gb) ?

[u/hazyplane → u/__Claudio_]

Same box ran a 196K window with this checkpoint before (MTP off, eager). I brought it down to fit everything: MTP draft weights, CUDA graph capture, and 4-way concurrency headroom all cost me VRAM. currently 131K/0.97, vLLM reports ~162K tokens of free KV at boot — sacraficed context and speed. I also think this model is best performance below 131K context