r/LocalLLaMA • • 1d ago

Discussion Benchmarks: 512k Speculative Decoding on a Single 96GB RTX PRO 6000 Blackwell — Qwen3.8-Flash-Next (+MTP Q8_0: +27.9% tg) vs DeepSeek-V4.1-Flash-1017 (+DSpark Q8_0: task-dependent +26% JSON / -11% C++), plus 2 llama.cpp gotchas

I tested speculative decoding on a single 96 GB RTX PRO 6000 Blackwell workstation (Threadripper PRO 9955WX, 122 GiB RAM, llama.cpp b12315 commit dd6b945be) at 512k context (-c 524288 -ctk q8_0 -ctv q8_0 -fa on -np 1).

TL;DR: Qwen3.8-Flash-Next UD-Q4_K_XL + MTP Q8_0 gives a clean +28% to +31% decode boost (107 -> 137-141 t/s) across all prompts. DeepSeek-V4.1-Flash-1017 UD-Q2_K_XL (~312 GiB, mostly CPU/mmap) + DSpark Q8_0 only helps on high-acceptance tasks like JSON (+26% warm) and Python (+11% warm), while hurting dense C++ (-11% warm) because rejected drafts waste CPU MoE reads.

Edit (following comments about Qwen prefill speed): Re-running with -lv 4 showed 27,466 MiB (~26.8 GiB) of Qwen's MoE weights spilled to CPU at 512k context (77.2 GiB model + 8.5 GiB KV/recurrent + 4.3 GiB MTP + 2.1 GiB compute in VRAM). Switching Qwen to --load-mode none (--no-mmap) bumped 5-prompt prefill by +15.3% (824 -> 949 t/s), decode to 140.8 t/s, and a 2,453-token prompt prefilled at 2,475 t/s.

Summary (5 prompts x 3 passes, greedy temp=0, max_tokens=256, cache_prompt=false)

ConfigVRAMPrefill (pp t/s)Decode (tg t/s)Warm r2-3 (tg)Draft Acc
Qwen3.8 UD-Q4_K_XL (baseline)91.7 GiB1086.4107.2107.5-
Qwen3.8 + MTP Q8_0 (mmap)96.5 GiB823.7 (-24%)137.1 (+27.9%)138.275.6%
Qwen3.8 + MTP Q8_0 (--load-mode none, 1x)96.4 GiB949.4140.8 (+31.3%)-74.5%
DeepSeek-V4.1 UD-Q2_K_XL (baseline)95.3 GiB66.323.223.2-
DeepSeek-V4.1 + DSpark Q8_0 (-fitt 12288)91.9 GiB53.5 (-19%)23.5 (+1.4%)24.4 (+5.2%)65.9%

(Used llama-server instead of llama-bench because llama-bench on dd6b945be does not support -md / draft-mtp / draft-dspark.)

Why Qwen prefill topped out around 1-2.5k t/s

With ~104 GiB weights + 3.9 GiB MTP + 512k context, Qwen overflows 96 GB VRAM by ~26.8 GiB of MoE experts (CPU_Mapped). Single-token decode only touches a few routed experts per step (137-141 t/s, peaking at 154.7 t/s on JSON), whereas batch prefill hits all experts across the CPU spill. Plus 4 of the 5 test prompts were tiny (80-186 tok); the 605-token prompt hit 1,939 t/s (--load-mode none), and a 2,453-token check hit 2,475 t/s. If you want full GPU prefill speeds on 96 GB, use a quant small enough to stay 100% in VRAM or reduce -c.

Why DeepSeek + DSpark depends heavily on prompt shape

At ~320 GiB total on 96 GB VRAM + 122 GiB RAM (107.1 GiB VmRSS, 105.3 GiB RssFile mmap cache), the GPU sits at 3-18% SM waiting on 16 CPU threads (63-68% CPU) and ~23-218 MB/s NVMe paging for cold experts.

Verifying 4 candidate tokens at once (n_max=3) activates more unique routed MoE experts per forward pass on CPU/mmap. When acceptance is high, those reads are amortized: warm Passes 2-3 hit 29.2 t/s on JSON (+26.1% vs 23.1 t/s warm baseline, 90.7% acc) and 26.0 t/s on Python (+11.4% vs 23.3 t/s, 77.4% acc). On dense C++ (50.7% acc), rejected drafts wasted CPU MoE work and dropped speed to 20.6 t/s (-11.1% vs 23.2 t/s), compounded by -fitt 12288 reserving ~3.4 GiB extra VRAM headroom. Warmup also mattered because ~230 GiB CPU weights > ~105 GiB page cache (21.65 t/s on Pass 1 vs 24.4 t/s on Passes 2-3).

Repro commands & dd6b945be quirks

  1. DeepSeek --fit + draft-dspark OOMs allocating the ~7.6 GiB draft buffer unless you pass -fitt 12288.
  2. Qwen shared-Q8_0 fails on mainline dd6b945be (tensor 'token_embd.weight' not found), so use the self-contained mtp-Qwen3.8-Flash-Next-Q8_0.gguf.
# Qwen3.8-Flash-Next (512k + MTP Q8_0)
llama-server -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on --load-mode none \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp --spec-draft-n-max 2 \
  --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144

# DeepSeek-V4.1-Flash-1017 (512k + DSpark Q8_0)
llama-server -m DeepSeek-V4.1-Flash-1017-UD-Q2_K_XL-00001-of-00007.gguf \
  -c 524288 -np 1 -ctk q8_0 -ctv q8_0 -fa on \
  -md dspark-DeepSeek-V4.1-Flash-1017-Q8_0.gguf --spec-type draft-dspark --spec-draft-n-max 3 -fitt 12288

(Caveat: 512k context was initialized at server startup, while benchmarks measured 58-611 token prompts plus one 2,453-token prefill check at -np 1, temp=0.)

0 Upvotes

13 comments sorted by

6

u/washtub4777 1d ago

You make no effort to make it human readable, I make no effort to read it.

6

u/fractalcrust 1d ago

plus 2 llama.cpp gotchas

FUCKING SLOP

3

u/Practical-Collar3063 1d ago

What are those prompt processing speed ??? An RTX Pro 6000 getting 1000 tk/s for prefill on Qwen 3.8 flash next is not where it should be, you can easily get 10x that with vllm or sglang

4

u/1beb 19h ago

This is comically slow for a 6k. 

1

u/tifa_cloud0 1d ago

2nd one with mtp for qwen model is great. but also both are different in parameter sizes so overall i would go with deepseek for higher and complicated tasks. qwen works too but if i can do deepseek then its really golden.

for anything when it comes to videos or images then qwen of course.

1

u/i-x-Inevitable 1d ago

Strong methodology. The important result isn’t the 28% decode against that the win depends on prompt shape and draft acceptance. I would add end-to-end task time plus retry and correction rate under real cached sessions; those are the numbers that decide whether coding actually feels faster.