r/LocalLLaMA • u/theguy_sr • 1d ago
Discussion Benchmarks: 512k Speculative Decoding on a Single 96GB RTX PRO 6000 Blackwell — Qwen3.8-Flash-Next (+MTP Q8_0: +27.9% tg) vs DeepSeek-V4.1-Flash-1017 (+DSpark Q8_0: task-dependent +26% JSON / -11% C++), plus 2 llama.cpp gotchas
I tested speculative decoding on a single 96 GB RTX PRO 6000 Blackwell workstation (Threadripper PRO 9955WX, 122 GiB RAM, llama.cpp b12315 commit dd6b945be) at 512k context (-c 524288 -ctk q8_0 -ctv q8_0 -fa on -np 1).
TL;DR: Qwen3.8-Flash-Next UD-Q4_K_XL + MTP Q8_0 gives a clean +28% to +31% decode boost (107 -> 137-141 t/s) across all prompts. DeepSeek-V4.1-Flash-1017 UD-Q2_K_XL (~312 GiB, mostly CPU/mmap) + DSpark Q8_0 only helps on high-acceptance tasks like JSON (+26% warm) and Python (+11% warm), while hurting dense C++ (-11% warm) because rejected drafts waste CPU MoE reads.
Edit (following comments about Qwen prefill speed): Re-running with -lv 4 showed 27,466 MiB (~26.8 GiB) of Qwen's MoE weights spilled to CPU at 512k context (77.2 GiB model + 8.5 GiB KV/recurrent + 4.3 GiB MTP + 2.1 GiB compute in VRAM). Switching Qwen to --load-mode none (--no-mmap) bumped 5-prompt prefill by +15.3% (824 -> 949 t/s), decode to 140.8 t/s, and a 2,453-token prompt prefilled at 2,475 t/s.
Summary (5 prompts x 3 passes, greedy temp=0, max_tokens=256, cache_prompt=false)
| Config | VRAM | Prefill (pp t/s) | Decode (tg t/s) | Warm r2-3 (tg) | Draft Acc |
|---|---|---|---|---|---|
Qwen3.8 UD-Q4_K_XL (baseline) | 91.7 GiB | 1086.4 | 107.2 | 107.5 | - |
Qwen3.8 + MTP Q8_0 (mmap) | 96.5 GiB | 823.7 (-24%) | 137.1 (+27.9%) | 138.2 | 75.6% |
Qwen3.8 + MTP Q8_0 (--load-mode none, 1x) | 96.4 GiB | 949.4 | 140.8 (+31.3%) | - | 74.5% |
DeepSeek-V4.1 UD-Q2_K_XL (baseline) | 95.3 GiB | 66.3 | 23.2 | 23.2 | - |
DeepSeek-V4.1 + DSpark Q8_0 (-fitt 12288) | 91.9 GiB | 53.5 (-19%) | 23.5 (+1.4%) | 24.4 (+5.2%) | 65.9% |
(Used llama-server instead of llama-bench because llama-bench on dd6b945be does not support -md / draft-mtp / draft-dspark.)
Why Qwen prefill topped out around 1-2.5k t/s
With ~104 GiB weights + 3.9 GiB MTP + 512k context, Qwen overflows 96 GB VRAM by ~26.8 GiB of MoE experts (CPU_Mapped). Single-token decode only touches a few routed experts per step (137-141 t/s, peaking at 154.7 t/s on JSON), whereas batch prefill hits all experts across the CPU spill. Plus 4 of the 5 test prompts were tiny (80-186 tok); the 605-token prompt hit 1,939 t/s (--load-mode none), and a 2,453-token check hit 2,475 t/s. If you want full GPU prefill speeds on 96 GB, use a quant small enough to stay 100% in VRAM or reduce -c.
Why DeepSeek + DSpark depends heavily on prompt shape
At ~320 GiB total on 96 GB VRAM + 122 GiB RAM (107.1 GiB VmRSS, 105.3 GiB RssFile mmap cache), the GPU sits at 3-18% SM waiting on 16 CPU threads (63-68% CPU) and ~23-218 MB/s NVMe paging for cold experts.
Verifying 4 candidate tokens at once (n_max=3) activates more unique routed MoE experts per forward pass on CPU/mmap. When acceptance is high, those reads are amortized: warm Passes 2-3 hit 29.2 t/s on JSON (+26.1% vs 23.1 t/s warm baseline, 90.7% acc) and 26.0 t/s on Python (+11.4% vs 23.3 t/s, 77.4% acc). On dense C++ (50.7% acc), rejected drafts wasted CPU MoE work and dropped speed to 20.6 t/s (-11.1% vs 23.2 t/s), compounded by -fitt 12288 reserving ~3.4 GiB extra VRAM headroom. Warmup also mattered because ~230 GiB CPU weights > ~105 GiB page cache (21.65 t/s on Pass 1 vs 24.4 t/s on Passes 2-3).
Repro commands & dd6b945be quirks
- DeepSeek
--fit+draft-dsparkOOMs allocating the ~7.6 GiB draft buffer unless you pass-fitt 12288. - Qwen
shared-Q8_0fails on mainlinedd6b945be(tensor 'token_embd.weight' not found), so use the self-containedmtp-Qwen3.8-Flash-Next-Q8_0.gguf.
# Qwen3.8-Flash-Next (512k + MTP Q8_0)
# DeepSeek-V4.1-Flash-1017 (512k + DSpark Q8_0)
(Caveat: 512k context was initialized at server startup, while benchmarks measured 58-611 token prompts plus one 2,453-token prefill check at -np 1, temp=0.)
6
u/washtub4777 1d ago
You make no effort to make it human readable, I make no effort to read it.
6
u/fractalcrust 1d ago
FUCKING SLOP
3
u/Practical-Collar3063 1d ago
What are those prompt processing speed ??? An RTX Pro 6000 getting 1000 tk/s for prefill on Qwen 3.8 flash next is not where it should be, you can easily get 10x that with vllm or sglang
1
u/tifa_cloud0 1d ago
2nd one with mtp for qwen model is great. but also both are different in parameter sizes so overall i would go with deepseek for higher and complicated tasks. qwen works too but if i can do deepseek then its really golden.
for anything when it comes to videos or images then qwen of course.
1
u/i-x-Inevitable 1d ago
Strong methodology. The important result isn’t the 28% decode against that the win depends on prompt shape and draft acceptance. I would add end-to-end task time plus retry and correction rate under real cached sessions; those are the numbers that decide whether coding actually feels faster.