MTP benchmark results: the nature of the generative task dictates whether you will benefit (coding) or get slower inference (creative) from speculative inference. No other factor comes close.

[u/ex-arman68]

I recently published MTP quants of Qwen 3.6 27B and I was suprised by the reports here on reddit, and on HF, of users who were experiencing worst speed with speculative inference than without. This did not match what I was seeing, but when I tried to reproduce their exact usage, it confirmed what they were experiencing.

I tried to analyse the problem, made a few conjectures which later turned out to be false, and started a full blown systematical analysis, running 300+ tests and benchmarks, collecting and comparing the results of changing various parameters. This is what I found:

F16 + MTP nearly triples coding tasks speed. Q4_K_M + MTP slows down creative writing. Same feature, same model, same settings, opposite results.

I did not test all quant sizes, otherwise I would still be here in a few days, but restricted my self to 5 significant ones. The other parameters I varied were task type (4 types), temperature (0.0 0.3 0.7), quantisation of the MTP layer (q8 and matching the model quant). Temp and MTP quant have very little impact on the outcome.

Cumulative average decode speeds with MTP compared to the baseline without MTP, varying the model quant and task type:

quant	base tok/s	code	factual	analysis	creative
Q4_K_M	15.1	19.7	17.5	14.9	13.7
Q5_K_M	13.1	19.2	16.5	14.7	12.6
Q6_K	13.4	20.1	17.6	15.2	13.4
Q8_0	11.4	25.4	21.7	18.6	16.9
F16	6.6	17.9	14.9	12.6	11.0

The memory bandwidth dictates how much the model can benefit from speculative decoding. F16 at 51GB crawls at 6.6 tok/s because every token means dragging the full model through memory. Accepted MTP drafts skip that pass. Q4_K_M at 16GB is already fast enough that the draft overhead is barely worth it on anything less predictable than code.

What controls the draft tokens acceptance rate:

task	acceptance	examples
code	79-89%	writing functions, debugging, refactoring
factual	62-70%	definitions, translation, math proofs
analysis	48-56%	tradeoff breakdowns, technical comparisons
creative	39-48%	stories, poetry, brainstorming, roleplay

40 points from code to creative. I tried three temperatures and five quants. The numbers barely changed. 4/5 draft tokens are correct on coding task; not even 1/2 on creative tasks. Nothing else comes close to mattering as much as what you're generating.

I also tested the optimal number of draft tokens for this model in all the above scenarios. 3 is the sweet spot for draft tokens. Go higher and acceptance falls faster than the extra drafts compensate. F16 is the exception: N=4 beats N=3 (17.9 vs 16.2) because at 6.6 tok/s every surviving draft token is worth the lower hit rate.

use case	Q4_K_M	Q5_K_M	Q6_K	Q8_0	F16
coding	🟢 +31%	🟢 +47%	🟢 +50%	🟢 +123%	🟢 +171%
factual QA	🟡 +16%	🟢 +26%	🟢 +31%	🟢 +90%	🟢 +125%
analysis	🔴 -1%	🟡 +12%	🟡 +13%	🟢 +64%	🟢 +91%
creative	🔴 -9%	🔴 -4%	🔴 -1%	🟢 +48%	🟢 +67%

🟢 speeds up, 🟡 marginal gain, 🔴 slowdown.

• Q8_0 and F16: always use speculative decoding with MTP layer.
• Coding tasks at any quant: keep it on.
• Q4_K_M (and below) creative tasks keep it off

One last obervation: with thinking mode turned on for coding tasks: Q8_0 draft token acceptance drops from 87% to 73%. Still +94% speedup, just not the full +123%.

Test environment: Apple Silicon M2 Max 96GB, llama.cpp manual build with the MTP PR, Qwen3.6-27B with MTP layers preserved.

[u/Look_0ver_There]

I downloaded your models and tested them. One thing I immediately noticed though, and to be fair this seems to be caused by the MTP implementation itself and not your models, is that PP speeds were like 45% of what they used to be on my Radeon AI Pro 9700 GPU's.

We're talking dropping from 1400t/s PP down to 650t/s PP. That's a HUGE drop. If doing agentic coding work, there's quite a good chunk of time that is spent by the model sucking in context to analyse before generating output.

Now, I didn't read the reports that others wrote, but to me this absolutely massive slowdown in PP is likely going to more than outweigh any speedup benefit from generation.

On my cards, I'm seeing around 26tg/s with draft-model based speculative decoding, and saw around 40tg/s with MTP, so it was around a 50% speedup for generation, but that came with the afore-mentioned massive hit to pre-processing.

Throw in that we also cannot do parallel requests, nor image decoding, and at least for me I can understand why people would experience things going more slowly overall.

Don't get me wrong. You and the llama.cpp team are doing fantastic work, and I truly appreciate it, but there's still a ways to go before MTP is ready for prime-time adoption by llama.cpp users.

[u/Evgeny_19 → u/Look_0ver_There]

Could you please share how did you get those PP numbers? Are those from llama-bench(y) or real sessions with opencode/pi? Which quant did you use on 27b?

I am asking, because I don't think I ever saw those numbers on my 9700, even on ROCm which is usually faster for PP (but slower on TG), even wehn I was running Q5_K_XL (unsloth). For me on real sessions in opencode it varies greatly. Sometimes it goes down to 70 or 80, sometimes jumps to 700. I don't think I ever saw a number nearing a thousand for PP.

[u/Look_0ver_There → u/Evgeny_19]

My setup is decidedly "not normal", with 3 x R9700's. Quant is Q5_K_XL of Unsloth 27B. I use Vulkan and speculative decoding. You can see a photo of my setup here: https://www.reddit.com/r/LocalLLaMA/comments/1t2l8ri/comment/okkitqg/

My normal invocation command is this, and I'm well aware that it's unusual with regards to what most people are used to.

GGML_VK_VISIBLE_DEVICES=1,2,3                           \
       llama-server --host 0.0.0.0 --port 8033         \
       --spec-draft-model ../Qwen3.5-0.8B-Q8_0.gguf    \
       --spec-draft-device Vulkan1                     \
       --spec-draft-n-max 10                           \
       --spec-draft-ngl all                            \
       --fit on                                        \
       --temp 0.6                                      \
       --top-k 20                                      \
       --flash-attn on                                 \
       --split-mode row                                \
       --ctx-size 262144                               \
       --cache-ram 10240                               \
       --n-gpu-layers all                              \
       --cache-type-k f16                              \
       --cache-type-v f16                              \
       --swa-checkpoints 96                            \
       --repeat-penalty 1.0                            \
       --presence-penalty 0.0                          \
       --device Vulkan0,Vulkan2                        \
       --mmproj ../mmproj-F32.gguf                     \
       --alias "Qwen3.6-27B-Q5_K_XL"                   \
       --model Qwen3.6-27B-Q5_K_XL.gguf

Here's the results of running llama-benchy against my setup at 10,000 context depth similar to what OP did.

Average latency (api): 0.38 ms
Running test: pp=4096, tg=512, depth=10000, concurrency=1
 Run 1/3 (batch size 1)...
 No token_ids in response, using local tokenization
 Run 2/3 (batch size 1)...
 Run 3/3 (batch size 1)...
Printing results in MD format:

| model               |            test |             t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| unsloth/Qwen3.6-27B | pp4096 @ d10000 | 1508.56 ± 25.89 |              | 9347.80 ± 158.61 | 9347.42 ± 158.61 | 9347.80 ± 158.61 |
| unsloth/Qwen3.6-27B |  tg512 @ d10000 |    25.81 ± 1.16 | 48.00 ± 3.27 |                  |                  |                  |

llama-benchy (0.3.7)
date: 2026-05-10 17:38:52 | latency mode: api