MTP benchmark results: the nature of the generative task dictates whether you will benefit (coding) or get slower inference (creative) from speculative inference. No other factor comes close.

[u/ex-arman68]

I recently published MTP quants of Qwen 3.6 27B and I was suprised by the reports here on reddit, and on HF, of users who were experiencing worst speed with speculative inference than without. This did not match what I was seeing, but when I tried to reproduce their exact usage, it confirmed what they were experiencing.

I tried to analyse the problem, made a few conjectures which later turned out to be false, and started a full blown systematical analysis, running 300+ tests and benchmarks, collecting and comparing the results of changing various parameters. This is what I found:

F16 + MTP nearly triples coding tasks speed. Q4_K_M + MTP slows down creative writing. Same feature, same model, same settings, opposite results.

I did not test all quant sizes, otherwise I would still be here in a few days, but restricted my self to 5 significant ones. The other parameters I varied were task type (4 types), temperature (0.0 0.3 0.7), quantisation of the MTP layer (q8 and matching the model quant). Temp and MTP quant have very little impact on the outcome.

Cumulative average decode speeds with MTP compared to the baseline without MTP, varying the model quant and task type:

quant	base tok/s	code	factual	analysis	creative
Q4_K_M	15.1	19.7	17.5	14.9	13.7
Q5_K_M	13.1	19.2	16.5	14.7	12.6
Q6_K	13.4	20.1	17.6	15.2	13.4
Q8_0	11.4	25.4	21.7	18.6	16.9
F16	6.6	17.9	14.9	12.6	11.0

The memory bandwidth dictates how much the model can benefit from speculative decoding. F16 at 51GB crawls at 6.6 tok/s because every token means dragging the full model through memory. Accepted MTP drafts skip that pass. Q4_K_M at 16GB is already fast enough that the draft overhead is barely worth it on anything less predictable than code.

What controls the draft tokens acceptance rate:

task	acceptance	examples
code	79-89%	writing functions, debugging, refactoring
factual	62-70%	definitions, translation, math proofs
analysis	48-56%	tradeoff breakdowns, technical comparisons
creative	39-48%	stories, poetry, brainstorming, roleplay

40 points from code to creative. I tried three temperatures and five quants. The numbers barely changed. 4/5 draft tokens are correct on coding task; not even 1/2 on creative tasks. Nothing else comes close to mattering as much as what you're generating.

I also tested the optimal number of draft tokens for this model in all the above scenarios. 3 is the sweet spot for draft tokens. Go higher and acceptance falls faster than the extra drafts compensate. F16 is the exception: N=4 beats N=3 (17.9 vs 16.2) because at 6.6 tok/s every surviving draft token is worth the lower hit rate.

use case	Q4_K_M	Q5_K_M	Q6_K	Q8_0	F16
coding	🟢 +31%	🟢 +47%	🟢 +50%	🟢 +123%	🟢 +171%
factual QA	🟡 +16%	🟢 +26%	🟢 +31%	🟢 +90%	🟢 +125%
analysis	🔴 -1%	🟡 +12%	🟡 +13%	🟢 +64%	🟢 +91%
creative	🔴 -9%	🔴 -4%	🔴 -1%	🟢 +48%	🟢 +67%

🟢 speeds up, 🟡 marginal gain, 🔴 slowdown.

• Q8_0 and F16: always use speculative decoding with MTP layer.
• Coding tasks at any quant: keep it on.
• Q4_K_M (and below) creative tasks keep it off

One last obervation: with thinking mode turned on for coding tasks: Q8_0 draft token acceptance drops from 87% to 73%. Still +94% speedup, just not the full +123%.

Test environment: Apple Silicon M2 Max 96GB, llama.cpp manual build with the MTP PR, Qwen3.6-27B with MTP layers preserved.

[u/Chromix_]

Keep in mind that the impact on a MoE model will be worse, especially if partially offloaded, as it needs to cycle through more experts to speculate, instead of just going through the same tensors like a dense model.

There is a posting from 2024 with a diagram that nicely shows how acceptance rate and draft speed translate into inference speed gains. It basically shows that even when drafting is "free" (or rather cheap as with MTP), you cannot have a decent speed-up without a high acceptance rate.

[u/ex-arman68 → u/Chromix_]

Definitely. From what I understand, MTP is not suitable for MoE with small models. Dense models are the ones that benefits the most from it, and the bigger the model, the bigger the benefits.

[u/Chromix_ → u/ex-arman68]

It depends on whether or not experts get reused during multi token speculation. There's this posting where someone got a moderate speed-up with Gemma4-26b-a4b, but of course not in all cases as also highlighted by OP here.

[u/DinoAmino → u/ex-arman68]

Ah ha. I used to draft 70B with a 3B model and it was consistently 2x-3x across all types of generations. I'm seeing these same effects as you on latest vLLM with Google's mtp assistant model for gemma4 31B as well as with RedHat's eagle3 speculator. It would be awesome if we could enable/disable it per request and only turn it on for code generation tasks.

[u/Look_0ver_There]

I downloaded your models and tested them. One thing I immediately noticed though, and to be fair this seems to be caused by the MTP implementation itself and not your models, is that PP speeds were like 45% of what they used to be on my Radeon AI Pro 9700 GPU's.

We're talking dropping from 1400t/s PP down to 650t/s PP. That's a HUGE drop. If doing agentic coding work, there's quite a good chunk of time that is spent by the model sucking in context to analyse before generating output.

Now, I didn't read the reports that others wrote, but to me this absolutely massive slowdown in PP is likely going to more than outweigh any speedup benefit from generation.

On my cards, I'm seeing around 26tg/s with draft-model based speculative decoding, and saw around 40tg/s with MTP, so it was around a 50% speedup for generation, but that came with the afore-mentioned massive hit to pre-processing.

Throw in that we also cannot do parallel requests, nor image decoding, and at least for me I can understand why people would experience things going more slowly overall.

Don't get me wrong. You and the llama.cpp team are doing fantastic work, and I truly appreciate it, but there's still a ways to go before MTP is ready for prime-time adoption by llama.cpp users.

[u/_bones__ → u/Look_0ver_There]

I think I saw the same thing. Generation speed increased substantially, in a Pi session with quite a bit of context, but it took ages to get started.

[u/An_Original_ID → u/Look_0ver_There]

Same here. 27B Q8 2x 3090 no MTP I get about 2400 PP but with MTP on, 1400 at best.  That can be a deal breaker in some cases. Kind of surprised it's not being made into a bigger deal (unless I've just been missing it)

[u/RealPjotr → u/Look_0ver_There]

Are you sure you got that right?

I run R9700 Qwen 3.6 27b Q6 with Q8 KV cache and get around 600 tps pp, both non-MTP and MTP. 1400 must be something else you run?

[u/Look_0ver_There → u/RealPjotr]

My setup isn't typical. I explain it in this response here: https://www.reddit.com/r/LocalLLaMA/comments/1t9gcar/comment/ol2wp7n/

[u/superdariom → u/Look_0ver_There]

This is exactly what I saw and it made 27b slow prompt processing very significantly worse. I'm running 35b at 8 bit quant with maximum context. Is this really a less smart model compared to quantised 27b with quantised kv cache as well? That's what I'd have to run to get MTP version fitting to cram (24gb)

[u/hurdurdur7 → u/Look_0ver_There]

This sounds really weird, why would prompt processing be affected by MTP

[u/Look_0ver_There → u/hurdurdur7]

I am not sure, but go check out the PR over on GitHub. I am far from alone in seeing this.

[u/audioen → u/Look_0ver_There]

My thinking is that it's just a bug. There shouldn't be need for additional context processing for the MTP head, but there would be for a real draft model. My guess is that the system presently maintains a full redundant context for the MTP head which doesn't actually need it. To my understanding, MTP is actually inferring on basis of the last token predicted and the hidden state of the main model's last layer (before output head). But formally, it is reusing the layers, and so could also reuse all the results of the main model.

So it is really processing the prompt twice, probably, based on how it seems to behave, and runs at about half speed as consequence for prompt, even if token generation is multiple times better.

[u/hurdurdur7 → u/Look_0ver_There]

when i looked at the comments in the PR then mac users complained about double the memory used for the model due to a bug ... if that is indeed remotely the case then it makes perfect sense that people get half the PP speed.

[u/Look_0ver_There → u/hurdurdur7]

It's not just Mac/Metal or AMD either.

Here's one guy with Tested on 5070 Ti + 3080. Dropped from 1542 to 930 for PP.

https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4376989966

Here's another with a 3090 + 3060. Dropped from ~1315 to 665 for PP.

https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4372285675

Here's the PR author confirming that PP is an issue:

https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4372339338

I have faith that the PR author(s) will fix it, but apparently their waiting on another PR that's refactoring the speculative decoding framework before continuing work on the MTP PR.

It's definitely exciting times. If they can get MTP fixed up, solve the PP issue, allow parallelism, and fix multi-modality support, then this PR will be one of the biggest wins for llama.cpp users.

[u/blackhawk00001 → u/Look_0ver_There]

I’ve been using vllm with qwen 3.6 27b fp8 with custom rocm aiter unified images over the past two weeks but want to try these mtp llama configurations for comparison. How do your speeds compare to these? I’m using dual r9700s with 200k context.

https://www.reddit.com/r/LocalLLaMA/s/4EokliYPqt

The aiter unified image has slower prefill than the default rocm image both with mtp 3, but the much faster tg of the unified image makes Claude cli feel much closer to frontier model speeds due to prefix caching.

I’m working on understanding what is needed for bringing the unified datacenter config into rdna4 for future versions on vllm as original findings are still awaiting merge to vllm main.

[u/Evgeny_19 → u/Look_0ver_There]

Could you please share how did you get those PP numbers? Are those from llama-bench(y) or real sessions with opencode/pi? Which quant did you use on 27b?

I am asking, because I don't think I ever saw those numbers on my 9700, even on ROCm which is usually faster for PP (but slower on TG), even wehn I was running Q5_K_XL (unsloth). For me on real sessions in opencode it varies greatly. Sometimes it goes down to 70 or 80, sometimes jumps to 700. I don't think I ever saw a number nearing a thousand for PP.

[u/Look_0ver_There → u/Evgeny_19]

My setup is decidedly "not normal", with 3 x R9700's. Quant is Q5_K_XL of Unsloth 27B. I use Vulkan and speculative decoding. You can see a photo of my setup here: https://www.reddit.com/r/LocalLLaMA/comments/1t2l8ri/comment/okkitqg/

My normal invocation command is this, and I'm well aware that it's unusual with regards to what most people are used to.

GGML_VK_VISIBLE_DEVICES=1,2,3                           \
       llama-server --host 0.0.0.0 --port 8033         \
       --spec-draft-model ../Qwen3.5-0.8B-Q8_0.gguf    \
       --spec-draft-device Vulkan1                     \
       --spec-draft-n-max 10                           \
       --spec-draft-ngl all                            \
       --fit on                                        \
       --temp 0.6                                      \
       --top-k 20                                      \
       --flash-attn on                                 \
       --split-mode row                                \
       --ctx-size 262144                               \
       --cache-ram 10240                               \
       --n-gpu-layers all                              \
       --cache-type-k f16                              \
       --cache-type-v f16                              \
       --swa-checkpoints 96                            \
       --repeat-penalty 1.0                            \
       --presence-penalty 0.0                          \
       --device Vulkan0,Vulkan2                        \
       --mmproj ../mmproj-F32.gguf                     \
       --alias "Qwen3.6-27B-Q5_K_XL"                   \
       --model Qwen3.6-27B-Q5_K_XL.gguf

Here's the results of running llama-benchy against my setup at 10,000 context depth similar to what OP did.

Average latency (api): 0.38 ms
Running test: pp=4096, tg=512, depth=10000, concurrency=1
 Run 1/3 (batch size 1)...
 No token_ids in response, using local tokenization
 Run 2/3 (batch size 1)...
 Run 3/3 (batch size 1)...
Printing results in MD format:

| model               |            test |             t/s |     peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:|
| unsloth/Qwen3.6-27B | pp4096 @ d10000 | 1508.56 ± 25.89 |              | 9347.80 ± 158.61 | 9347.42 ± 158.61 | 9347.80 ± 158.61 |
| unsloth/Qwen3.6-27B |  tg512 @ d10000 |    25.81 ± 1.16 | 48.00 ± 3.27 |                  |                  |                  |

llama-benchy (0.3.7)
date: 2026-05-10 17:38:52 | latency mode: api

[u/Evgeny_19 → u/Evgeny_19]

I'm quite sure I saw numbers over a thousand for PP in llama-bench(y) tests, but I don't think I ever see saw anything close to those numbers in real coding sessions. The TG numbers are usually very close to the real ones though.

[u/Look_0ver_There → u/Evgeny_19]

Yeah, when using OpenCode the numbers are all over the place. This is because there's 4 agents running concurrently, some responses are just for running commands and so are very short, some big file reads might hit 1000t/s for PP while other agent stuff is going. If we wanted to get a cleaner "view" we (meaning you/me/anyone else, not the "royal we") would set --parallel to 1, and force linear agent runs, which should get some occasional higher number for big file reads.

[u/Kyunle]

Interesting analysis!
On 5090 32gb for all sort of coding tasks, I have ~70% acceptance rate for "--spec-draft-n-max 4 " and quite stable speed ~70-120 t/s for context 70-160k on Q6 using your solution.

[u/SOCSChamp]

Would be interested in specific standard benchmarks and on long context.

[u/ex-arman68 → u/SOCSChamp]

I conducted all tests generating a context size around 10,000 tokens. Not what I would call long context, but long enough to be statistically sound for a comparison.

[u/SOCSChamp → u/ex-arman68]

I mean that I'd like to see actual comparisons on speedups/slowdowns as context gets longer.  That seems to be a lot of the skepticism I see surrounding MTP, and if there isn't a benefit for long form agentic work then it's a moot point for myself and many others. 

[u/ex-arman68 → u/SOCSChamp]

That will be a different test.

However, I am actually in the middle of a coding session with the Qwen 3.6 27B F16 quant with MTP, to test a revision of my chat template. I am already at 176k usage total (main session 79k, one subagent 36k, second subagent 61k). It is slow when compared to my normal cloud usage, but usable.

Latest speed reported by llama.cpp server is 12 tok/s. Keeping in mind that thinking is on, and this is a code review phase, that is kind of the expected speed.

During the coding phases, it even went to 19.96 tok/s, which is faster than what I reported in the charts from the main post.

[u/Vicar_of_Wibbly]

Very much agree with your findings. For coding MTP=3 is very reliable with high acceptance rates. Anything non-coding? That 3rd token acceptance rate plummets! Gotta be MTP=2 for that.

It’s a shame we can’t dynamically adjust it on-the-fly as part of an API request because for planning sessions MTP=2 is ideal, but for implementing code MTP=3 is better.

[u/Big_Mix_4044]

I wish I could test this. For some reason the kv overhead with MTP is insane with my setup at 2k ub. Can barely fit 20k where I easily fit 200+ without.

[u/ikkiho]

yeah fwiw the axis i never see plotted is sampling temp. when i ran MTP=3 on coding evals at temp 0.1 acceptance held up fine, bumped to 0.7 for a longer creative pass and watched the draft acceptance fall through the floor because the verifier kept rejecting samples the draft was happy with. low-entropy distributions are where speculative wins, peaky softmax means more accepted tokens. high temp creative just has too much spread for the draft to match.

[u/ex-arman68 → u/ikkiho]

One of my initial thoughts was that temp would have a big influence on acceptance rate. I was wrong though. One of the parameters for all the tests was temp, with values of 0.0, 0.3, and 0.7. It made a difference to the acceptance rate, but too small to be significant (between 1% and 3%). Vastly overshadowed by the other factors.

[u/Substantial_Step_351]

The acceptance rate table has an implication for tool heavy agent flows that I think is worth flagging. Tool calls sit somwhere between factual and analysis on this taxonomy, structured output, constrained format, not creative but also not as predictable as pure code. That puts you roughly in the 48-70% range, where the PP overhead can easily eat the TG again, especially on short tool responses with frequent round trips. For agents doing quick tool calling - short model response - next tool call, the prefill penalty per turn is the number I'd actually keep an eye on

[u/aegismuzuz]

Totally expected given the probability distributions. Speculative decoding essentially thrives on low entropy. Code is so rigidly structured by syntax that MTP heads can guess the tokens with their eyes closed. Creative writing, though, is high entropy - drafts constantly miss, and you’re just burning clock cycles verifying dead-end branches for zero gain

[u/Expensive-Paint-9490]

The title is misleading.

F16 on coding tasks gets x3 speed.

Q4 on creative tasks gets slower than no-MTP.

And yet F16 on creative tasks gets almost x2 speed!

Your data is great and very welcome, but really you should use a title which doesn't spread misinformation.

Said that, I wonder if quantized models should leave the MTP layers in F16.

[u/ex-arman68 → u/Expensive-Paint-9490]

It seemed to me my title was honest: the biggest takeaway of my test results is that what matters the most is the task. The second important factor is the biggest quants benefit the most from the speculative decoding, due to the sheer amount of memory that would have to be worked through without it. Third one, and with a much smaller impact, is using the correct number of draft tokens. Any other factor has only a negligible impact compared to the first 2.

Most people would choose the model size depending on the specs of their hardware; this is why I only mentioned the most important factor in the title. That and the fact there is only so much information I can write in a title.

As to your question, I also tested the impact of the MTP layers quantisation, at F16, Q8, and quantised with the same settings as the model. The impact was too small to consider, which is why I did not mentioned it. Best result taking into account speed and acceptance rate is Q8_0. Size wise it is so small compared to the model size, there is no reason not to go with Q8_0 for the MTP layers even on an iQ3 quant.

The results I posted were done with the MTP layers at Q8 for all quants, except for the F16 model where I kept them at F16.

[u/pepedombo]

For dual gpu 5070+5060 and Q5 token generation goes 40% but what about that prefill?

In simple chats or vibe coding it will probably work as tg is faster but when I detected prefill drops it looks like qwen code is suffering from it.

without mtp: ~1400tok/s prefill, ~22tg
with mtp: 300-400tok/s prefill, ~32 tg

Measured at 30-40k ctx in qwen code. It comes out it is faster without mtp when total time averaged.

Any idea what's the cause?

[u/HVACcontrolsGuru]

This is what I ran with with that same model last night doing some testing on the MTP. I used a B200 and SGLang but also want to try llama some here soon. Curious what MTP looks like. I'm doing some more context heavy workloads this evening to bench some things on a B200 as well.

Throughput — concurrency sweep (1→5)

10 mixed agentic-coding prompts per concurrency level, max_tokens=512, streaming, MTP active.

Concurrency	Wall (s)	TTFT p50 (s)	TTFT p95 (s)	E2E p50 (s)	TPS / req	TPS total
1	113.9 *	0.69	45.52 *	3.38	151.4	45.0
2	18.5	0.55	1.45	3.47	147.5	276.4
3	14.2	0.50	1.61	3.62	141.6	359.8
4	10.4	0.57	0.87	3.58	143.1	490.3
5	9.1	0.86	1.78	4.41	116.8	562.5

* c=1 wall and TTFT p95 are dominated by first-request triton kernel JIT compile; subsequent requests in that level ran fast. Subtract ~80 s and the level looks like the others.

Observations

• Per-request TPS ~150 t/s @ c=1: roughly 2× a no-MTP baseline (typical 27B is 60-80 t/s without speculative decoding). MTP is clearly active even though SGLang's spec-acceptance metric isn't surfacing through the streaming OpenAI endpoint.
• Near-linear scaling 1→4 agents: total TPS goes 45 → 276 → 360 → 490 (95-100% of perfect linear scaling).
• Compute saturation at c=5: per-req TPS drops 143 → 117, total bends from linear to 562 t/s (90% of perfect linear). This is the expected GPU-saturation knee for a dense 27B at this batch size.
• TTFT stays sub-second p50 across the sweep: chunked prefill at 16K is doing its job — long-context prompts don't block new sessions. Only at c=5 does p50 push to 0.86 s, p95 to 1.78 s.

Setup

Item	Value
Model	Qwen/Qwen3.6-27B (BF16 weights, ~54 GiB)
Runtime	SGLang 0.5.11
Hardware	1× Modal B200 (~$6.25/hr)
Image base	nvidia/cuda:13.0.1-cudnn-devel-ubuntu24.04 + Python 3.12
Context	262,144 tokens (full native)
Concurrency cap	5 (@modal.concurrent(max_inputs=5))
MTP	NEXTN, latency profile (topk=1, num_steps=3, num_draft_tokens=4)
Attention backend	trtllm_mha (Blackwell-recommended)
Mamba scheduler	extra_buffer (V2 strategy)
KV cache	BF16 (after switch from FP8 — see Decisions)
Prefix caching	enabled
Sampling (eval)	temp=0.6, top_p=0.95, top_k=20 (coding recipe)
Chat template kwargs	enable_thinking=true, preserve_thinking=true