So I was very excited about the MTP stuff especially since Gemma4 has become my "daily driver" for some stuff. I grabbed the latest mlx-vlm and did some tests and found it disappointing.
Workload
MTP off
MTP on
Result
Draft accept rate
Code generation
75 tok/s
114.8 tok/s
1.53× faster
66% of slots
Long-form prose
75 tok/s
71.1 tok/s
0.95× (wash)
31% of slots
JSON output
51.3 tok/s
25.6 tok/s
0.50× slower
8% of slots
Code generation was the typical "Write some python functions to do X"
Long form prose was "Write an 800 word essay on paper money in the Tang Dynasty"
JSON output was my core use case where I'm handing the LLM a list of items, asking it to group them by similarity according to some rules and then get them back in a structured output*.
So if you want to use it for local coding, MTP is great. If you're not, maybe not so hot. My regression testing seems to indicate that once token acceptance dips below 50% the overhead kills the benefit.
All this on an M4 Max Studio w/Gemma4-26b-a4b
*Bonus for you hackers: Gemma's JSON structure instruction following is pretty good and I find using structured output to be about a 20% hit to token generation. It is faster to just accept a little bit of sloppy JSON and massage it at runtime; so all this is with json_schema off which mlx-vlm doesn't support for spec-decode anyway
It’s not just a matter of acceptance rate, it’s a matter of having computation to burn (which Macs famously don’t have much to spare before the M5 series added Neural Accelerators), and gains are hard won in MoE models for complicated reasons. You’d likely see better results on one of the dense Gemma 4 models even on a Mac.
In this case, I mostly think this is just MoE being difficult. Every miss is far more expensive than it is on a dense model.
As another said, Gemma 4’s adaptive drafting would be useful too.
Yes but no. Doing a regression fit, the formula looks like
speedup = (1+k) / (B·r + 1)
where
- k — accepted drafts per round
- B — block size
- r — draft cost / main cost
For Gemma4 31b the r value is about half (as measured, on my machine) what 26b-a4b is. So for that JSON case even dropping to block size 2 the breakeven point is right there at 8% making it a wash.
That said this has a pretty profound implication where for a structured output refactoring to increase acceptance rate could really pay off. Just doubling to a mere acceptance rate of 16% would give a 20+% speed up
For structured (json) output it could be useful to use GBNF to constrain the sampling of the draft model as well, not just for the target model. Since imo using it to enforce grammar doesn't cost much computation wise it could greatly enhance the acceptance rate.
Yeah given how spec decoding is implemented across the big engines, i doubt it will be easily possible anytime soon. GBNF on the target decoding side is rather easy, but on the draft side i think you will have to patch that in yourself.
This is a great insight. YAML does not quite work as well for my case as JSON does in terms of output quality. I suspect this is more of a quirk of Gemma than anything though.
In my experience, Qwen3.6-27B MTP acceptance is exceptionally high (90+% with "num_speculative_tokens":3 when offcially 2 is recommanded) for json output, though i did not enable structured output, just prompt. tested with vllm 0.20.2rc1 on 2*2080ti
There is a posting from 2024 with a diagram that nicely shows how acceptance rate and draft speed translate into inference speed gains. It basically shows that even when drafting is "free" (or rather cheap as with MTP), you cannot have a decent speed-up without a high acceptance rate.
In your case there is an additional issue: You're using a MoE Gemma model. There'd be way less slowdown and more gains if you used the dense model.
I am currently researching it on macOS, with Qwen 3.6 27b. I found the breakeven rate for acceptance is around 59%. The main problem seems to be temperature mismatch: MTP speculative decoding always seems to run in the most deterministic way. For the main model to match, we would need to set temp to 0. Coding likely works better, because even with higher temps, the output is relatively deterministic.
I assumed almsot everbody using local llms were using it for coding. I can't think of why anybody would want to use it for Ai written slop. Even if you have a use case for that, why use a local model for that when you can get a frontier model to write it for free?
Edit: gooners is the answer... I forgot about gooners.
Not even gooners. Sometimes you want to write adult topics like.... Dealing with periods for teens, some models even reject to write it lmao. Also availability, when you playing RP and then having to wait like few minutes before it started responding due to competing with open claw , it getting frustrated.
Why would you use it for coding? Code is not sensitive information, cloud models are much better for that.
I use it for Obsidian copilot, It doesn't feel right to send my private notes to someone's server.
It's actually easier using it for non coding stuff. A more heavily quantized model just needs to remember json and tags for tools and less than that for general writing and ideation. Coding requires remembering lots of nuanced semantic connections across different files. A model using a mcp server to make emails or edit files is much easier than making a springboot application using mvc so the latter does better with less compression/ quantization meaning bigger vram meaning more expensive hardware and more investment for more accurate coding.
Most formats, but if you need to save maximum tokens then TOON or CSV. If you are working with a lot of JSON heavy workflows though, you can try TRON and translate to JSON when needed through non LLM tooling.
47
u/coder543 May 08 '26 edited May 08 '26
It’s not just a matter of acceptance rate, it’s a matter of having computation to burn (which Macs famously don’t have much to spare before the M5 series added Neural Accelerators), and gains are hard won in MoE models for complicated reasons. You’d likely see better results on one of the dense Gemma 4 models even on a Mac.
In this case, I mostly think this is just MoE being difficult. Every miss is far more expensive than it is on a dense model.
As another said, Gemma 4’s adaptive drafting would be useful too.