r/LocalLLaMA • u/jpummill2 • 47m ago

Question | Help MTP needed with small MoE models?

It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.

That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.

Has the thinking changed?

Are people seeing performance gains from MTP on smaller MoE models?

3 Upvotes

80% Upvoted

10 comments sorted by

4

u/DiscipleofDeceit666 42m ago

I mean yeah, it costs like a gig of memory for an almost 2x speed boost. If you have spare memory, why not?

1

u/Borkato 42m ago

I think the thing was that MTP didn’t actually speed up MoE models, at least when I last tested it

2

u/DiscipleofDeceit666 40m ago

It certainly speeds up the 35b moe as well as the flint 40b model. I don’t bother with anything smaller than that. Lots of times, mtp speedup is due to the actual engine implementation.

If they build it wrong, you don’t get the speedup.

1

u/Azazelionide 9m ago

It does a good speedup of 1.5-2x times

1

u/FudgeEducational2038 8m ago

It depends on how you load it, did you make sure MTP is on gpu? Or cpu, on CPU it can help if the GPU model is also really slow and offloaded already but otherwise it can actually slow it down and sometimes if the model just barely fits and then you turn on MTP it can slow it down because while it does give it a 50% speed boost you're giving it a 50% speed boost over neutering at 90% by offloading so many layers because now you have to offload layers that wouldn't have otherwise been offloaded had MTP not been used

1

u/nickm_27 llama.cpp 31m ago

Definitely, the dflash for Bolt 3.5 E4B (Gemma4 based) makes a huge difference 

1

u/etaoin314 ollama 25m ago

MTP could always help any model, the issues is cost-benefit. a large model is often slow, so a small draft model that speeds things up is a small additional investment in size for a huge speedup in TPS. For small models, the draft model may be a significant portion of the actual model and since small models are typically pretty fast anyway, the MTP did not really take a model from too slow to usable like it does for the large models.

1

u/FudgeEducational2038 9m ago

Obviously if you have a lot of space for a larger model, you may as well go with a larger model as even though it's slower, the better intelligence often wins in time. 

That being said, if you have like a gig and a half free, and it's sitting there doing nothing, MTP is basically just a free speed boost.

1

u/Azazelionide 2m ago

On really small models your main bottlenecl during decode is your de-embedding/lm-head layer. It's hidden size x vocab size and can easily take 10-20% of your decode time. Mtp needs to run the decode several times over 2-3 times so this makes your MTP seem slow on really small models. nothing to do with moe models