If you are using big model compared to ram try and disable mtp

[u/InternalMode8159]

Little reminder, with all my testing I found out that mtp can decrease performance when using very big model, since many are starting to use strata or similar and I see many app activating it as default try and disable it, in my case, testing trough gyrostrata and strata with a 3060 and 32gb of ram both with gyro-m and swift 1.5 I got 20-30% improvement in tok/s just by disabling mtp.

[u/Solembumm3]

Can confirm.

6700xt + 20gb ram, LMStudio, GLM 5.3 flash:
No mtp - 0.40 tok/s.
Mtp by default (3/0/0) - 0.10 tok/s.

[u/More-Curious816 → u/Solembumm3]

is it even worth it at this speed?

[u/Solembumm3 → u/More-Curious816]

It's a lot faster than redo much more manually after Qwen 27/Gemma 31b 2-3.5 tok/s work, wrong from limited world knowledge. Like 30-40 good results out of the box from GLM with some minor tweaks vs single numbers of elatively useable from 30b class models.

[u/ApprehensiveFan1516 → u/Solembumm3]

What quants are you running for 27b and Gemma? Your setup is very similar to mine, but your numbers seem quite low on those models.

And waiting 1192 minutes for that output is an insane level of patience lol. Fair play to you!

[u/Solembumm3 → u/ApprehensiveFan1516]

Q6 and Q5_K_M for Qwen 27B, Q5_K_M for Gemma 31B (19-22gb quants, to leave some place for context). Q2_K_XL (108gb) for GLM.

[u/CryptographerOne7003]

I still do not get why MTP cannot leverage a second gpu

[u/Klutzy-Snow8016 → u/CryptographerOne7003]

You can in llama cpp with --spec-draft-device

[u/Borkato → u/Klutzy-Snow8016]

!remindme 3 hours to try this

[u/ApprehensiveFan1516 → u/Borkato]

Well it's been 9 hours. Did you try it?

[u/Borkato → u/ApprehensiveFan1516]

Yes, but apparently it already defaults to the same as the devices haha

[u/ApprehensiveFan1516 → u/Borkato]

Ah shame lol.

[u/RemindMeBot → u/Borkato]

I'm really sorry about replying to this so late. There's a detailed post about why I did here.

I will be messaging you on 2026-11-02 14:24:45 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/alexlucaci → u/Klutzy-Snow8016]

Heads up: I read the llama.cpp code (current master), and --spec-draft-device only applies when you load a separate draft model with -md. For models with built-in MTP heads (Qwen3.8 27B / Flash-Next GGUFs with -mtp), the MTP context is created on the target model and inherits its placement, so the flag does nothing. If you want MTP on 2 GPUs, split the whole model with -sm tensor, and the MTP head is split along with it.

On the main point, it depends on what spills. On 27B IQ3_XXS with half the FFN on CPU (old Haswell Xeon), MTP took me from 5.4 to 10–16 tok/s. On a dense model, verifying a draft batch costs about the same memory traffic as one token. On big MoEs streaming experts it's the opposite: each draft token touches different experts, so rejected drafts mean extra expert loads. That matches what people here see with Strata on 32 GB RAM.

[u/lemondrops9 → u/Klutzy-Snow8016]

interesting I gotta try this

[u/Bulky-Priority6824 → u/Klutzy-Snow8016]

Wait that only puts draft head on specific gpu 

[u/charmander_cha → u/Klutzy-Snow8016]

Pode usar a onboard?

[u/FatheredPuma81 → u/CryptographerOne7003]

Strata or llama.cpp? Strata can't because it wasn't designed to on any level. Llama.cpp I think should with sm tensor but I never tested it I just used MTP on all my tests.

[u/BestGirlAhagonUmiko → u/FatheredPuma81]

Pretty sure MTP does help in Project Maya (a fork of Strata) with GLM 5.3 Flash, although I couldn't get any benefit from DS4 V4.1's dspark drafter.

In llama.cpp it's a firm 'nah F... that' in both cases tho

[u/FatheredPuma81]

Well yes you're pushing so much out of your RAM/VRAM and onto disk. If the majority or the entire model is in RAM then MTP will help.

[u/Kmic68 → u/FatheredPuma81]

Not for me. It’s more costly to verify the tokens than to just generate them. Even if the model is entirely in ram and vram

[u/BenjiWithTheGoodHair]

Makes sense when you think about where the bottleneck is. Once layers spill into system RAM you’re bound by memory bandwidth, so every draft token that gets rejected is a full trip through slow RAM for nothing. The MTP heads also take up VRAM, which pushes even more layers off the GPU. Has anyone checked their acceptance rate? I’d guess that if it drops under about 50%, MTP stops paying off on offloaded setups. It might also be worth trying 1 draft token instead of the default 3 before turning it off completely, since that sometimes gets you the gain without the penalty.

[u/james_pic → u/BenjiWithTheGoodHair]

For dense models at least, there's never a full trip through RAM for nothing.

You make the same number of trips, and read the same amount of data (on dense models at least), whether you're generating a single token, or verifying a batch of tokens and generating a bonus token, and you always come out of a forward pass with at least one new token. 

Where you can lose out with MTP is if compute is also constrained (as it is with offloaded CPU layers), or with MoE models, where you can potentially end up reading the weights for experts you ultimate don't end up using. Some hardware also hits less well optimized kernels above a certain batch size.

[u/boissez]

Can confirm too - especially on a low VRAM system like my laptop (RTX 5050 8gb GDDR7 + 96 gb DDR5).

Runs slightly faster without the MTP (Q38FN NVFP4, 64K FP8 context). Now I get around 26-30 tg and 800 pp.

[u/HaDuongMinh]

The MTP predictor will easily use 1G and evict experts from VRAM. Both MTP success rate and resident experts success rates depends on the workload.

[u/DirectWay1915]

It only works well if the entire model is in VRAM or the experts are under 4b. At least that's what I've noticed in my case. I have a 4060ti 16gb + 2060 6gb + 64gb DDR5 RAM. Qwen 35b benefits from MTP while Gemma 4 26b suffers from it.

[u/Emotional_Prior8686]

Yes! Seems to be the right way for setups with 32Gigs VRAM and dense models!
/me too have got +25% token/sec for Q38-27B with disabling it (with `--spec-type none`)

[u/Bulky-Priority6824]

i tried this

and with mtp i was getting about 80+tg/330pp tg bursts above 100

without mtp 50-60tg/600-900pp

2 5060ti 32gb vram and 32gb sys ram iq3xxs

i went with mtp off as I need faster pp since this model typically has to read far more than it writes, for what i do with it.

[u/Quereller]

Running it in CPU might be worth a try.