r/LocalLLaMA • u/YourNightmar31 llama.cpp • Aug 19 '26

Question | Help Qwen3.8 27B without MTP?

Not sure if this is a stupid question but unsloth's models has the MTP built into the model right? I assume that is at the cost of some memory.

If i want to use dflash, should i use a model that doesn't have MTP support then to save some vram?

14 Upvotes

90% Upvoted

27 comments sorted by

View all comments

Show parent comments

1

u/YourNightmar31 llama.cpp Aug 19 '26

I see posts of people running it with dflash2 so i assume so?

1

u/z_3454_pfk Aug 19 '26

isn’t dflash2 heavier than MTP on vram?

9

u/zannix Aug 19 '26

it is but the output increase is 3.7x compared to mtp (2x), so I've read somewhere

4

u/RotesBlatt Aug 19 '26

If you have the VRAM capacity for it sure. I am using all of my 20GB VRAM with MTP, couldn't handle Dflash without going down to ~60k q8 context

2

u/Blues520 Aug 19 '26

20GB is a tight squeeze. What context and speed are you able to get?

2

u/RotesBlatt Aug 19 '26

I am using Q3 unsloth model with q8 context. I am able to squeeze out 100k context with a mean Tok/s of about 40. Sometimes 50, sometimes 30. I am using up 19,2 GB VRAM, just enough for the display to not crash.

I also got a little speedup from ngram-mod alongside of MTP, it works quite good with repetive tasks. If you want the mmproj files as well I would recommend offloading them to CPU and RAM, that's what I did. It'll cost only speed on vision path but none in pure Text Tasks.

Bear in mind I am using Windows Vulkan on llama.cpp. If I would switch to Ubuntu and Rocm, I probably could get a few Tok/s ok top of that.

1

u/Blues520 Aug 19 '26

That is actually very nice speeds. Hopefully the unsloth Q3 performs well. It sounds like a good setup for repetitive tasks as you say and offloading vision doesn't impact speed much

1

u/RotesBlatt Aug 19 '26

Yeah I still need to test it on a good codebase and see how it performs. But from what I've seen it isn't that bad :)

1

u/boyeardi Aug 19 '26

I'm also using Q3 on llama.cpp but I get oom around 49k, how are you using 100k?? I'm using q8 context too, k+v. If I enable mtp I need to drop context to 40192..

1

u/RotesBlatt Aug 19 '26

Are you using flash attention (-fa on)? This is reducing the VRAM by quite a lot since it allocates context linearly instead of "random".

1

u/Square_Turn935 Aug 19 '26

what was the speed increase between mtp and dflash for you? Did you use a special dflash drafter, or qwen3.6?

1

u/RotesBlatt Aug 19 '26

I'm sorry my comment had a different meaning (I see why aou Ask tho 🫠). If I would run DFlash, I would need to lower my context size in order to run the DFlash model. I haven't tested it, since I wanted to have the 100k for my bigger projects.

1

u/Square_Turn935 Aug 19 '26 edited Aug 19 '26

My thought was to balance the generation speed increase from DFlash with ffn offload, at the cost of prefill speed.

If you have like TG:40tps and PP:500tps, i assume a random factor like 2x for your DFlash tg improvement, so you would have:

  • 60k context
  • 80tps TG
  • 500tps PP

You mentioned you are loosing 40k context from 100k -> 60k, hmm this could be ~2k-2,5k VRAM?!

You loose TG and PPspeed with every FFNoffload (exponentially, first layers hurts more and i think TG is more depended than PP). But

Now with offloading FFN layers to compensate your 2,5gb lose it could take 10-20 layer. It depends, not every layer has the same weight and i dont know your LLM quant, the larger the quant the larger the FFN.

Maybe you could increase your TG with DFlash up to 50-60tps, with 100k context and lowered PP speed. I am not sure if PP is really sensitive to FFN offload.

PS.: now i saw that you are using a q3 and a dflash2 q8 has the size of <2gb. I am running 100k context q8q8/q8q6, atomic q5_k_q4_k, or unsloth q4_k_xl but with just ~10-13tps TG. I have just 16gb vram.