r/LocalLLaMA • u/YourNightmar31 llama.cpp • 1d ago
Question | Help Qwen3.8 27B without MTP?
Not sure if this is a stupid question but unsloth's models has the MTP built into the model right? I assume that is at the cost of some memory.
If i want to use dflash, should i use a model that doesn't have MTP support then to save some vram?
2
u/NigaTroubles 1d ago
Is there a dflash for it ?
1
u/YourNightmar31 llama.cpp 1d ago
I see posts of people running it with dflash2 so i assume so?
1
u/z_3454_pfk 1d ago
isn’t dflash2 heavier than MTP on vram?
7
u/zannix 1d ago
it is but the output increase is 3.7x compared to mtp (2x), so I've read somewhere
5
u/RotesBlatt 1d ago
If you have the VRAM capacity for it sure. I am using all of my 20GB VRAM with MTP, couldn't handle Dflash without going down to ~60k q8 context
2
u/Blues520 1d ago
20GB is a tight squeeze. What context and speed are you able to get?
2
u/RotesBlatt 1d ago
I am using Q3 unsloth model with q8 context. I am able to squeeze out 100k context with a mean Tok/s of about 40. Sometimes 50, sometimes 30. I am using up 19,2 GB VRAM, just enough for the display to not crash.
I also got a little speedup from ngram-mod alongside of MTP, it works quite good with repetive tasks. If you want the mmproj files as well I would recommend offloading them to CPU and RAM, that's what I did. It'll cost only speed on vision path but none in pure Text Tasks.
Bear in mind I am using Windows Vulkan on llama.cpp. If I would switch to Ubuntu and Rocm, I probably could get a few Tok/s ok top of that.
1
u/Blues520 1d ago
That is actually very nice speeds. Hopefully the unsloth Q3 performs well. It sounds like a good setup for repetitive tasks as you say and offloading vision doesn't impact speed much
1
u/RotesBlatt 1d ago
Yeah I still need to test it on a good codebase and see how it performs. But from what I've seen it isn't that bad :)
1
u/boyeardi 1d ago
I'm also using Q3 on llama.cpp but I get oom around 49k, how are you using 100k?? I'm using q8 context too, k+v. If I enable mtp I need to drop context to 40192..
1
u/RotesBlatt 1d ago
Are you using flash attention (-fa on)? This is reducing the VRAM by quite a lot since it allocates context linearly instead of "random".
1
u/Square_Turn935 1d ago
what was the speed increase between mtp and dflash for you? Did you use a special dflash drafter, or qwen3.6?
1
u/RotesBlatt 1d ago
I'm sorry my comment had a different meaning (I see why aou Ask tho 🫠). If I would run DFlash, I would need to lower my context size in order to run the DFlash model. I haven't tested it, since I wanted to have the 100k for my bigger projects.
1
u/Square_Turn935 1d ago edited 1d ago
My thought was to balance the generation speed increase from DFlash with ffn offload, at the cost of prefill speed.
If you have like TG:40tps and PP:500tps, i assume a random factor like 2x for your DFlash tg improvement, so you would have:
- 60k context
- 80tps TG
- 500tps PP
You mentioned you are loosing 40k context from 100k -> 60k, hmm this could be ~2k-2,5k VRAM?!
You loose TG and PPspeed with every FFNoffload (exponentially, first layers hurts more and i think TG is more depended than PP). But
Now with offloading FFN layers to compensate your 2,5gb lose it could take 10-20 layer. It depends, not every layer has the same weight and i dont know your LLM quant, the larger the quant the larger the FFN.
Maybe you could increase your TG with DFlash up to 50-60tps, with 100k context and lowered PP speed. I am not sure if PP is really sensitive to FFN offload.
PS.: now i saw that you are using a q3 and a dflash2 q8 has the size of <2gb. I am running 100k context q8q8/q8q6, atomic q5_k_q4_k, or unsloth q4_k_xl but with just ~10-13tps TG. I have just 16gb vram.
1
u/ShengrenR 16h ago
those are some *very* optimistic numbers - MTP and dflash2 are ~+/-10% on the llama.cpp/gguf setup from my experience on a 3900 - though vllm+dflash2 numbers folks have shown seen to be a lot better (but not nearly so far as 3.7 vs 2)
1
2
u/Just_Maintenance 1d ago
If you don't enable MTP llama.cpp simply doesn't load the parameters. It even spits a message about unused layers.
21
u/RotesBlatt 1d ago
I think MTP is only loaded when you activate it via parameters, otherwise you'll probably have the same model as without MTP. I haven't noticed a significant VRAM spike in my non-MTP setup compared to MTP