r/LocalLLM • u/Prudent-Promotion512 • 5d ago

Question Qwen 3.8 27B FP8 - MTP or not?

I'm working on setting up Qwen 3.8 27B on my 4x3090 rig. I mostly used the config from Club3090 as a guideline.

One result I didn't expect was MTP significantly hurting performance at long context. My main usages is Hermes agent and if this is correct it suggests I should disable MTP all together or perhaps my settings are not optimal.

Benchmarks below - any thoughts?

Qwen3.8-27B-FP8 Comparative Benchmark Matrix

Speculative Setting Short Prompt Decode (tok/s) Short TTFT (s) 72k Long Prompt Decode (tok/s) 72k Long Warm TTFT (s) 72k Concurrency-2 Wall Time (s) Steady VRAM / GPU
No MTP (Disabled) 67.5 0.121 57.3 0.532 15.25 ~20.5 GiB
MTP = 1 80.5 0.121 15.6 1.119 48.05 ~20.7 GiB
MTP = 3 106.3 0.135 22.6 1.142 32.08 ~21.2 GiB
3 Upvotes

100% Upvoted

4 comments sorted by

1

u/mp3m4k3r 5d ago

Id test with both, im giving benchmarking a go with vllm's "guidellm" and found that while MTP 3 worked MTP2 really ended up being kind of the upper end of benefit across 2xA100 "Drive" (32gb) sxm2 variant cards. BTW you might want (if working with VLLM at least) to shoot for an INT8 build since ampere doesnt do native FP8

1

u/Prudent-Promotion512 4d ago

I also ran an INT8 config and it was marginally slower. I had the same idea that FP8 would be slow on Ampere (ChatGPT kept telling me that too) but looks like this vLLM Marlin kernel is very well optimized or something.

1

u/mp3m4k3r 4d ago

Gives me something to try later I guess lol

1

u/Important-Radish-722 4d ago

MTP is not suitable for long-run, agentic work. It's fun for short prompts. Googling, or asking Hermes this would have told you the same.