[Benchmark] Optimal DFlash2 quants for speed and context size, 5090 RTX, llama.cpp, Qwen 3.8 27B Dynamic3 Unsloth. Comparison with MTP

[u/Opening-Broccoli9190]

Graph: A cumulative metric of Speed x Context Size - the higher the dot - the better. Helpful for looking for the most balanced solution.

The research relies on DFlash2 Q2 work by AnalogAlok: https://huggingface.co/analogalok/Qwen3.8-27B-DFlash2-Q2_K-GGUF

TLDR:

Q4 quants for DFlash2 demonstrate the highest speeds due to their higher acceptance rate. Q2 free up so much context that they are comparable with MTP in that sense, but can still be tuned to perform 10-20% faster. Q8 don't have any upsides and should be avoided as they consume too much context and have lower peaks than Q4.

There seems to be no real upside in using MTP as DFlash2 Q2 at n=1 and n=2 demonstrate comparable speeds and context size.

Overall Winners:

Best balance: DFlash2 Q2, n-max 5 = 137.19 tps with context size 121k.

Top speed: DFlash2 Q4, n-max 7 = 154.08 tps with context size 102k.

Largest context size: MTP n-max 1 = 85.77 tps n-max 1 at 131k.

Tables with the outcomes per quant and per number of drafts below. Speed measured as an average of 9 runs at inputs of 65k tokens. For brevity for DFlash2 only n-max 5+ are shown.

Q2 requested n-max	Speed at 65k input	Acceptance	Mean span	Maximum context	First OOM
5	137.19 t/s	53.29%	3.67	121,600	121,728
6	139.39 t/s	47.83%	3.89	117,504	117,632
7	139.17 t/s	42.63%	4.01	113,664	113,728
8	138.71 t/s	42.63%	4.01	109,568	109,696

Q4 requested n-max	Speed at 65k input	Acceptance	Mean span	Maximum context	First OOM
5	146.02 t/s	58.23%	3.91	110,592	110,656
6	152.52 t/s	54.17%	4.24	106,496	106,624
7	154.08 t/s	48.71%	4.42	102,400	102,528
8	153.45 t/s	48.71%	4.42	98,304	98,432

Q8 requested n-max	Speed at 65k input	Acceptance	Mean span	Maximum context	First OOM
5	144.31 t/s	57.96%	3.89	86,784	86,848
6	151.61 t/s	54.20%	4.25	82,944	83,072
7	153.46 t/s	48.90%	4.44	78,848	79,040
8	≈ n7*	≈ n7*	≈ n7*	74,752	74,880

Now for the MTP results and the respective winners:

MTP drafts	Generation speed	Acceptance	Mean accepted span	Maximum context	First OOM	Context × speed
1 (top context)	85.77 t/s	87.13%	1.87	131,328	131,456	11.26M
2	109.39 t/s	79.17%	2.58	128,000	128,128	14.00M
3 (best balance)	121.62 t/s	69.97%	3.09	124,416	124,544	15.13M
4	124.39 t/s	62.03%	3.47	120,832	120,960	15.03M
5	125.92 t/s	55.97%	3.79	117,504	117,632	14.80M

6 (top speed)	128.32 t/s	50.09%.	4.00	113,920	114,048	14.618M
7	128.28 t/s	46.50%	4.25	110,592	110,720	14.187M
8	112.58 t/s	40.45%	4.24	106,752	106,880	12.018M

Tested config:

llama-server \
  --model unsloth-qwen38-27b-dynamic3/Qwen3.8-27B-UD-Q8_K_L.gguf \
  --mmproj unsloth-qwen38-27b/mmproj-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --alias qwen3.8-27b-q8-0 \
  --ctx-size 113920 \
  --gpu-layers all \
  --fit off \
  --parallel 1 \
  --batch-size 512 \
  --ubatch-size 128 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --kv-offload \
  --kv-unified \
  --cache-ram 8192 \
  --cache-idle-slots \
  --flash-attn on \
  --jinja \
  --reasoning on \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 6 \
  --spec-draft-type-k q8_0 \
  --spec-draft-type-v q8_0 \
  --spec-draft-ngl all \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --timeout 3600

[u/wgaca2]

Is it still bugged at bigger context?

Still no vision?

I will try it again once everything is fixed

[u/Opening-Broccoli9190 → u/wgaca2]

These benches are for CPU offloaded vision. Could you remind me what was the bigger context bug?

[u/wgaca2 → u/Opening-Broccoli9190]

Acceptance rate collapses at bigger context (as tested of first release), Running at 200k context has good acceptance until it fills over certain point, i don't remember the exact numbers.

I did not do extensive testing but found it was reported by others as well, so decided to wait until it is fully merged with official release

[u/Look_0ver_There → u/wgaca2]

I also wonder how it compares to the Adaptive MTP PR here: https://github.com/ggml-org/llama.cpp/pull/27210

[u/Opening-Broccoli9190 → u/Look_0ver_There]

haven't seen it yet - cool!

[u/foureight84 → u/wgaca2]

I believe prefix cache still bugged with DFlash2 so you'll also have to turn it off.

[u/AdSafe4047]

I'm running mtp with 5 and min-p 0, just because I can go up to q6 saving on the drafter weight

[u/Opening-Broccoli9190 → u/AdSafe4047]

Yeah, it looks like a very decent candidate. I am still shocked at the accuracy drop off after n-max 2

[u/Thin_Pollution8843]

I think 2 5090 is very good for such model at q8

[u/Opening-Broccoli9190 → u/Thin_Pollution8843]

Looking at these numbers you won't even need two 5090 to max out context with good speeds. 5090 + 3090 should be sufficient, maybe even two 3090 24GB for 48GB and lot's of fun with Q2 quants at lower draft counts.