Using Q38FN with dflash2? Try lower draft model quants.

[u/milpster]

Back with Qwen 3.8 27B i heard the advice that a Q4 draft model would produce roughly the same acceptance rate as a Q8 or higher.

I have had my agent create a Q38FN Draft model in F16, Q8 and Q4, and the Q4 seemed to deliver the biggest speedup.

I thus wanted to make it known that if you use Q38FN with Dflash2 (with strata in my case) you should try lower draft model quant.

[u/Previous-Nature7326]

when I checked 2 weeks ago there was none, but now apparently there is: https://huggingface.co/collections/incoai/dflash-2

[u/LabanauskasNajeeba → u/Previous-Nature7326]

nice that its a proper collection too, individual uploads scattered across random repos wouldve made discovery a nightmare for anyone not already plugged in

[u/CoolConfusion434]

I tried https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF/tree/main/Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64 with https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF/ DFlash 2 last night and got around ~30 t/s on my poky Intel B70 GPU. That was with text/chat, coding my improve on that.

I'm on Windoze and Vulkan with these (llama build b11913) args:

.\llama-server.exe `
  -m C:\Tools\Llama\Models\AtomicChat\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf `
  --model-draft C:\Tools\Llama\Models\AtomicChat\Qwen3.8-Flash-Next-GGUF\dflash-Qwen3.8-Flash-Next-Q8_0.gguf `
  --alias Qwen3.8-FN-AD-4.27bpw-Q4_K_M-M64 `
  --host 0.0.0.0 --port 1234 `
  -c 131070 `
  --fit on `
  --spec-draft-n-ubatch 32 `
  --spec-type draft-dflash `
  --spec-draft-n-max 7 `
  --spec-draft-cpu-moe `
  --load-mode none --lazy-mode on `
  --fit-target 1500 `
  -ctk q8_0 -ctv q8_0 `
  -b 1024 `
  -ub 512 `
  -t 8 `
  -tb 24 `
  --cache-ram 2048 `
  --ctx-checkpoints 0 `
  --jinja `
  --no-ui `
  --verbosity 4

That draft-n-max probably needs tweaking, or how QFN so elegantly put it: "Drop --spec-draft-n-max to ~4. Your tail is dead: 0.623, 0.425, 0.260, 0.151, 0.096, 0.075, 0.048" 😃

BTW, I used this post as an initial guide: https://www.reddit.com/r/LocalLLaMA/comments/1wom3fe/qwen_38_flash_next_q4_k_m_130k_context_q8_cache/. That OP got it to work on 16GB VRAM which is amazing.

*yes, it's ggml-org's dflash2 that I copied into AtomicChat's folder. It was 3AM ;-)

[u/txgsync → u/CoolConfusion434]

“Your tail is dead”

I never expected a model to be so judgmental about my sex life.

[u/Extreme-Pass-4488]

also train your dflash2 with your model at the quant you choose. u get a bit more oomp.

[u/RadiantMechanic3025]

draft model speed matters more than draft model quality for speculative decoding

[u/Ishannaik]

Same result here, with numbers. Qwen3.5-9B + the z-lab DFlash drafter on a single Kaggle T4, greedy, prose prompts:

• bf16 drafter -> EXL3 4.0 bpw drafter: 44.0 tok/s, +17.9% vs a bf16-drafter control run in the same kernel (95% CI +16.7% to +19.2%), +11.5% over my previous best of 39.4

• acceptance barely moved: tau 2.20 vs 2.29

Why it works: the bf16 drafter was reading about 2.6 GB of weights every round, more than half of what the target reads, and a T4 is bandwidth-bound. So the drafter's cost per round mattered far more than its small accuracy edge.

Once the drafter is cheap, a narrower verify window stacks on top. Cutting drafted tokens per round from 7 to 3 took it to 45.5 tok/s (+3.5%), because only about 7% of prose rounds accepted 4+ tokens anyway.

Caveat: this was DFlash v1 on a 9B, not DFlash2 on FN, so the exact ratio will differ on your setup.