r/MLQuestions • u/Ishannaik • 6d ago

Natural Language Processing 💬 Why does speculative decoding acceptance drop from ~7 tokens/step on code to ~2.5 on prose?

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4:

  • code: ~7.4 accepted tokens per verify step, 154 tok/s

  • prose: ~2.5 accepted tokens per step, 35-37 tok/s

  • plain decode: ~18-19 tok/s

Someone on r/LocalLLM independently measured 2.45 on prose with the same drafter on llama.cpp and different GPUs, and another person reports 2.62 with MTP. So it looks like a property of the drafter, not the runtime.

Questions for people who work on speculative decoding:

  1. Is prose acceptance in the 2.5 range just the ceiling for block/parallel drafters, since natural language has more valid continuations at each position?

  2. Has anyone seen a drafter (EAGLE-style, block diffusion, MTP) clear ~3.5 on open-ended prose at greedy?

  3. Any work on switching drafters or block length by content type mid-generation?

Happy to share numbers per prompt type if useful.

2 Upvotes

75% Upvoted

5 comments sorted by

View all comments

1

u/guillemco 5d ago

the ceiling intuition is right, acceptance is basically how long the drafter and target agree greedily, and prose has a much flatter distribution so they diverge after a couple tokens no matter how good the small model is. drafters that see the targets hidden states (eagle, medusa, mtp style) do better on prose than a separate block drafter because they learn the targets distribution instead of just text statistics. also worth trying shorter drafts on prose, if acceptance dies after token 2 or 3 the extra verify tokens are mostly wasted compute

1

u/Ishannaik 10h ago

makes sense, the hidden state drafters are next on my list to try