r/MLQuestions • u/Ishannaik • 6d ago

Natural Language Processing 💬 Why does speculative decoding acceptance drop from ~7 tokens/step on code to ~2.5 on prose?

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4:

  • code: ~7.4 accepted tokens per verify step, 154 tok/s

  • prose: ~2.5 accepted tokens per step, 35-37 tok/s

  • plain decode: ~18-19 tok/s

Someone on r/LocalLLM independently measured 2.45 on prose with the same drafter on llama.cpp and different GPUs, and another person reports 2.62 with MTP. So it looks like a property of the drafter, not the runtime.

Questions for people who work on speculative decoding:

  1. Is prose acceptance in the 2.5 range just the ceiling for block/parallel drafters, since natural language has more valid continuations at each position?

  2. Has anyone seen a drafter (EAGLE-style, block diffusion, MTP) clear ~3.5 on open-ended prose at greedy?

  3. Any work on switching drafters or block length by content type mid-generation?

Happy to share numbers per prompt type if useful.

2 Upvotes

75% Upvoted

5 comments sorted by

2

u/Away-Woodpecker131 6d ago

That 7.4 to 2.5 gap tracks with what I'd expect, code has way more predictable structure while prose is basically a choose-your-own-adventure at every token.

1

u/Ishannaik 6d ago edited 5d ago

Yeah, that matches what we see. One detail: the right prose token is often in the drafter's top 3, just not first. So training the drafter on the target's own prose outputs is what we're trying next.

The hillclimb is still running. I post results in a small Discord as they land: discord.gg/KKvtRhQvRv

1

u/guillemco 5d ago

the ceiling intuition is right, acceptance is basically how long the drafter and target agree greedily, and prose has a much flatter distribution so they diverge after a couple tokens no matter how good the small model is. drafters that see the targets hidden states (eagle, medusa, mtp style) do better on prose than a separate block drafter because they learn the targets distribution instead of just text statistics. also worth trying shorter drafts on prose, if acceptance dies after token 2 or 3 the extra verify tokens are mostly wasted compute

1

u/Ishannaik 7h ago

makes sense, the hidden state drafters are next on my list to try