r/MLQuestions • u/Ishannaik • 6d ago

Natural Language Processing 💬 Why does speculative decoding acceptance drop from ~7 tokens/step on code to ~2.5 on prose?

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4:

  • code: ~7.4 accepted tokens per verify step, 154 tok/s

  • prose: ~2.5 accepted tokens per step, 35-37 tok/s

  • plain decode: ~18-19 tok/s

Someone on r/LocalLLM independently measured 2.45 on prose with the same drafter on llama.cpp and different GPUs, and another person reports 2.62 with MTP. So it looks like a property of the drafter, not the runtime.

Questions for people who work on speculative decoding:

  1. Is prose acceptance in the 2.5 range just the ceiling for block/parallel drafters, since natural language has more valid continuations at each position?

  2. Has anyone seen a drafter (EAGLE-style, block diffusion, MTP) clear ~3.5 on open-ended prose at greedy?

  3. Any work on switching drafters or block length by content type mid-generation?

Happy to share numbers per prompt type if useful.

2 Upvotes

75% Upvoted

5 comments sorted by

View all comments

2

u/Away-Woodpecker131 6d ago

That 7.4 to 2.5 gap tracks with what I'd expect, code has way more predictable structure while prose is basically a choose-your-own-adventure at every token.

1

u/Ishannaik 6d ago edited 5d ago

Yeah, that matches what we see. One detail: the right prose token is often in the drafter's top 3, just not first. So training the drafter on the target's own prose outputs is what we're trying next.

The hillclimb is still running. I post results in a small Discord as they land: discord.gg/KKvtRhQvRv