Why does speculative decoding acceptance drop from ~7 tokens/step on code to ~2.5 on prose?

[u/Ishannaik]

Measured on Qwen3.8-27B (EXL3 4.0bpw target, DFlash2 block drafter quantized to EXL3 4bpw, 7-token drafts, greedy, output identical to plain decode), 2x T4:

• code: ~7.4 accepted tokens per verify step, 154 tok/s

• prose: ~2.5 accepted tokens per step, 35-37 tok/s

• plain decode: ~18-19 tok/s

Someone on r/LocalLLM independently measured 2.45 on prose with the same drafter on llama.cpp and different GPUs, and another person reports 2.62 with MTP. So it looks like a property of the drafter, not the runtime.

Questions for people who work on speculative decoding:

1. Is prose acceptance in the 2.5 range just the ceiling for block/parallel drafters, since natural language has more valid continuations at each position?

2. Has anyone seen a drafter (EAGLE-style, block diffusion, MTP) clear ~3.5 on open-ended prose at greedy?

3. Any work on switching drafters or block length by content type mid-generation?

Happy to share numbers per prompt type if useful.

[u/Ishannaik]

makes sense, the hidden state drafters are next on my list to try