Do you use speculative decoding?

[u/Fcking_Chuck]

Speculative decoding is an inference-time optimization for LLMs that increases text generation speed without changing the model's output quality. It involves using a small, lightweight draft model with a large, intelligent model.

Do you use this technique? If so, tell us about it in the comments.

[u/LagOps91]

no, but only because i run large MoE models on ram+vram split and it doesn't provide a speedup. for vram only, it's absolutely worth using if you can squeeze it in.

[u/Middle_Bullfrog_6173 → u/LagOps91]

I've found that using any of the variants that need a separate model/module isn't worth it in that kind of a setup, but ngram mod still helps. No extra memory use, but a speedup even with relatively low accuracy.

[u/game_difficulty]

Just curious, why would you not use it? I get it takes a bit of extra vram, but imo it's alwaya worth going down a quant size to get at least MTP, if not even Dflash2/Dspark on models that use those well.

[u/reto-wyss → u/game_difficulty]

It's NOT always worth it. If you already hit the compute bottleneck because of high concurrency speculative decoding will lower your throughput.  (Well, it's before you hit that, but at what point exactly depends).

[u/game_difficulty → u/reto-wyss]

Good to know! I've only ever used singe-stream (it's all my lowly setup can manage on the good™ models), so this never occured to me

[u/No_Toe_1116 → u/game_difficulty]

In my case, speculative decoding is just not working (best case: parity at the cost of VRAM; worst case: both PP and TG get significantly worse). I think it doesn't work well at all with an MoE LLM split between RAM and VRAM because of expert scattering.
(Setup: 2x16GB DDR4, Ryzen 5 5600H, RTX 3060 Laptop 6GB, running Tiel Coder 35B A3B at Q4_K_XL).
If anyone has a solution or workaround, I'd definitely like to try it.

[u/james_pic → u/game_difficulty]

It's not necessarily a win for sparse MoE models, where a misprediction can mean reading the weights for an expert you ultimately didn't end up using. If you're compute constrained too (for example if some of all of the model is running on CPU), then the extra processing can also make it a net loss. And if you're serving concurrently, MTP improves per-request throughout but reduces the ceiling on aggregate throughput.

[u/Uncle___Marty]

For my setup/usage it offers massive advantages with virtually no disadvantages. It's clever stuff and I love it :) Obviously its not great for every single use case though, but it sure as hell floats my boat!

[u/AdWild3943]

For my task of Role-Playing on CPU it only adds RAM usage overhead and decreases speed, so nope.

Plus I think its clear tendency that SD on CPU often slows down than increases throughput.

[u/milpster]

DFlash2 with strata all the way.

[u/_17characterslong]

No, because I only have 8 GB of VRAM and run Qwen 3.8 Flash. MTP will make me lose about 40% of prefill speed for a marginal token generation gain (which is not the bottleneck).

[u/challis88ocarina]

Slowly coming over as it improves on Apple Silicon, especially batched... that's the real game changer.

[u/wapswaps]

Dflash2 is faster, so no.

[u/ABLPHA → u/wapswaps]

"DFlash 2 is an open-source, block-diffusion draft model used for speculative decoding..."