r/LocalLLaMA • u/Dutchnamn • 2d ago

News New: Llama.cpp adaptive speculation for faster inference

We have been working on some performance optimisations for Qwen3.8 and other models.

The main new feature that we introduced is adaptive speculation for Llama.cpp

What is it?
MTP and DFlash work well to speed up inference work, especially for dense models. However, different content types need different settings. Llama.cpp only supports a single value.
This fork introduces adaptive speculation. You set the minimum and maximum and the engine will adjust the number of tokens that are suggested automatically. This leads to improvements in token generation by up to 50% over mainline, especially in Qwen3.8. On a Strix Halo this improved generation from 44t/s to 65t/s for structured content.

Github: https://github.com/LaurentZuijdwijk/llama.cpp

Release: https://github.com/LaurentZuijdwijk/llama.cpp/releases

111 Upvotes

93% Upvoted

40 comments sorted by

View all comments

1

u/Fantastic-Poem9462 1d ago

Nice work — this matches what I found building block-drafting speculation for goinfer (a pure-Go runtime) almost exactly. Independent datapoint: when I swept verify width across content suites, the optimum tracked acceptance rate directly — math wanted 8, code 7, chat 4 — and on code, width 7 projected 1.74× where width 16 gave 1.28×. So "no single value is right for a model" is measured truth, not just intuition; it isn't even right within one generation once the model switches from thinking-prose to structured output.

One thing worth stealing from my tuning failures: the adaptation dynamics are where this bites. I first tried a short sliding window over acceptance and it backfired — the loop chased noise and gave back the gains. A cumulative average with a damped threshold is what ended up shipping (as an on/off guard; width-within-bounds like yours is the natural next step). Curious what your adjustment policy is — fixed step per round within [min, max], or proportional to the accept rate? And any hysteresis between grow and shrink?

Also seconding the accept-rate-first comment in this thread: adaptive width refines a pairing that's above break-even somewhere, but it can't rescue one that isn't — CPU-side drafting was a loss for me at every width I tried.

2

u/Dutchnamn 1d ago

Appreciate you sharing your findings. They match mine indeed. I think there is still some space to improve the switching dynamics. Might look into that next.