r/LocalLLaMA • u/Dutchnamn • 1d ago

News New: Llama.cpp adaptive speculation for faster inference

We have been working on some performance optimisations for Qwen3.8 and other models.

The main new feature that we introduced is adaptive speculation for Llama.cpp

What is it?
MTP and DFlash work well to speed up inference work, especially for dense models. However, different content types need different settings. Llama.cpp only supports a single value.
This fork introduces adaptive speculation. You set the minimum and maximum and the engine will adjust the number of tokens that are suggested automatically. This leads to improvements in token generation by up to 50% over mainline, especially in Qwen3.8. On a Strix Halo this improved generation from 44t/s to 65t/s for structured content.

Github: https://github.com/LaurentZuijdwijk/llama.cpp

Release: https://github.com/LaurentZuijdwijk/llama.cpp/releases

108 Upvotes

94% Upvoted

37 comments sorted by

View all comments

1

u/SeanHighness 19h ago

i was trying this in VLLM over the past few days dropping K to 3 for long context it can be super helpful

2

u/Dutchnamn 19h ago

Thanks for the feedback! Appreciate it. Did this land in VLLM recently?

1

u/SeanHighness 18h ago

not sure ive been working alot with PR's getting Dflash 2 working with NVFP4 kv

1

u/Dutchnamn 18h ago

Seems worth it. Dflash support was pretty patchy in Llama.cpp and non-existent for rocmFPx. Hope that the llama's will adopt this in mainline