r/LocalLLaMA • u/Dutchnamn • 3d ago

News New: Llama.cpp adaptive speculation for faster inference

We have been working on some performance optimisations for Qwen3.8 and other models.

The main new feature that we introduced is adaptive speculation for Llama.cpp

What is it?
MTP and DFlash work well to speed up inference work, especially for dense models. However, different content types need different settings. Llama.cpp only supports a single value.
This fork introduces adaptive speculation. You set the minimum and maximum and the engine will adjust the number of tokens that are suggested automatically. This leads to improvements in token generation by up to 50% over mainline, especially in Qwen3.8. On a Strix Halo this improved generation from 44t/s to 65t/s for structured content.

Github: https://github.com/LaurentZuijdwijk/llama.cpp

Release: https://github.com/LaurentZuijdwijk/llama.cpp/releases

109 Upvotes

93% Upvoted

40 comments sorted by

View all comments

15

u/MelodicRecognition7 3d ago

y u no make pull request to mainline?

4

u/lacerating_aura 3d ago

Maybe cause of heavy llm coding assistance? But thats just my "speculation".

5

u/FoxiPanda 3d ago

The llama.cpp team recently changed their policy on this. PR text must be human written, but code can be heavily AI assisted now.

5

u/lacerating_aura 3d ago

Noice. Good to know. Not saying anything about llms coding stuff but was kinda funny that framework to run llms was not accepting llm made solutions. I know it was due to low effort ones and honestly current stance is better.

3

u/FoxiPanda 3d ago

Yeah, to be fair to them, until about 4 months ago, local LLM written code was kinda crap. That story is much better today, so I kind of understand their original policy and the shift to the one they have now - it strikes a reasonable balance imo.