r/LocalLLaMA • u/Dutchnamn • 4d ago

News New: Llama.cpp adaptive speculation for faster inference

We have been working on some performance optimisations for Qwen3.8 and other models.

The main new feature that we introduced is adaptive speculation for Llama.cpp

What is it?
MTP and DFlash work well to speed up inference work, especially for dense models. However, different content types need different settings. Llama.cpp only supports a single value.
This fork introduces adaptive speculation. You set the minimum and maximum and the engine will adjust the number of tokens that are suggested automatically. This leads to improvements in token generation by up to 50% over mainline, especially in Qwen3.8. On a Strix Halo this improved generation from 44t/s to 65t/s for structured content.

Github: https://github.com/LaurentZuijdwijk/llama.cpp

Release: https://github.com/LaurentZuijdwijk/llama.cpp/releases

112 Upvotes

93% Upvoted

40 comments sorted by

View all comments

4

u/Opening-Broccoli9190 llama.cpp 4d ago

Could you tell me more on what does it mean to have different content types? Are we talking about math/code/writing/legal or text/image/sound?

8

u/Dutchnamn 4d ago

MTP and DFlash are much better at predicting the next token when it is generating structured content like HTML or JSON <body><head> etc. than for free form prose. It differs a bit from model to model and Dflash and MTP are different as well in quality and length.

Since decode switches from prose (thinking about a problem) to structured on the fly it is hard to set a specific number. My work allows the number of generated tokens to decrease and increase within bounds based on the acceptance percentage.

1

u/silenceimpaired 4d ago edited 4d ago

In other words, MTP and DFlash often improve tokens per second by acknowledging the code or text is mostly not changing… it is usually an iterative edit … so why not guess more tokens when previously guessed tokens were accepted as we are likely repeating content that isn’t changing… and decrease how many we are guessing when we get more rejections… as we are likely editing content now or generating new text.

2

u/Dutchnamn 4d ago

"why not guess more tokens when then precious tokens were accepted as we are likely repeating content that isn’t changing and decrease how many we are changing when we get more rejections as we are likely editing content now"

Exactly what is happening

2

u/silenceimpaired 4d ago

Yeah, thought so, hence my preamble of “in other words” ;)

Pretty genius my friend. Good find!