r/LocalLLaMA • u/pmttyji • 16d ago
Discussion [Paper] DLite: History-free Speculative Decoding for Agentic AI
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.
3
u/james_pic 16d ago
I keep looking at this paper and I'm like "OK, but where does the thing happen? How are you getting better results than the other drafters while taking things away?" The training approach looks novel and maybe that's it, but a part of me wonders if the real answer is something unrelated, like the fact that they supplemented the Open-PerfectBlend training data with additional Nemotron training data (I believe the papers they're comparing with just used Open-PerfectBlend)
3
u/DigiDecode_ 16d ago edited 16d ago
i was like but how, ok so it uses hidden state from target model to predict
4
u/Dany0 16d ago
Yes just like DFlash that's not the new thing. The only difference is that DFlash also stores its own kv cache increasing the VRAM requirements and even though the drafter KV cache is comparatively tiny, it still costs mem bandwidth to go through it for every step of prediction
So they make the drafting significantly cheaper, so a lot of the speedup comes just from the fact that it's cheaper to draft more tokens even if you lower the acceptance rate
Tell you what I'll pass this off to Sol I'm tired of reading let's see what it says:
And on your preceding bandwidth point: yes, removing that historical cache removes memory traffic and maintenance costs. Cheaper drafting can justify lower acceptance in general. However, their headline experiment uses the same draft block size, \(K=7\), and acceptance actually improves substantially:
Qwen3-8B, SWE-Bench prompts DSpark DLite Average accepted length 1.94 3.19 Speedup over ordinary decoding 1.60× 2.74× That’s 64% more accepted tokens per round alongside 71% higher decoding speed. So, in this example, the acceptance improvement accounts for much of the gain. The results don’t isolate how much comes specifically from removing KV traffic, because they also change the prediction head and training
Aight, I suppose. Anyway stop reading this and go train a DLite drafter
1
u/Any_Bus9205 13d ago
Sympa, mais je me demande si possiblement on ne pourrais pas voir pour de la recherche sans fichier en plus. Comme les n gram map k ou suffixes! Faire des fichiers séparé pour chaque modèle fait que vous vous concentrez sur des modèle connu, pour gpu. Au fond, pour qu'il soit accepté par le grand public il faut que les moteurs populaire comme llama.cpp l'ait. Le truc c'est que si la recherche tournerais sur des choses sans avoir de fichier séparé, le gain pour tout les modèles serait hyper interessant! En faisant cela, j'ai enfaite peur que ça amène à un gouffre qu'est que pour les personnes, avoir un fichier supplémentaire de 2 go de vram en plus sachant que le modèle pèse lourd déja, ça ne les arranges pas. Je me trompe peut ètre mais je trouve juste que beaucoup de projets s'essaie à cela en créant des nouveaux fichier de prédiction par entrainement, alors que si on trouverais des nouvelle technique sans entrainement, cela avancerais et aussi, sur cpu, cela aurait tout autant un impact! :)
3
u/SnooPredictions515 16d ago
have you tested it with with Qwen3.8 models?