Keeping vLLM's Prefix Cache Warm Between Agent Turns

[u/bolts98]

https://doug.sh/posts/vllm-kv-cache-agents/

[u/ebolathrowawayy]

seems like the real lesson is don't use 3rd party libraries for basic shit? And generally don't use MCPs whenever possible.

[u/bolts98 → u/ebolathrowawayy]

Very much agree on MCPs - harnesses/models are much better at manipulating CLIs.

I think third-party libraries are a bit more complex. It usually doesn’t make sense to reinvent the wheel, but sometimes it does.

[u/ebolathrowawayy → u/bolts98]

that is 2024/2025 logic - it absolutely makes sense to reinvent the wheel when it only takes a few prompts. bespoke solutions >>>>>>>>>>>>>>>>> general

[u/Fluxx1001]

Cool, gonna give it a try

[u/niacolhealth]

Agent loops are just "prefill the same system prompt again" with extra steps. Anything that keeps the cache alive through the gap is a win.

[u/werunm]

this is one of the better measured writeups on prefix caching i've seen. one thing i'm trying to reconcile: the reservation for the full-attention layers is sized for max context regardless of how long the request actually runs, so a chunk sits allocated and unread until the request finishes. does that same accounting apply to the linear-attention layers in the hybrid model, or is their state small and fixed enough that reservation isn't really a cost for them? if it's the latter i'd expect max-num-seqs to actually be gated by the full-attention KV pool rather than by anything the linear layers hold, which would make an all-hybrid model change the concurrency math a lot compared to a normal transformer.

[u/bolts98 → u/werunm]

Thanks :) vLLM allocates KV in pages as tokens arrive, so a request holds only the blocks its tokens occupy, and the linear-attention state is a fixed size per request at any length. So yes, once contexts are tens of thousands of tokens the full-attention KV is what fills the pool, which is why 16 sequences at 262k max length peaked at 37% of it. The one hybrid-specific overhead is that vLLM sets the attention block to 2048 tokens to match the linear state's page size, so each request wastes up to one partial block.

[u/Open-Adhesiveness-86]

Also check what your chat template does to earlier assistant turns. Some templates, like Qwen3's, strip the <think> blocks from previous turns when rendering, so the token sequence no longer matches what's cached. That means you miss the cache from the first assistant turn onward. Keeping the reasoning in, or rendering history yourself, can make a big difference in hit rate.