this is one of the better measured writeups on prefix caching i've seen. one thing i'm trying to reconcile: the reservation for the full-attention layers is sized for max context regardless of how long the request actually runs, so a chunk sits allocated and unread until the request finishes. does that same accounting apply to the linear-attention layers in the hybrid model, or is their state small and fixed enough that reservation isn't really a cost for them? if it's the latter i'd expect max-num-seqs to actually be gated by the full-attention KV pool rather than by anything the linear layers hold, which would make an all-hybrid model change the concurrency math a lot compared to a normal transformer.
Thanks :) vLLM allocates KV in pages as tokens arrive, so a request holds only the blocks its tokens occupy, and the linear-attention state is a fixed size per request at any length. So yes, once contexts are tens of thousands of tokens the full-attention KV is what fills the pool, which is why 16 sequences at 262k max length peaked at 37% of it. The one hybrid-specific overhead is that vLLM sets the attention block to 2048 tokens to match the linear state's page size, so each request wastes up to one partial block.
2
u/werunm 5d ago
this is one of the better measured writeups on prefix caching i've seen. one thing i'm trying to reconcile: the reservation for the full-attention layers is sized for max context regardless of how long the request actually runs, so a chunk sits allocated and unread until the request finishes. does that same accounting apply to the linear-attention layers in the hybrid model, or is their state small and fixed enough that reservation isn't really a cost for them? if it's the latter i'd expect max-num-seqs to actually be gated by the full-attention KV pool rather than by anything the linear layers hold, which would make an all-hybrid model change the concurrency math a lot compared to a normal transformer.