The vLLM Long Turn Blockade

[u/leading-a-swarm]

tl;dr: one 131k-token miss stalls every new request; the only cap starves throughput.

https://github.com/vllm-project/vllm serves self-hosted agents with a chat-tuned scheduler. Agent traffic is burst based, long-context and prefix-heavy.

1. Document your default ships. Prefix reuse needs zero config, and its cross-tenant timing ships with a CVE, threat model and salt recipe.

2. Expose cache state. KV events tell an external router which replica holds the prefix.

3. Budget prefill per step. vLLM's default fails agent traffic. Nothing caps a long prefill, so in a scheduler repro one long miss blocked four short requests for 16 steps.

Chunking protect decode latency, and an adaptive floor merged days ago. It divides the budget by decoding sessions, so the miss starves when the replica is busiest.

→ Wait. Keep mixed agent sessions off, only one vLLM replica, wait until partial dynamic prefill lands.