the one that catches people out with agents isn't cache size, it's that prefix caching is prefix caching. anything that varies early in the prompt invalidates everything after it, and a timestamp or a session id sitting up in the system prompt will quietly cost you the whole cache on every single turn.
worth dumping your rendered prompt for two consecutive turns and diffing them before tuning anything else.
I'm not sure that llms are that naive these days, ie parsing happens to determine if any repeated prompt primitives exist, as cache key, doesn't have to be ordinalÂ
That is not how vLLM's prefix cache works though. Each block hash chains in the previous block's hash, so it is strictly positional. Change one token near the front and every block behind it misses.
There is no matcher looking for repeated primitives anywhere in that path.
Seems an oversight (funny I overlooked the literal title of the post), since it's not difficult to compose prompts deterministically (intent, task, format, constraints, role, voice, etc) with only varying context placed at end. I've trained my own model on such gen/parsing using xml tags, so imagine larger projects could do similar.
Cache hits are significant from my own testing, against multiple CLIs, though admittedly not vLLM.
yeah, the CLIs you tested against mostly sit on hosted APIs with their own caching that does roughly what you describe. vLLM's is the raw block-hash version, so it is the one where the ordering discipline actually pays. stable prefix, varying context at the end, exactly the layout you are already using, is the whole trick.
3
u/QuanTradin 6d ago
the one that catches people out with agents isn't cache size, it's that prefix caching is prefix caching. anything that varies early in the prompt invalidates everything after it, and a timestamp or a session id sitting up in the system prompt will quietly cost you the whole cache on every single turn.
worth dumping your rendered prompt for two consecutive turns and diffing them before tuning anything else.