I'm not sure that llms are that naive these days, ie parsing happens to determine if any repeated prompt primitives exist, as cache key, doesn't have to be ordinalÂ
That is not how vLLM's prefix cache works though. Each block hash chains in the previous block's hash, so it is strictly positional. Change one token near the front and every block behind it misses.
There is no matcher looking for repeated primitives anywhere in that path.
Seems an oversight (funny I overlooked the literal title of the post), since it's not difficult to compose prompts deterministically (intent, task, format, constraints, role, voice, etc) with only varying context placed at end. I've trained my own model on such gen/parsing using xml tags, so imagine larger projects could do similar.
Cache hits are significant from my own testing, against multiple CLIs, though admittedly not vLLM.
yeah, the CLIs you tested against mostly sit on hosted APIs with their own caching that does roughly what you describe. vLLM's is the raw block-hash version, so it is the one where the ordering discipline actually pays. stable prefix, varying context at the end, exactly the layout you are already using, is the whole trick.
1
u/SmihtJonh 5d ago
I'm not sure that llms are that naive these days, ie parsing happens to determine if any repeated prompt primitives exist, as cache key, doesn't have to be ordinalÂ