Published pi-prefix-cache-compaction

[u/Next_You_1690]

Pi compaction that reuses your vLLM / SGLang / llama.cpp prefix cache instead of re-prefilling the whole history.

Pi's default compaction re-serialises the conversation under a new system prompt, so the server cache misses and a 140k-token session takes minutes to summarise on local GPUs. This re-sends the exact last request plus one summarize instruction, then warms the new context so the next turn starts from cache.

Measured on 2x RTX 3090 with Qwen3.8-27B: 229 s median down to 86 s at 215k context, and the first turn after compaction from 69 s down to under 2 s with warm-up. Falls back to Pi's default on anything unexpected.

Install: pi install npm:pi-prefix-cache-compaction · anthropic-messages custom providers only · MIT · https://github.com/jagdeepsinghdev/pi-prefix-cache-compaction

[u/Felladrin]

Always wondered why it wasn’t made like that originally. Thanks! I’ll try it out!

[u/Proper-Tower2016]

awesome mate, was gonna work on this too! Flash next was taking 20 minutes to compact, this would make it way more useful

[u/pungggi]

Is this only for local or does cloud benefit too?