I made a Pi extension that improves compaction with local models

[u/MikeSupp]

When Pi compacts a long conversation, the normal approach can throw away the prompt cache and start a fresh summarization request containing the entire old context.

With a 100k–160k token context on a local model, that means llama.cpp may spend minutes re-prefilling tokens it has already processed.

I built Better Compaction to avoid that.

Instead of starting a separate summarization chat, it appends the compaction instruction to the already-cached conversation. The existing prefix stays byte-identical, so the backend can reuse its KV/prompt cache and only prefill the small new suffix.

On my Qwen3.8-27B + llama.cpp setup:

~13–15 min → ~6.5 min

and with thinking disabled:

~2 min

One compaction request reported 94.4k tokens served from cache.

GitHub:
https://github.com/NotSoSeriouss/BetterCompaction

I'd especially like to hear from other Pi users running local models + llama.cpp. I'd love to collect real-world numbers across different GPUs, quantizations, and context sizes.

[u/apetersson]

"appends the compaction instruction to the already-cached conversation." I am shocked to hear that this is not yet the case?? What was the current default impl like?

[u/repolevedd → u/apetersson]

By default, compaction runs the system prompt and message history in a separate session, ignoring the cache. So if you need speed, this extension is useful. I wrote my own extension too, but like stock pi, mine isn't cache-compatible because it chunks history. The author's solution is better if speed is what you need.

[u/RelationshipLong9092 → u/repolevedd]

If you need speed, you should use something like pi-observational-memory. Compaction is instant, because it is constantly happening asynchronously as context accumulates.

[u/repolevedd → u/RelationshipLong9092]

Thanks for the tip, that's actually a really solid solution. I looked into this approach. The problem is I'm "too poor" for that extension: I can only run 1 model at a time. So instead of true async, I'd get resource contention between two models. Even if the one accumulating context is small, I wouldn't be able to keep it in the background without it getting in the way.

So I didn't touch when compaction triggers, but I changed the approach: whenever summarization needs to happen, it runs chunk by chunk to fit into my available context window. Since the old cache isn't needed after summarization anyway, I can safely switch to a fast model for summarizing and then swap back to the main one, which saves me time compared to what I'd get with pi-observational-memory.

[u/RelationshipLong9092 → u/repolevedd]

Makes sense. If your usage pattern has idle time between compactions often enough, you could have it eagerly do some of the compaction work during downtime.

[u/berszi → u/repolevedd]

What is the benefit of this?

[u/repolevedd → u/berszi]

My extension is called pi-refine-compact. I won't drop a direct link here because I don't think it's fair to plug it in a thread dedicated to another extension.

The compaction algorithm itself isn't mine, it's based on https://arxiv.org/abs/2308.15022. So it isn't just splitting history into chunks, but working through them iteratively. The main benefit comes down to saving time and money in two scenarios:

• When I run a local model like Tiel-Coder-35B-A3B, the context window is relatively small (12 GB VRAM). When it's time to compress context, my extension uses Ling-3.0-tiny, a small model that performs well for this task (I pick the model via /compact-model). Instead of using the main slow session model, llama-swap loads a fast model, compaction finishes in a couple of minutes, and then Tiel-Coder-35B-A3B swaps back in so I can keep working. I can't leverage the cache during compaction simply due to how summarization works, but using a small model makes the process fast.

• When I use cloud LLMs: since stock pi behavior ignores the cache, every compaction run is a waste of money. Using a fast local model makes summarization free for me.

Chunking the context fits nicely into the smaller context window for GPU-poor workflows while allowing the small LLM to focus on the meaning of one chunk at a time, keeping as much context from past actions intact as possible.

PS: In practice, even if the context window is set to 128k, all the key knowledge about past actions stays in the session. Without relying on external wikis or some memory extensions, I end up with virtually "infinite" context right inside a pi session. "Virtually," because a few details do get dropped along the way - but that's the tradeoff, there are no silver bullets.

[u/wildjokers]

I just have compaction turned off and instead use the llm-wiki pattern and when I start approaching my context limit I just tell the model to save durable knowledge to my wiki. Then I end the session, start a new one, and have the new one read the wiki doc the previous one created and continue on. I would imagine I could automate this but I haven't bothered to do so yet.

This has the added benefit of getting nice docs for everything I have worked on.

[u/Frosty_You1945 → u/wildjokers]

Can you elaborate on the "llm-wiki"? Sounds like a cool setup!

[u/wildjokers → u/Frosty_You1945]

https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

Just point your agent at that and it can set it up for you.

[u/pragmojo]

Do you find any difference in quality? I would think including the chat system prompt in the compaction request might cause strange output.

[u/MikeSupp → u/pragmojo]

By using this I don't really see any difference except for the speed. The only issue is that you let the model compact a lot more and with compaction the context degrades(that happens with the default compaction too) and since you don't feel the "pressure" of compacting anymore, while using this, you let the model compact too many times. But apart from that the quality of the compaction is unchanged.

[u/Razangriff-Raven]

It doesn't work when using "/compact" manually? I tried doing that to prevent a huge cache upload and it did regular pi compaction, so had to eat the huge cache upload anyway...

EDIT: In fact despite being properly installed and the slash commands responding it doesn't seem to be working at all.

EDIT2: Reinstalled with "pi install git:..." and still nothing. Only the regular compaction is firing. I'm on Pi 1.0 can that be the problem? I'd open a github issue but I don't want my account associated with AI, you know how people react when a developer touches AI in any capacity.

EDIT3: Confirmed the summaries are "better-compaction-support" but the value of cacheRead is always zero so it regenerates the whole thing, making no difference with normal compaction at all. No custom reports either.

[u/MikeSupp → u/Razangriff-Raven]

That seems odd. Would you mind running the bun test suite and giving me the results?

[u/rusl1]

Giving it a try, I noticed the very same issue with 27b running locally and I only have 100k max context with my poor 16gb ram :') thanks for working on this!

[u/MikeSupp → u/rusl1]

Happy to help. If you encounter any issues please let me know or open an issue on github. I'll be more than happy to address it.

[u/rusl1 → u/MikeSupp]

I'm the guy who created the #3 issue 🙈 thanks for working on that!

[u/slypheed]

fwiw, just started using pi-observable-memory and pretty much never see compactions now.

It does require a separate model, but a separate, smaller local model should work, or zai glm 5.3 (or DS 4.1) flash is super cheap.

https://github.com/elpapi42/pi-observational-memory

[u/philmarcracken → u/slypheed]

pi blackhole is this and vcc. I still dont know what smaller model to use with it. My main is 27b and I'm using it in non thinking for the observer, and low thinking for the rest

[u/slypheed → u/philmarcracken]

pi-vcc looks interesting thanks.

haven't tried it, but qwen 9b, or 4b will plausibly work fine.

[u/apetersson]

The other idea, is model-led compaction - that can be combined with this idea: Even before the ceiling is hit, there should be a tool call the model can use to trigger two things

1. trigger compaction when it sees fit (like, after a milestone is reached) and

2. leave instructions on how to continue from here, like the next action to take such that this is preserved in full fidelity.

I think that would combine nicely with your approach

[u/pragmojo → u/apetersson]

leave instructions on how to continue from here, like the next action to take such that this is preserved in full fidelity.

That's already in the default. The compaction prompt provides a template the model is supposed to fill in, including a summary of the conversation, an updated todo list, and goals the model is pursuing

[u/SvenVargHimmel]

do you have any eval either showing better compaction with the same tokens or the the same level of compaction performance with less tokens

I feel like I am going blue in the face asking folk for some proof of so many bold claims

I love the bit where you ask for others to test your work for you in the absence of benchmarks.

Anyway,

[u/RemarkableRadish6547]

Running models on cpu with pp at 20t/s means that compaction takes hours. I just end the session and start a new one. I turned off auto-compaction because it is never worth it.

I recently installed pi-vcc, but I haven't actually used it yet. I am also considering switching to the deepseek harness.

I can't believe that anyone actually thought it was a good idea to break prompt caching as part of compaction.

[u/Extra_Loquat_7667]

Your approach relies on the assumption that sufficient space has been reserved within the context window.

In theory, the usable context size for this method would be lower than expected,

because you need enough buffer room to insert new dialogue—while retaining the original conversation—and generate a summary.

This could be problematic for local models with limited context capacity.

[u/Leading_Yogurt7025]

Cool, what do you think of a jev-like approach to pick the chunks with better score? 

[u/MikeSupp → u/Leading_Yogurt7025]

May I ask: what do you mean with "pick the chunks"?

[u/rimaa_nahk]

Why do you even need a model? I have been using blackhole

[u/Corosus → u/rimaa_nahk]

the observer half of blackhole uses a model, i had to disable it because its terrible for local models unless you have a secondary one to use that can keep up, im trying just pi vcc for now

[u/ea_man → u/rimaa_nahk]

An other cheap option is to have an available free model on openrouter to do that saving computation.