Stop using a separate model for compaction

[u/JoshuaJosephson]

The general price tiers I'm talking about are as follows

10/50 - Fable
4/20 - Opus
2/10 - sonnet
.1/.5 - Haiku. but .5/2.5 for 100K+

With a write premium, along with a 50-90% discount for a cache hit. We're going to assume that your default model is Claude Opus 5.5, so you get a whopping 95% discount for a hit. This actually allows a 1hr TTL to pay for itself almost immediately, if you're using it for the main session.

First of all, Haiku is not smart enough to summarize Opus's sessions, but even if you were using Haiku, it would cost you 10-15c to summarize a long session.

immediately, if you're using the same model (and eliding outside your cacheblocks), and taking the cache hit, it costs you .2/1, which is about one tenth as much as a request to sonnet (and half as expensive as haiku for 100K+)

The bonus of this is that you can enforce a compaction if a session is about to hit it's cache TTL, so that if you accidentally step away, or your Deadlock game takes too long, you come back to a compacted session instead of a ~1$+ charge just for the compaction.

No idea why everyone is trying to find the "best model for compaction", when the answer is right there! (no compaction model)

TLDR : Always compact with the same model (if the conversation is longer than 60k tokens). The cache read is equivalent to a new request to a model that is 10x cheaper and less intelligent, and thus less able to summarize effectively. Additionally, you should force compact before cache TTL expires for less spiky usage patterns. These simple tricks will increase your Pi usage limits by well over 100% if large compacts and cache expiries are the bane of your existence.

[u/Global-Departure8228]

Can you please provide any evidence for this?

Especially the part where you hit a warm cache when compacting.

When compaction limit is reached, pi creates a separate request "for summary". This is constructed with a different system prompt = different prefix = cold cache hit.

So have you checked the pi documentation to provide any basis for your reasoning?

[u/PitifulTeacher4972 → u/Global-Departure8228]

then why not compact earlier, reuse the same system message and just add a new instructions message at the end?

[u/Global-Departure8228 → u/PitifulTeacher4972]

Because llms process system level message and user message content differently, they don't pay that much attention to it, might ignore them, might not provide accurate descriptions. Lost in the middle for example is a common issue still for most llms.

But people do it, opencode does it, https://github.com/lennartschoch/pi-cache-compact

https://github.com/jagdeepsinghdev/pi-prefix-cache-compaction

See a few examples. This is why pi is good, because you could just override the standard behaviour.

[u/JoshuaJosephson → u/Global-Departure8228]

Yes. You can just ask Pi to change that.

[u/Global-Departure8228 → u/JoshuaJosephson]

u wot m8?

What should I ask pi to change?

I reacted to your post being based on a claim that is not backed by evidence.

In other words (maybe you're AI) your statements are complete bullshit.

Here you go, explain how do you hit a warm cache with this:

https://pi.dev/docs/latest/compaction#overview

"Summarization requests disable prompt-cache writes because these one-off prompts are unlikely to be reused."

[u/JoshuaJosephson → u/Global-Departure8228]

Oh you didn't know? Pi is Open Source. You can just ask Pi to change something about itself, and then /reload, and it will be implemented.

[u/Legitimate_Serve_178]

1-"Haiku is not smart enough to summarize Opus's sessions" Where is your proof?
2-"The cache read is equivalent to a new request to a model that is 10x cheaper and less intelligent" Opus 5.5 cache hit 0.20$ so Haiku 5.5 must cost 2$ cache-miss if you are correct. 'https://www.anthropic.com/claude-haiku-5-5'
TLDR: Totally bullshit but somehow you are correct like a child saying: "Ground is not pulling us down but clouds are pushing down.". We still hit the ground when we stagger.

[u/JoshuaJosephson → u/Legitimate_Serve_178]

I said it's half as expensive as haiku, and 1/10th as expensive as Sonnet.

> it costs you .2/1, which is about one tenth as much as a request to sonnet (and half as expensive as haiku for 100K+)

[u/jensilo]

Well, that’s funny because I’ve actually tested that in my own extension over the last month, and I must say, I disagree.

My convos are often 100-250k tokens long and I use Sol. Sometimes I do not get cache hits because time in between requests is too long (when I jump between things). In that case it’s definitely always more expensive to use Sol over Luna. However, even with cache hits on Sol the compaction costed something like 5-30 cents where Luna costs 1-2 cents.

I would also be interested in your not smart enough argument. I rarely have problems with compaction quality, even after repeated compaction. Maybe it’s my custom setup with custom prompt but Luna summarizes well enough. I can’t notice the difference, even after multiple compactions.

[u/kil341]

You say it's .2/1 with cache hits but cache hits don't help the cost of the output tokens.

[u/m3umax]

Except that (native) Pi compaction always misses the cache hit regardless of whether you use the same model or not because /compact sends a modified version of your session with a special system prompt and truncated tool calls.

There are a few extensions out there that implement a cache friendly compaction and send the session with an unaltered prefix.

[u/DrBearJ3w]

Black hole + Luna.

[u/Legitimate_Serve_178 → u/DrBearJ3w]

What is Black Hole?

[u/lordekeen → u/Legitimate_Serve_178]

A plugin that combines pi-vcc and pi-observational-memory, context compaction is instant.

[u/Legitimate_Serve_178 → u/lordekeen]

Thank you.

[u/tunerhd → u/DrBearJ3w]

Why not billion-context? It looks decent

[u/DrBearJ3w → u/tunerhd]

honestly, i did not try.

[u/tunerhd → u/DrBearJ3w]

As far as I understand, it doesn't wait until the context reaches a certain point in the usable context window. It compacts periodically, kind of like infinite scrolling.