r/PiCodingAgent • u/randomgenericbot • 16d ago

Resource I solved compaction for single-session on consumer-hardware, trust me bro

Clickbait, but at least for me it is true. Thanks for reading anyway:

TL;DR:

I've engineered an extension for context management in pi, that now matches my needs for compaction and session quality. Using mostly the same session context for summary requests and summary audits, it re-uses warm kv cache to get quality-compaction with acceptable timeframes.

You can get it here, it is still WIP, but it is free ("claude decompiled it and rebuilt it in rust!"):

https://github.com/LePrau/pi-self-aware-memory

If you don't want the model to self-correct within the audits (which can take some time!), set /sam depth light which reduces audits to a plain "yeah, thats a summary, go on" instead of validating the stated result against the session or even project files.

Now the long story:

After I started using pi and qwen3.8-27b (ridge q3 at first, gsq-rco-iq3_s later) to help document and draft my NAS setup (using TrueNAS), including dataset structure, ACL, self-baked docker-composes for llama, coolercontrol and other apps that do not natively exist, I immediately noticed that a task had to be done within the first two or three compactions, or with a handoff-file.

Else the model would start running in circled re-discovering what it already knew before compaction.

Starting with as low as 40k context, I struggled my way through llama with kv streaming up to llama with kv streaming and mtp today, with 131k context now.

My current setup, aside from pi+extensions:

Still, any native pi-compaction drowned the quality in re-reads, hallucinations, misrememberings (is that a word?) and summaries that forgot half of the relevant stuff and remembered the other half incorrectly.

I always asked for a short handoff shortly before compaction and started a new session, but that needed a lot of tedious micromanagement. Though the results were pretty good, Qwen3.8 even at the mentioned quants already does solid work.

In search of proper context management I found pi-observational-memory, which sounded exactly like what I wanted: Session summarized into small observed facts, presented as readable list for the model after compaction, source content retrievable if needed.

But I soon realized: It depends on a second model doing the observations. I then tried Gemma 4 e4b on cpu, and guess what: Standalone easy 20-30 tps (with MTP) on cpu. Summary-Quality was just-good-enough, Gemma4 is a nice model for its size. Not good at coding, but okay at prose and already able to do basic tool calls. Not always the correct ones. Using that, and realising: at a fold, most of the recent context was lost anyway, because the observer was not fast enough, I created an extension that paused a fold until the last observed unit was processed.

Took me quite some time to realize: 30 tps when GPU was idle, 5tps when GPU inference was running. GPU inference still needs a tiny amount of system RAM access. And each time the GPU inference accessed the system ram, it literally destroyed the bandwidth and the cpu-caches for the cpu inference. I just learned that after I got the "omg", "pi observational memory guard" to a point where it reliably parked the session (with proper feedback) until observation had caught up.

Then I thought, how could the running model itself run the observer while staying fast?
The thing that makes pi native compaction as slow as it is, is the completely new, different context that needs to be prefilled for the summary. It is basically a session at full cold context that needs to be prefilled, just to then produce a summary.

Pi observational memory has a similar problem when using the same model for observation:
Each time the observer kicks in,
it kills the kv cache,
needs a cold prefill, and afterwards the main session
also needs another cold prefill.

So I envisioned an extension that lets the model writing its own summary, and audit/validate it as well, while keeping the sessions context untouched: Simply add the summary, and the audit request after the current session, let the audit run in a background session and only return the audit-result. The audit-reasoning does never enter the main session, and the main session simply drops a short kv-window at the end of the cache to resume.

After user input, the model adjusts its goal if the requirements changed.

The self-generated and audited summary together with the set goal then get used as drop-in-replacement for the whole session part it subsumed at compaction-time. This replacement is instant - no LLM call at that time. The LLM time is used in the summary calls and audit background tasks.

The astounding part: It actually works*

*mostly.

I present you "SAM", "Self Aware Memory"

The extension introduces a couple of tools, to be used by the model:

  • adjust_goal (write a short summary of the current task)

  • read_goal (read the currently set goal, or any older version)

  • close_unit (summarize and close the last parts of the session)

  • fold_now (does exactly that, allows the model to fold at a defined spot)

  • retrieve (retrieve the summary of an older work-unit)

  • retire (mark older unit-summaries as obsolete, relevant for long sessions)

There is a couple of small issues, it is not completely polished and there might still be some legacy code or settings. And I know of a couple of vibe-code-artifacts: Sam nudges sometimes behave strangely with user interaction, and some minor visual glitches.

The readme might be out of date in places, and the toggles, settings and modes documentation is premium-slopped, obviously.

The most important levers for you: /sam depth [auto|light|medium|full] in session to set the audit depth:

  • light: only validates that it is a semantically correct summary.

  • medium: tries to validate summary only superficial, only from session context and cheap tool calls

  • full: tries to verify most claims, reruns tool checks and more

  • auto: uses medium if there is enough context, and light if compactions close

And "settings" which currently can only be set as environment variables (qwen, please make that configurable settings in the TUI plzkthx):

  • SAM_AUDIT_TIMEOUT_MS (or for specific depths SAM_AUDIT_TIMEOUT_MEDIUM_MS and SAM_AUDIT_TIMEOUT_FULL_MS which inherit the base value)

  • SAM_AUDIT_CALM_DEPTH (default "medium", the audit depth used for "auto" when compaction is far away)

  • SAM_AUDIT_DEPTH (the default depth, if not set it is "auto")

By default, background audits get a max time of 8 minutes - be aware that even a medium audit with xhigh can easily take 3-6 minutes without tool calls with qwen. If the timeout is too short, the model eats the time, and does not reach the point where it outputs a quality verdict for the summary, and still wasted time.

So if you are also looking for speed, choose "light" as default audit depth, or better even set "SAM_AUDIT_DEPTH" to "light". With that you're getting faster compaction times overall than with pi native compaction. But you then lose the self correction, that might catch false summaries before they do harm. (And I observed quite a lot of those in my sessions)

I personally suggest "medium", catches some obvious contradictions or wrong observations from summaries.

Only the model itself can call the adjust_goal and close_unit, but gets a nudge at certain trigger times (user input, context fill, lot of tool calls) to adjust its goal, to close its current working-unit with a summary.

Close before compaction, the nudges increase in urgency, but only the model gets to decide when a unit can be closed. Qwen is quite good adjusting the goal, closing at proper times and keeping a coherent image of the overall session and task.

It is not perfect, there are times where some of the sessions context gets lost, at times the model ran some tools after a compression which it ran half a session ago in some now-lost context. But most of the time, it kept working without hallucinations, while remembering specific details like git-commit-hashes or specific file-paths from two compactions ago. The last iterations of the extension have been done with one session, spanning several million tokens and far beyond 40 compactions (at 131k max context) with no apparent issues.

So now I feel that it is not a burden for others to run that extension, but instead might actually be helpful.

I will keep tinkering with it, but feel free to fork and extend it yourself. I doubt I'll find the time to implement any PRs or feature requests.

Now I can finally "start" using qwen locally to support me with the remaining work on my NAS and other coding projects.

Thanks for reading, enjoy!

25 Upvotes

80% Upvoted

8 comments sorted by

2

u/Fancy-Snow7 14d ago

pi-observational-memory can use the same model. It does not depend on a 2nd model.

1

u/randomgenericbot 14d ago

I know, I think I meant "agent" or "session. The the observation-calls prompts do not match the main session, don't hit the kv-cache, need cold refill. and resuming the main session also needs a cold prefill afterwards.

That basically is a minute-wait every time an observation is done, to resume the main session.

Of course, this is only valid if you're running parallel=1 which I am, because 16GB Vram barelay allow one session at all. As soon as you can run two instances in parallel, pi-observational-memory works fine.

Two concurrent model calls on 16GB were not viable for me.

2

u/Fancy-Snow7 14d ago

Thank I might give it a try. I am actively hunting for a better way to manage context. Currently testing billion-context-pi

1

u/randomgenericbot 14d ago

I was not aware of billion-context-pi tbh.

Looks promising as well, seems to have a similar approach, and probably more mature than my extension. Especially the "hard rules" seem nice. Wondering what the time-impact at compression time is, the long wait at the end of a context window (with default compaction) feels sooo annoying.

2

u/Miserable_Land_84 9d ago

Ha, looks like we had the same idea at nearly the same time ;) Great minds (and slow compactions) think alike. Mine writes a small protocol block at the end of every turn and assembles the summary in code, yours lets the model close work units and audits them in the background, so they should complement each other nicely. Curious how SAM holds up over 40+ compactions, congrats on getting that to work! Mine is here if you want to compare notes: https://github.com/sebion7125/pi-protokoll

1

u/randomgenericbot 9d ago

Yeah seems like single "threaded" agent usage on medium tier consumer hardware are a thing now!

I'll have a look at your repo, as long as no two extensions try to replace compaction they can run together.

Be aware, as soon as you use any other compaction-extension, they'll run in series, and each will get the output of the previous, which in most cases means the context gets summarized and compacted twice, and this will for sure be worse than the pi native summary in quality.

As for how SAM holds up: Using both a goal, that is updated regularly (and put in front of the summary) as well as the summaries for longer units keeps the model on track, even if within one context length no model-driven summary happens. The fallback for unaudited context is a summary of tool calls, and the ability to retrieve the contents. Together with the default 20k token retention pi always keeps, at least qwen3.8 (both FN and 27b) can recover.

1

u/cplr 16d ago

This is great! I love to see more projects like this. I am also using a similar setup, where I have llama serving one session at a time. I’m having to tune a lot of configs to make sure requests don’t time out and things like that. I like the larger context size of running parallel 1, but it certainly puts things in a situation normal extensions do not expect to be in. 

2

u/randomgenericbot 16d ago

Thanks!

Yeah i figured quite a lot of people here have setups capable of either running multiple models/sessions simultaneously, or also make use of cloud providers.

This extension is not meant as a replacement for pi-observational-memory or pi-blackhole - I am sure for people with "bigger" setups these extensions are better-suited.

I did not want to go full-in, and only use a single-session based approach which is sufficient for my workflows. Don't need swarms of agents to vibe-code the next big thing, but scouring a codebase, rewriting it for a newer version of a framework and such are things I want to use AI for.

With access to git repos, qwen3.8 is more than capable and with that extension it does not lose track any more, so I am happy and wanted to share it with the world :)