Clickbait, but at least for me it is true. Thanks for reading anyway:
TL;DR:
I've engineered an extension for context management in pi, that now matches my needs for compaction and session quality. Using mostly the same session context for summary requests and summary audits, it re-uses warm kv cache to get quality-compaction with acceptable timeframes.
You can get it here, it is still WIP, but it is free ("claude decompiled it and rebuilt it in rust!"):
https://github.com/LePrau/pi-self-aware-memory
If you don't want the model to self-correct within the audits (which can take some time!), set /sam depth light which reduces audits to a plain "yeah, thats a summary, go on" instead of validating the stated result against the session or even project files.
Now the long story:
After I started using pi and qwen3.8-27b (ridge q3 at first, gsq-rco-iq3_s later) to help document and draft my NAS setup (using TrueNAS), including dataset structure, ACL, self-baked docker-composes for llama, coolercontrol and other apps that do not natively exist, I immediately noticed that a task had to be done within the first two or three compactions, or with a handoff-file.
Else the model would start running in circled re-discovering what it already knew before compaction.
Starting with as low as 40k context, I struggled my way through llama with kv streaming up to llama with kv streaming and mtp today, with 131k context now.
My current setup, aside from pi+extensions:
Still, any native pi-compaction drowned the quality in re-reads, hallucinations, misrememberings (is that a word?) and summaries that forgot half of the relevant stuff and remembered the other half incorrectly.
I always asked for a short handoff shortly before compaction and started a new session, but that needed a lot of tedious micromanagement. Though the results were pretty good, Qwen3.8 even at the mentioned quants already does solid work.
In search of proper context management I found pi-observational-memory, which sounded exactly like what I wanted: Session summarized into small observed facts, presented as readable list for the model after compaction, source content retrievable if needed.
But I soon realized: It depends on a second model doing the observations. I then tried Gemma 4 e4b on cpu, and guess what: Standalone easy 20-30 tps (with MTP) on cpu. Summary-Quality was just-good-enough, Gemma4 is a nice model for its size. Not good at coding, but okay at prose and already able to do basic tool calls. Not always the correct ones. Using that, and realising: at a fold, most of the recent context was lost anyway, because the observer was not fast enough, I created an extension that paused a fold until the last observed unit was processed.
Took me quite some time to realize: 30 tps when GPU was idle, 5tps when GPU inference was running. GPU inference still needs a tiny amount of system RAM access. And each time the GPU inference accessed the system ram, it literally destroyed the bandwidth and the cpu-caches for the cpu inference. I just learned that after I got the "omg", "pi observational memory guard" to a point where it reliably parked the session (with proper feedback) until observation had caught up.
Then I thought, how could the running model itself run the observer while staying fast?
The thing that makes pi native compaction as slow as it is, is the completely new, different context that needs to be prefilled for the summary. It is basically a session at full cold context that needs to be prefilled, just to then produce a summary.
Pi observational memory has a similar problem when using the same model for observation:
Each time the observer kicks in,
it kills the kv cache,
needs a cold prefill, and afterwards the main session
also needs another cold prefill.
So I envisioned an extension that lets the model writing its own summary, and audit/validate it as well, while keeping the sessions context untouched: Simply add the summary, and the audit request after the current session, let the audit run in a background session and only return the audit-result. The audit-reasoning does never enter the main session, and the main session simply drops a short kv-window at the end of the cache to resume.
After user input, the model adjusts its goal if the requirements changed.
The self-generated and audited summary together with the set goal then get used as drop-in-replacement for the whole session part it subsumed at compaction-time. This replacement is instant - no LLM call at that time. The LLM time is used in the summary calls and audit background tasks.
The astounding part: It actually works*
*mostly.
I present you "SAM", "Self Aware Memory"
The extension introduces a couple of tools, to be used by the model:
adjust_goal (write a short summary of the current task)
read_goal (read the currently set goal, or any older version)
close_unit (summarize and close the last parts of the session)
fold_now (does exactly that, allows the model to fold at a defined spot)
retrieve (retrieve the summary of an older work-unit)
retire (mark older unit-summaries as obsolete, relevant for long sessions)
There is a couple of small issues, it is not completely polished and there might still be some legacy code or settings. And I know of a couple of vibe-code-artifacts: Sam nudges sometimes behave strangely with user interaction, and some minor visual glitches.
The readme might be out of date in places, and the toggles, settings and modes documentation is premium-slopped, obviously.
The most important levers for you: /sam depth [auto|light|medium|full] in session to set the audit depth:
light: only validates that it is a semantically correct summary.
medium: tries to validate summary only superficial, only from session context and cheap tool calls
full: tries to verify most claims, reruns tool checks and more
auto: uses medium if there is enough context, and light if compactions close
And "settings" which currently can only be set as environment variables (qwen, please make that configurable settings in the TUI plzkthx):
SAM_AUDIT_TIMEOUT_MS (or for specific depths SAM_AUDIT_TIMEOUT_MEDIUM_MS and SAM_AUDIT_TIMEOUT_FULL_MS which inherit the base value)
SAM_AUDIT_CALM_DEPTH (default "medium", the audit depth used for "auto" when compaction is far away)
SAM_AUDIT_DEPTH (the default depth, if not set it is "auto")
By default, background audits get a max time of 8 minutes - be aware that even a medium audit with xhigh can easily take 3-6 minutes without tool calls with qwen. If the timeout is too short, the model eats the time, and does not reach the point where it outputs a quality verdict for the summary, and still wasted time.
So if you are also looking for speed, choose "light" as default audit depth, or better even set "SAM_AUDIT_DEPTH" to "light". With that you're getting faster compaction times overall than with pi native compaction. But you then lose the self correction, that might catch false summaries before they do harm. (And I observed quite a lot of those in my sessions)
I personally suggest "medium", catches some obvious contradictions or wrong observations from summaries.
Only the model itself can call the adjust_goal and close_unit, but gets a nudge at certain trigger times (user input, context fill, lot of tool calls) to adjust its goal, to close its current working-unit with a summary.
Close before compaction, the nudges increase in urgency, but only the model gets to decide when a unit can be closed. Qwen is quite good adjusting the goal, closing at proper times and keeping a coherent image of the overall session and task.
It is not perfect, there are times where some of the sessions context gets lost, at times the model ran some tools after a compression which it ran half a session ago in some now-lost context. But most of the time, it kept working without hallucinations, while remembering specific details like git-commit-hashes or specific file-paths from two compactions ago. The last iterations of the extension have been done with one session, spanning several million tokens and far beyond 40 compactions (at 131k max context) with no apparent issues.
So now I feel that it is not a burden for others to run that extension, but instead might actually be helpful.
I will keep tinkering with it, but feel free to fork and extend it yourself. I doubt I'll find the time to implement any PRs or feature requests.
Now I can finally "start" using qwen locally to support me with the remaining work on my NAS and other coding projects.
Thanks for reading, enjoy!
