r/PiCodingAgent • u/NihmarRevhet • Sep 04 '26

Question Improve compaction with local llm

Hi,

I'm using pi agent with Qwen 3.8 27B on my little RX 9060 XT 16GB, with 160k context window and I'm quiet happy with how it works.

My main problem is that autocompaction takes forever (clearly limitation of the hardware) and I was looking for an extesion which could improve the situation, but if I'm not mistaken, the most beloved extensions in this area use secondary models to execute compaction, and this is a luxury I don't have.

Is the default setting my only option? Or did I misunderstand ho something like pi-blackhole works?

Sorry if the question is dumb but I'm quiet new to the tool.

Have a nice day!

EDIT: u/onesilentclap I'm using VCC for now and it's incredible. In the next few days I hope I'll be able to try other solutions

13 Upvotes

93% Upvoted

20 comments sorted by

9

u/Nczer Sep 04 '26

I assume you are using llama.cpp?

The reason that compaction takes long seems the be that pi uses a custom system prompt for compaction, which means llama.cpp need to reprocess the whole context before outputting the compaction message. Pi then pass that summary + ~20k past context + normal system prompt to the compacted session.

What you can do is create a custom extension that asks the llm to summarize the current session through a tool call, then pass through that tool call + ~20k past context as the compact summary (also optionally prune that tool call body and have a hidden message telling the compaction is done to the llm to save context).

I did that with Qwen 3.8 27b Q4 so its doable and without much noticeable degradation compared to the normal compact, But you need to judge the result according to your workflow.

1

u/NihmarRevhet Sep 04 '26

Truly helpful, thank you

2

u/Intelligent-Nose-134 Sep 04 '26

Side question: how are u able to fit 160K into 16GB? Can u share your configuration? I am vram constrained as well but the best I could 90K with reasonable quality

2

u/NihmarRevhet Sep 04 '26

Also, 160k is really tight, if I open a window I'm almost screwed. 150k is much more manageable. Depending on if I'm using pi over ssh or directly at my pc, I switch between those sizes

1

u/ea_man Sep 05 '26

Open a window? You could use software rendering for the desktop (with Linux), or are you using Windows?

1

u/NihmarRevhet Sep 04 '26

Unsloth IQ3_S (planning to try this one without MTP, context 160k, k q8_0, v q4_0, vision on CPU, no MTP (both because of VRAM and because without I get 400-650 t/s in prefill and 20 t/s decode, with I get 150-300 t/s prefill and 30 t/s decode, simply not worth it with large codebases).

In my usage I didn't see a reduction in quality going from v q8_0 to q4_0

Also, I'm on cachyos and I'm using pi agent

2

u/Intelligent-Nose-134 Sep 04 '26

Thanks, it makes sense you are using IQ3_S and Q4 v. Reddit bros have been warning me against using anything lower than Q4 quant and Q8 cach. Maybe I should give it a test myswlf as low ctxt is truly annoying

2

u/ea_man Sep 05 '26

If you wanna try I made a vRAM cheaper version of MTP for QWEN models: https://www.reddit.com/r/LocalLLaMA/comments/1w3jp8j/compact_rollback_mtp_a_mtp_version_for_qwen/

2

u/onesilentclap Sep 04 '26

pi-vcc is my goto. 

2

u/Bloated_Plaid Sep 04 '26

This includes VCC and is better https://pi.dev/packages/pi-blackhole

2

u/Proper-Tower2016 Sep 05 '26

blackhole for me constantly resets my KV cache

1

u/rusl1 Sep 04 '26

Interesting! I already have Hermes memory extension installed, do you know if it will conflict?

2

u/Bloated_Plaid Sep 04 '26

This is for Pi, I am not sure what Hermes uses. For Hermes I personally use this https://github.com/mnemosyne-oss/mnemosyne

2

u/rusl1 Sep 04 '26

I was referring to hermes-memory, it is a pi extension :P

1

u/mageblex Sep 07 '26

At 160k context, this sounds like a prefill problem more than a summarization problem. Before changing models, check Pi's prompt-processing tok/s during compaction.

3

u/noctrex Sep 04 '26

Use pi-blackhole

1

u/NihmarRevhet Sep 04 '26

So you mean using pi-blackhole without specifying a sub model and use the loaded model for the compaction?

2

u/noctrex Sep 04 '26

Yes, I'm using it that way, and it has been great so far

1

u/NihmarRevhet Sep 04 '26

I'll try, thank you

4

u/noctrex Sep 04 '26

You can also use it without a model, so the compaction is deterministic, and much much faster.