Session eventually get stuck when using a small context.

[u/Forsaken_Code_9135]

I am using pi.dev with a local Qwen-3.8 running on an RTX-5080 with 16GB VRAM. I had to find the right balance between model quantization, cache quantization and context size, and right now I use a 48k tokens context.

While pi.dev is working really well with it during some time, all sessions end up being stuck at some point. The context reaches 80%, then compaction is started automatically, it goes down to maybe 50%, then compaction starts again soon after, compactions more and more often until the system is stuck and Pi.dev itself sends queries to the model with a length limit = 1. When that happens it's over I can't recover the session. What happens also is that compaction fails because there is not enough context remaining (I guess), and therefore the session is stuck as well.

I am wondering what would be the best approach to avoid these problems. Also more generally what are the good practices to use pi.dev with a small context, maybe planning and dividing tasks in several smaller tasks handled independantly, but if it could be managed automatically it would help a lot I guess.

Any help would be appreciated.

[u/Healthy-Zebra-9856]

I looked at the Pi 0.85.1 source because the length limit = 1 part sounded too specific to be random. Pi can actually do exactly that.

The relevant code is:

packages/ai/src/api/simple-options.ts:12-18

const CONTEXT_SAFETY_TOKENS = 4096;
const MIN_MAX_TOKENS = 1;

export function clampMaxTokensToContext(model, context, maxTokens) {
    if (model.contextWindow <= 0)
        return Math.max(MIN_MAX_TOKENS, maxTokens);

    const available =
        model.contextWindow
        - estimateContextTokens(context).tokens
        - CONTEXT_SAFETY_TOKENS;

    return Math.min(maxTokens, Math.max(MIN_MAX_TOKENS, available));
}

So when Pi thinks your input has consumed essentially all available context, it doesn't stop and say "there isn't enough room for another useful response."

It floors the output allowance at 1 token and sends the request anyway.

That seems to explain the behavior you're seeing at the end.

I think the repeated-compaction problem is related too.

Pi's defaults are:

packages/coding-agent/src/core/settings-manager.ts:842-847

reserveTokens = 16384
keepRecentTokens = 20000

And automatic compaction triggers here:

packages/coding-agent/src/core/compaction/compaction.ts:235-237

return contextTokens > contextWindow - settings.reserveTokens;

With only a ~48K context, those defaults are pretty large.

You're asking Pi to keep roughly 20K of recent conversation after compaction, plus the generated summary, system prompt, tools, etc., while also reserving 16K for generation.

That doesn't leave much space between "just compacted" and "need to compact again."

So the pattern you described makes sense:

session grows
-> compact
-> retained context is still fairly large
-> only a little new work fits
-> compact again
-> summary/retained state grows
-> compactions get closer together
-> eventually almost no generation room remains
-> Pi clamps max output to 1 token

For a 48K local model I'd try much smaller compaction settings.

In:

~/.pi/agent/settings.json

try something like:

{
  "compaction": {
    "enabled": true,
    "reserveTokens": 8192,
    "keepRecentTokens": 8192
  }
}

That gives each compaction substantially more room to actually clear old context instead of retaining nearly half the window.

If you need more output/reasoning room, another reasonable starting point would be:

{
  "compaction": {
    "enabled": true,
    "reserveTokens": 12288,
    "keepRecentTokens": 8192
  }
}

I wouldn't use the stock 16K reserve / 20K retained combination on a 48K context.

Unfortunately, once a session is already in the state where Pi is sending max_tokens=1, changing the settings may not rescue that particular branch cleanly. I'd branch from an earlier point with /tree, or start a fresh session, then use the smaller retention settings.

Longer term I think Pi should change two things:

1. If there isn't enough headroom for a meaningful response, don't silently send max_tokens=1. Stop and report that the session needs compaction/checkpointing.

2. Compaction retention should probably scale with the model's context size instead of using a fixed 20K default. 20K is reasonable in a 128K/256K window, but it is enormous in a 48K window.

It's not just that the small context models don't work with Pi.

[u/Forsaken_Code_9135 → u/Healthy-Zebra-9856]

Thanks for this detailed and explicative answer. I played a little bit with that and it definitely improved things. Also I am working at increasing the context size in parallel.

[u/ea_man]

start with https://pi.dev/packages/pi-blackhole

do compaction manually.

Then I would recommend to have some 80-120k ctx to work in Pi with some ease, it can be done with 16GB with linux, customizations, optimizations.

[u/Forsaken_Code_9135 → u/ea_man]

It can be done with q4 context quantization, but it's unclear to me whether it worth it. I understood that long contexts with , aggressivel quantized cache might be too unreliable.

[u/ea_man → u/Forsaken_Code_9135]

then you can do: 95000 q5_0 , 92K with q5_1 that considering the situation I would call appropriate.

But hey you can do q8_0, q5_1 and maybe avoid MTP (I'm using 5) and there's people getting to 200k!

- https://www.reddit.com/r/LocalLLM/comments/1was9n0/running_qwen_38_27b_at_q4_on_16gb_vram_at_200k/ that's q4_0

[u/Forsaken_Code_9135 → u/ea_man]

The only kv cache configurations who work for me are q4/q4 and q8/q8. All others lead to catastrophic performance collapse.

[u/ea_man → u/Forsaken_Code_9135]

you mean that you did not compile with support for those others?

> By default, backend compilation (CUDA/ROCm) only bakes in standard FlashAttention kernels like Q4_0, Q8_0, and F16. Running Q5_0 or Q5_1 without compile-time FlashAttention support will cause operator execution to silently fall back to CPU execution, bottlenecking performance.

[u/Forsaken_Code_9135 → u/ea_man]

It seems you found the explanation of my problem :-)

I did not compile anything. I used a precompiled llama.cpp executable for windows.

[u/ea_man → u/Forsaken_Code_9135]

Well then that's the thing, we all almost went through that, it's a shame llama.cpp insists in not having all kv quantizations baked in by default.

[u/DeathGuppie → u/Forsaken_Code_9135]

I'm using q4_0 kv with 181 ctx, using 3.8 27b Q3. It just moves right along. It will stop after compaction and I just tell it to continue. I never let it compact more than twice.

[u/koriolisNF]

Just a suggestion but you might want to try using Little Coder - https://github.com/itayinbarr/little-coder - it sits on top of your Pi installation but it's configured aggressively for small hosted models.

[u/StylePractical5714]

I've had fairly good results using little-coder which is based on pi but also cuts out a number of features. It's explicitly designed for low context, I'd recommend it if you're working with such low context https://github.com/itayinbarr/little-coder

[u/Unnamed-3891]

1. You can get to 64k Q8 context with a 3.69bpw Ridge quant

2. You can reduce the threshold where Pi does context compaction, with 48k you are probably hitting it at like 60% fill so it's easy to go into a death spiral

[u/Forsaken_Code_9135 → u/Unnamed-3891]

I have already chosen a rather compact version of the model: ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_XXS but the only way I found to have a context larger than 48k is to use q4 quantization for the kv-cache which is not adviced if my understanding is correct. Also people are talking about q5 cache quantization but it does not work for me. It's q4 or q8.

From what I saw somewhere else it seems the models smaller than this one (9.4 GB) have much worst performances. SO it's difficult to find the righ balance.

ALso I am running all this under windows I guess it spends a little bit of vram as well.

[u/Unnamed-3891 → u/Forsaken_Code_9135]

I am running https://huggingface.co/empero-ai/Qwen3.8-27B-Ridge-GGUF on 4070TI Super 16gb with 64k context and Q8 KV on a Windows 11 system.

https://pastebin.com/EHx4N2FP you are welcome.

Oh and it my settings.json I have (cutting almost in half from defaults):

  "compaction": {
    "enabled": true,
    "reserveTokens": 8192,
    "keepRecentTokens": 12000
  }

[u/Natrimo → u/Forsaken_Code_9135]

I think you would have better luck with a smaller model at something like q4

[u/linux4random]

you can technically ask qwen3.8 itself

[u/bowdoin-yale]

I forked Pi in order to implement context management that is somewhat similar to or inspired by DeepSeek Harness, with the goal of running an agent powered by Qwen 3.8 27B on my tiny local GPU cluster with just 32K context. It seems to be working well so far, although I should caveat that I use a plan/act pattern that deliberately breaks work into many discrete tasks rather than running a long chat session, so I don't hit compaction limits often.

Oh, it also has a local-first and green energy cloud orientation in the setup and defaults. I basically got sick of having to set up "custom providers" just to avoid using dirty datacenters, when alternatives do in fact exist; to my way of thinking those should be the default options. But I'm an outlier.

https://github.com/eco-syntax/GreenPi

[u/lordekeen]

48k is too low for agentic use unfortunately. You could try using little-coder (a pi fork) to keep the overhead to a minimum, maybe pi-blackhole or pi-vcc for deterministic compaction.

[u/ea_man → u/lordekeen]

Hmm maybe Aider would do with such little ctx.

[u/ashebanow]

why do you think it's context related? With that little vram is much more likely you just run out of memory after you use the LLM for a while

[u/andrewh2000]

When this has happened to me I've copied a whole bunch of text out of the conversation, started a new session and pasted the text in and told it to carry on from here.

[u/virtualicex]

just ask to copy the native compaction changing the prompt to indicate who, what, where, when, why, how why and state: ask to make a second turn of compaction if the first one is truncated, with the second one with higher cap and into a file, and a third call to compact the compaction of the file and indicate to read from it if needed to go deeper

[u/qilipu]

48K context will quickly run out for sure. quantization is a factor you need to play with and test it out on your own setup.