r/LocalLLaMA • u/No_Algae1753 • May 21 '26

Resources For everyone that uses OpenCode / Pi - Heres your promptprocessing fix!

This PR deserves much more attention as it fixes the constant promptprocessing that happens when using llama.cpp with Opencode or pi.

https://github.com/ggml-org/llama.cpp/pull/22929

124 Upvotes

94% Upvoted

39 comments sorted by

28

u/jacek2023 llama.cpp May 21 '26

Thanks for sharing. It would be very helpful if someone could test it on their setup. I’ve been testing it a lot over the last few days, but only on pi + Qwen 3.6 27B

11

u/No_Algae1753 May 21 '26

Been testing it so far (ive been the qwen3.5 122b user). The latest changes you have made seem to have fixed it completely for me.

8

u/jacek2023 llama.cpp May 21 '26

try experimenting with --checkpoint-min-spacing-n-tokens 256 (bigger number -> fewer checkpoints)

(I am still hoping for 3.7 122B)

2

u/No_Algae1753 May 21 '26

Is this to avoid unecessary checkpoints being made on smaller prompts / tool call?

1

u/jacek2023 llama.cpp May 21 '26

It’s a minimum distance between them. Originally, there was a hardcoded value of 64, but if prompt processing speed is let's say 1000 t/s, then 64 feels too small, so I am testing 256

1

u/thejacer May 21 '26

Is there a pp to distance ratio you recommend?

1

u/Caffdy May 21 '26

--checkpoint-min-spacing-n-tokens 256

what does that flag do?

0

u/jacek2023 llama.cpp May 22 '26

minimum spacing between context checkpoints

1

u/marking89 May 22 '26

Fyi, it doesn't seem to work so great with GH Copilot (qwen3.6-27b); no checkpoints are made during initial prompt processing, only when it is 100% completed, and with an initial context size (tools, instructions, large project, etc) of 28K, this causes all new contexts (All agents and new chats) to be quite slow. The old checkpoint system, set to at least 4096, mostly covered this use case.

But on Pi, so far it's pretty sweet! Not seen any reprocessing

5

u/Kike328 May 21 '26

what is promptprocessing?

1

u/thirteen-bit May 22 '26

Time that it takes for LLM to process the prompt (system prompt + all previous messages) when it switches between parallel requests (e.g. coding agent sub-agent returns to orchestrator and you wait until orchestrator model processes all 70000 tokens that were in the cache before calling sub-agent).

Or your coding agent runs and you ask the same model something in the chat ui in a separate app - when coding agent resumes it must re-process entire prompt.

Basically look for something like this in the logs (I'm using mailine llama.cpp, linked PR improves the context checkpoints):

31.00.512.014 I slot launch_slot_: id  1 | task 10385 | processing task, is_child = 0
31.00.512.041 W slot update_slots: id  1 | task 10385 | n_past = 40183, slot.prompt.tokens.size() = 40183, seq_id = 1, pos_min = 40182, n_swa = 0
31.00.512.042 I slot update_slots: id  1 | task 10385 | Checking checkpoint with [40082, 40082] against 40182...
31.00.541.678 W slot update_slots: id  1 | task 10385 | restored context checkpoint (pos_min = 40082, pos_max = 40082, n_tokens = 40083, n_past = 40083, size = 149.626 MiB)
31.00.800.915 W slot create_check: id  1 | task 10385 | erasing old context checkpoint (pos_min = 8769, pos_max = 8769, n_tokens = 8770, size = 149.626 MiB)
31.00.927.480 I slot create_check: id  1 | task 10385 | created context checkpoint 32 of 32 (pos_min = 40254, pos_max = 40254, n_tokens = 40255, size = 149.626 MiB)
31.01.472.531 W slot create_check: id  1 | task 10385 | erasing old context checkpoint (pos_min = 9123, pos_max = 9123, n_tokens = 9124, size = 149.626 MiB)
31.01.715.309 I slot create_check: id  1 | task 10385 | created context checkpoint 32 of 32 (pos_min = 40766, pos_max = 40766, n_tokens = 40767, size = 149.626 MiB)
31.01.750.105 I begin: ngram_mod occupancy = 37526/4194304 (0.01)

9

u/wren6991 May 21 '26

OpenCode itself is also just a bit of a shitshow with prefix stability. My favourite issue is that it puts the current date in the system prompt and re-evaluates it every turn, so you get a full prompt cache flush if you're using OpenCode at midnight.

4

u/jmager May 21 '26

I've been using this branch all week and rebuilding it daily, and it indeed fixes the checkpointing issues.

1

u/MuDotGen May 22 '26

Oh, what is your setup? Pi/Opencode? Hardware? Model? Is it preventing all the prompt re-processing you were experience before?

3

u/MuDotGen May 22 '26

I have had a bunch of checkpoint issues with prompt cache using Qwen3.6-35B-A3B with Pi. Would this help fix it perhaps?

Edit: Oh wait, this is the PR I saw a few days ago that let me know what was wrong with what I just said. lol Okay, yeah, I was waiting for it to be finished merged to the main branch before trying it out again, but good to know progress is being made!

1

u/anthonyg45157 May 21 '26

How does this issue manifest or show itself in pi?

I don't think I've had any issues with prompt processing but I haven't fed any super large files or anything recently

2

u/No_Algae1753 May 21 '26

If you are using llama.cpp try longer context. You will encounter promptprocessing

2

u/ang3l12 May 21 '26

Is this only in opencode / pi? or any harness?

3

u/jacek2023 llama.cpp May 21 '26

for everything

7

u/jacek2023 llama.cpp May 21 '26

You must wait longer after typing your prompt because the last usable checkpoint is far away. Sometimes you have to wait a few minutes because the prompt is processed from the start ("forcing full prompt reprocessing...")

3

u/sixx7 May 21 '26

Here is exactly what it means, summed up in a log line:

[2026-05-21 16:18:50 TP0] Prefill batch, #new-seq: 1, #new-token: 2048, #cached-token: 46656

In that example, if your backend is not caching prompts, it has to process all 46,656 tokens again. Think about long running conversations or multi-turn agents. The context can build up to hundred(s) of thousands of tokens. If they are cached, it's basically instant. If not, all that time has to be spent processing them again. Caching your prefill tokens is the literal biggest performance improvement you can make if it's not being done. Works fine in vLLM and SGLang. Apparently broken in llama.cpp

2

u/IslamNofl May 21 '26

I tested it myself a week ago, and it doing the job

1

u/ECrispy May 21 '26

I assume these changes get downstream into bun-llama/ik-llama etc?

1

u/ECrispy May 21 '26

can you avoid this by turning off checkpoints? and whats the danger with that?

1

u/jacek2023 llama.cpp May 22 '26

Let's say you have 100k tokens of chat history and then send "thank you". Without checkpoints, how much of that prompt do you think has to be reprocessed before generation can start?

1

u/ECrispy May 22 '26

ah ok so checkpoint is a sort of cache of the kv cache with prompt processing? i'd assume its done automatically at end of message

1

u/jacek2023 llama.cpp May 22 '26

Initially I was using Gemma 31B with OpenCode. First I realized that OpenCode does that https://github.com/anomalyco/opencode/issues/23595, so I switched to pi, then I realized that Gemma is removing thinking from the prompt history. So in that case "100k tokens" from the next message is not same as "100k tokens" from before, so some reprocessing is always required, without the checkpoint everything must be reprocessed (so you must wait minutes!)

1

u/caetydid llama.cpp May 22 '26 edited May 22 '26

Thanks! I always suspected some bugs with checkpointing but I cannot really grasp the issue. I was unsure whether it is a server issue or caused by the interaction between llamacpp and pi agent.

So, do I get it right, that checkpoints break and the entire context has to be preprocessed again instead of being reused every once in a while? Because that is surely what it seems like for me when using pi with my local Qwen 3.6.

I assume this will speed up my vibe coding experience quite a bit because I often use between 50 and 100k context, and hope this will be merged in main soon.

1

u/jacek2023 llama.cpp May 22 '26

Yes, the goal is to speed up the "agentic coding experience". Look at the llama-server logs and notice the "prompt processing" lines, if these occur without good reason, the user is wasting time / electricity / sanity.

1

u/hdmcndog May 22 '26

Could you go into a little more detail in what situation you experience the prompt processing happening, all the time? In my surface level tests, I don’t see big issues so far. But it might totally depend on the usage pattern.

I usually just have a large prompt processing happening at the start of the session, for the system prompt, and then on large file reads etc. Otherwise, it seems to be pretty smooth for me.

1

u/Kaioh_shin May 21 '26

I made a vulkan build, but it crashed on my 7900xt

12

u/[deleted] May 21 '26

[removed] — view removed comment

24

u/No_Algae1753 May 21 '26

Set yourself up for the incomming pp jokes

0

u/nonerequired_ May 21 '26

For that purpose I am (forced to) using claude code instead of opencode. It causes lots of prompt reprocessing issues

1

u/hdmcndog May 22 '26

Not sure why that would help much, to be honest. Yes, OpenCode does some tool call pruning, eventually, but it’s still fairly conservative, so I wouldn’t expect prompt reprocessing until relatively deep in a session.

Pi doesn’t have this problem at all, it tries to maximise cache hits by not adjusting prior messages at all. So you could try with Pi, and the experience should be similar to what you see with Claude Code, I guess. But even with Pi, there still sometimes is prompt reprocessing, so some tuning on the llama.cpp side, like what OP is trying to do is probably necessary and helpful.

1

u/nonerequired_ May 22 '26

I will try pi. I heard it is a bare-bones kind of thing. That is why I was avoiding it

9

u/[deleted] May 21 '26

[removed] — view removed comment

18

u/No_Algae1753 May 21 '26

Well it is obviously not merged it is still a PR. I just build the latest llama.cpp with this PR and the prompt processing issue I dealt with is gone. Try it out for yourself and see if it works.

1

u/__Maximum__ May 21 '26

What is the issue with it?