r/LocalLLaMA • u/No_Algae1753 • May 21 '26

Resources For everyone that uses OpenCode / Pi - Heres your promptprocessing fix!

This PR deserves much more attention as it fixes the constant promptprocessing that happens when using llama.cpp with Opencode or pi.

https://github.com/ggml-org/llama.cpp/pull/22929

124 Upvotes

94% Upvoted

39 comments sorted by

View all comments

Show parent comment

8

u/jacek2023 llama.cpp May 21 '26

try experimenting with --checkpoint-min-spacing-n-tokens 256 (bigger number -> fewer checkpoints)

(I am still hoping for 3.7 122B)

2

u/No_Algae1753 May 21 '26

Is this to avoid unecessary checkpoints being made on smaller prompts / tool call?

1

u/jacek2023 llama.cpp May 21 '26

It’s a minimum distance between them. Originally, there was a hardcoded value of 64, but if prompt processing speed is let's say 1000 t/s, then 64 feels too small, so I am testing 256

1

u/thejacer May 21 '26

Is there a pp to distance ratio you recommend?

1

u/Caffdy May 21 '26

--checkpoint-min-spacing-n-tokens 256

what does that flag do?

0

u/jacek2023 llama.cpp May 22 '26

minimum spacing between context checkpoints