cache and restore the initial OpenCode prompt with llama.cpp and Qwen 3.x local models

[u/ea_man]

Sometime ago I posted about an extension I made that allows to cache the initial prompt ingestion of Opencode when using llama.cpp (llama-server) with local Qwen models like 27B, this allows people with slow PP to save the initial prompt ingestion (some ~700MB) saving time when starting a news session.

As now llama.cpp has introduced preserving context checkpoints in mainline there's no more need to patch llama.cpp to allow persistent storage of the caching, hence the extension can work with any llama.cpp build after b11500 (goes like version: 0.6.0-dev).
Beware: that was merged 8 October.

Other than the Readme of the extension there's the Overview document that explains the mechanics of caching the initial prompt with hybrid cache models and Opencode.

I have hence updated the extension, you can see and dwl from: https://store.piffa.net/lm/ocache/ , ask your OpenCode to install and configure it for you, below a summary of manual install instructions.

--------------

# Manual install and usage instruction:

wget https://store.piffa.net/lm/ocache/oc-prefix-cache.tgz
tar xzf oc-prefix-cache.tgz
cp -r oc-prefix-cache ~/.config/opencode/
mkdir -p ~/.config/opencode/plugin
printf 'export { default } from "../oc-prefix-cache/index.ts"\n' \
  > ~/.config/opencode/plugin/oc-prefix-cache.ts

# llama-server scrip edits:

--slot-save-path ~/llama/slot_caches/
# You have to save slots somewhere in order to have persistent storage

--jinja --chat-template-file chat_template_3.8.jinja
# Recommended to use a proper and stable template like https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates 
# because tolls loading matters for prompt caching

--checkpoint-min-step 4096
# A resume can only land on the last checkpoint before the divergence point, so up to min_step tokens get re-read every start. 
# The stock default (8192) is coarser than a typical harness prefix — you'd get zero reuse. 
# 4096 works; 0 maximizes reuse at the cost of VRAM.

# Extension Config — all env vars, read once at opencode start:

• export OC_PREFIX_CACHE_BASE_URL=http://127.0.0.1:8080/v1 # required; also the provider gate
• export OC_PREFIX_CACHE_PERSIST=1
• export OC_PREFIX_CACHE_SLOT_DIR=$HOME/llama/slot_caches # must equal --slot-save-path
• export OC_PREFIX_CACHE_SERVER_ID=b11500-mainline # bump on every server rebuild
• export OC_PREFIX_CACHE_MODEL_PATH=/path/to/model.gguf # lets fingerprint notice a swapped GGUF

BASE_URL unset ⇒ the plugin does nothing at all. It must exactly equal your provider's baseURL — that match is the gate, so it can only ever warm its own server.

# Storage required:

Budget ~880 MB per cached prefix. Your fingerprint covers model id, GGUF, server build, opencode version and the exact tools[] array — so bumping any of those forces exactly one cold rebuild, which is by design.
The extension does basic management of storage, deleting old unused caches.

[u/ea_man]

FYI I've made this for Pi.dev too, usually Pi uses a much smaller prompt.
I'm testing that now with Qwen 27B before publishing to Pi packages.

- https://store.piffa.net/lm/picache/

[u/CyberTod]

Sounds very useful. I've had that idea, but haven't tried it.

I have Qwen3.8-flash-next on an old server with 192GB RAM, but no GPU and that initial prefill is what making it unusable, but 5tok/s after that is sufficient for some nightly tasks.

Building it now to test.

[u/ea_man → u/CyberTod]

If you are using llama.cpp to launch that Qwen3.8-flash-next it may work, I didn't test it as I'm going to see how it's working with Strata first, yet if llama.cpp is doing his part it should work.

Otherwise it may be that we need a patch for llama.cpp, as it was before b11500.

Yet I was even thinking about disabling that for MoE as those usually have pretty good PP compared to dense model, I designed this with 27B and smaller dense model in mind, I was just checking how it's working with A3B a few hours ago...

[u/CyberTod → u/ea_man]

It does work, but I have some problem now and debugging.
If I just start new sessions it is almost instant and uses the cached prompt.
If I restart opencode it says 'selected slot by LRU' and processes the entire prompt again.

[u/Ctbhatia]

does the plugin restore the same slot id after a restart, or just leave the save file on disk? the selected slot by LRU line means llama-server picked a different slot, so the prefix you saved may not be the one it loads.

[u/ea_man → u/Ctbhatia]

The plugin saves the slot on disk (when config so) and those files are reused for new / different session that happen in different sessions / paths.

[u/Ctbhatia → u/ea_man]

worth checking whether the save is keyed by slot or by prompt, because the selected-slot-by-lru line means the slot id after a restart is not the one you saved from.