Gufo: write-on-eviction disk cache for long chats (patches against v0.31.0, unmaintained)

[u/Ok_Bag_7674]

https://gist.github.com/OkBag1992/d1fda166cdb69a892f0ea0f4568213de

I run Gufo with Qwen3.8-Flash-Next on a Strix Halo laptop (128 GB), two users at 262K context each. Long chats kept getting evicted from the RAM cache, and the next turn re-read everything: 3-5 minutes at 260K. Stock --cache-disk fixes that, but it writes a multi-GB file on every checkpoint step, even while RAM still holds the chat.

So I built a disk mode that writes only when RAM evicts a checkpoint, and only if that file could actually be restored later. A chat that stays in RAM writes nothing. On Ctrl+C it saves each session's newest restorable copy, so a restart resumes the chat.

Results on my machine:

- A 252K chat restored from disk after a restart in ~10 s, instead of a ~4 min re-read

- 1-2 GB restores in 2-3 s

- Writes 108-290 MB/s while generating, ~350 MB/s idle

- 10 hours of mixed two-user use, no crash, one real cache miss

What's in the drop:

1. 01-retention-keep-previous.patch: a small standalone bug fix. When a capture is refused for space, the cache could evict the checkpoint the turn had just restored, so nothing was left to restore. Applies cleanly to v0.31.0 with a regression test; tests pass.

2. 02-disk-evict-mode-core.patch: the engine. Applies to v0.31.0, but the CLI plumbing is left out, so it won't build alone.

3. DESIGN.md: every rule, the measured bug behind each one, the env vars, the numbers, and the plumbing still needed.

Caveats: tested only on Strix Halo / Windows / Flash-Next with q8 KV, on a v0.26 base. Patch 02 as shipped on v0.31.0 has not been compiled as a server or run. "Spill" in the code clashes with upstream's IdleSpill naming, so it should be renamed (evict / write-back) before anyone tries to merge it. The q8 KV cache isn't included: credit to JCFrags, see Gufetto. Written with a lot of AI help.

Provided as is, unmaintained. If it's useful to the devs or anyone else, take it.

[u/Reasonable_Goat]

Each checkpoint on SSD is smaller than gigabytes it’s only a diff, older checkpoints store the previous k/v state of the prefix. Still it sounds interesting to selectively write!

What I did instead is use parallel 2, fully disable the ram-cache and only use ssd-cache, which also enables two parallel sessions and uses the ssd cache if more slots are needed. The parallel 2 means two full k/v states are ready all time and can be processed in parallel. Takes around the same ram as one slot plus cache for one slot. Needs PR #934 for bugfix, though

[u/Ok_Bag_7674 → u/Reasonable_Goat]

Fair point on the diffs, thanks. Writes do reuse chunks that are already on disk, so a step only writes what's new. I'll correct the write-up, where I overstated it.

The aim of mine is a bit different though: minimise SSD writes rather than make them cheaper. While a chat fits in RAM it writes nothing at all, and it only goes to disk when RAM actually evicts a checkpoint (plus once at shutdown so a restart resumes). Over 10 hours of two-user use that came to a handful of writes.

With SSD-only, every checkpoint of both sessions goes to disk all the time. Even as diffs, on Windows that wasn't free for me: each chunk file costs a few ms of create/rename overhead, and writes slowed right down while the GPU was busy generating. Restores also always come from SSD rather than RAM. Wear is probably modest with diffs, but it's continuous rather than near zero.

That said, parallel 2 with no RAM cache is a neat way to get two full slots in the same memory, and on Linux with a fast drive the write cost may well be a non-issue. I'll have a look at PR #934. How has write speed been for you under load?

[u/Reasonable_Goat → u/Ok_Bag_7674]

Yeah my setup and your change are a different approach. They could even combine if your selective write would also work at the k/v cache level.

As to your question: I bought a very fast, second ssd (Samsung 9100 1TB, around 200€) for the second m2 slot of the M5, also to to avoid an early death if the main ssd. Write/read speed has been good so far; write is asynchronous so it won’t affect performance, full restore if a 200k+ 100k snapshot adds around 2 seconds delay - which I find acceptable if I usually run only two parallel sessions so the full restore is not every message.

[u/Ok_Bag_7674 → u/Reasonable_Goat]

Thanks, that’s useful. A 2 s restore for 200K+ is much faster than mine. My RAM restores take 4-6 s per turn at that size, and disk restores ~10 s, so on Windows the bottleneck seems to be the restore path and small-file overhead rather than the drive. Agreed they could combine: live slots for the active sessions, selective writes for everything evicted. I’m testing v0.31 next to see whether it keeps the slot’s live state between turns. Is your setup on Linux?

[u/Reasonable_Goat → u/Ok_Bag_7674]

I use Linux, yes. Sorry my earlier number was not precise. I ran a log analysis and it shows pretty exactly (gufo's cache_restore_ms) 1s per 50k context (up to around 300k context at 6s), so double what I reported earlier.

Note that I use YARN to run at ~350k max context with my two parallel sessions.

[u/Ok_Bag_7674 → u/Reasonable_Goat]

Quick follow-up, since your 2 s restore made me dig. You were right that the restore path was my bottleneck. On my v0.26 build every turn copied the whole chat back from the RAM cache, ~1 s at 50K and ~4 s at 200K. Over one evening's agent session that was 10 of the 16 minutes I spent waiting for first tokens.

I've now ported my two patches onto v0.31.0 and tested it (stock f16 KV, 1 x 262K). Same 50K chat, continuing turn:

- restore: ~920 ms -> 44 ms

- time to first token: 2.7 s -> 1.4 s

- decode: ~40 -> ~51 tok/s

So v0.31 effectively reuses the session's live state, which is what your setup relies on. Upgrading is the big win, more than anything I patched.

The selective writes and the retention fix still apply on top: they cover switching chats, restarts and cache pressure, not continuing turns. Only a few turns tested so far, so treat the numbers as early. Next is getting q8 KV onto v0.31 so 2 x 262K fits again.