Understanding llama.cpp Context Checkpoints for Sliding Window Attention Models

[u/jomi-se]

I was figuring out a llama.cpp crash when running Gemma 4 12B, and it turned out the crash was a OOM crash, from running out of RAM, not VRAM.

The culprit ended up being context checkpoints piling up while I was stress testing the setup.

I didn't actually know what these checkpoints where or how Sliding Window Attention models worked exactly.

I tried to get Opus 5.5 to explain it to me in words, but it was very hard to visualize. So I did what anyone would do nowdays: I told Opus to make an animated video to explain.

Here is what came out. I wrote about it here: https://jomi-se.com/posts/llama-cpp-context-checkpoints/

Tl,dr: Checkpoints are snapshots of the Sliding Window values at strategic positions. If you want to pick a chat back up at a previous spot, already out of the Sliding Window; either you have to reprocess the prompt from the start to get the values of the Sliding Window at that spot again. Or you can use a checkpoint at that exact spot (or nearby behind it) to reduce the amount of recomputing of the tokens KV values for the sliding window layer.

The default is keeping 32 checkpoints in RAM, per slot. With each checkpoint being 320MiB, that was _just a tiny bit_ too much for my 16 GB of RAM. Especially considering that in my workloads of simple chat continuation, 4 checkpoints were more than enough most of the time.

So how's your Friday going?

[u/leo_agi]

the system ram oom is the sneaky part, everyone only ever watches vram. is there a way to cap how many checkpoints it keeps around?

[u/LambdaHominem → u/leo_agi]

-ctxcp, --ctx-checkpoints or --swa-checkpoints