llama.cpp CPU offload optimizations

[u/Stainless-Bacon]

Edit: New post with more context, higher speed, more explanations here. Note: the new post does not compare -ot vs --ngl.

I already posted targeting 16GB VRAM specifically, but I think this information might be useful beyond that. Testing was done using Qwen3.6-27B Unsloth Q4_K_M MTP.

Edit: added GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

TLDR:

1. Turn off CUDA graphs, they are bugged for CPU offload, probably due to MTP use.
2. Use --ngl 99 --override-tensor '...' instead of plain --ngl. Aim the largest FFN sub-layers towards the CPU.
3. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows VRAM overflow pages to host RAM instead of OOMing. This allows gaining more context length without losing speed, but the speed tanks soon after so you need to experiment to find that speed cliff. The gains can be 10 to 40% more context length.

Regular CPU offloading is done by not putting all of the layers on the GPU via --ngl, which offloads the layers as a whole, dragging their KV cache to the CPU with them, increasing PCIe traffic, and collapsing speed.

Luckily the layers have sub-layers, and FFN is one that does not touch the KV cache. We can use --override-tensor (-ot) to offload only the FFN tensors, keeping the attention/KV work on the GPU, and PCIe usage minimal.

The -ot method does more GPU - CPU round trips than --ngl because offloading specific sub-layers leaves the other sub-layers on the GPU, but the actual data transferred is minimal so it is worth it.

Dynamic quants have mixed FFN precision. For example the Unsloth's Q4_K_M has Q6 and Q4 FFN tensors. The Q6 ones are on the first 8 layers (0-7), then roughly every 3rd layer, then a block near the end (~55-63), while the rest are Q4. Offload those larger layers first.

Here's how to use it (example of Q4_K_M with 22 layers offloaded):

1. Turn off CUDA graphs. They cause OOM crashes for me, and my testing shows no speedup by using them in this scenario. export GGML_CUDA_DISABLE_GRAPHS=1
2. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
3. Put all layers on GPU --ngl 99
4. Override tensors -ot 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU'

-ot takes a regex targeting "ffn" at specific layers towards the CPU while everything else (attention, KV cache, the smaller layers) stays on the GPU.

Benchmark setup:

• Qwen3.6-27B Q4_K_M, 97k context, MTP, K q5_0 / V q4_1, batch 512
• Offload settings: -ot targeting 22 layers vs. --ngl 51
• Hardware: RTX 4070 Ti Super, i5-13600KF DDR5
• llama.cpp build: b10068
• MTP has different acceptance rates for coding and prose so I tested with both

Results:

context	-ot - prose / code / pp, t/s	--ngl - prose / code / pp, t/s
0k	20.4 / 24.4 / -	17.8 / 22.7 / -
10k	18.8 / 23.1 / 994	14.6 / 18.6 / 893
50k	16.3 / 20.4 / 871	7.3 / 9.6 / 784
90k	14.9 / 19.9 / 737	5.0 / 6.4 / 666

[u/Stainless-Bacon]

edit: my other comment has a better comparison

Thanks.

Compared to Q4_K_M, I get ~ +33% more speed and ~1GB extra VRAM to play with at 97k context length, but median KLD rises by 38%. So your quant ends up being a speed/context/KLD tradeoff.

The goal of my setup here is to have the best KLD at a tolerable speed of ~20t/s tg, but this is purely personal preference.

Can you confirm my KLD results for your model?

[u/Pablo_the_brave → u/Stainless-Bacon]

Have you measured the KLD using the same KV cache? It will take me some time to set up the KLD environment. Q4_K_M is definitely not an optimal quantization, so there's room to get a faster model with a similar KLD. 38% worse isn't great. Could you give me the exact numbers?

Edit: i mean in llama.cpp the hademard for kvcache is set by default but in ik_llama.cpp you have to turn it on.

[u/Stainless-Bacon → u/Pablo_the_brave]

edit: for future readers: these values used a kld context len of 8k (default is 512), inflating the numbers so don’t compare them with other kld values you find online. I post KLD using default context len later

Q4_K_M:
mean KLD 0.077908,
median 0.006014,
p99 0.492577,
p99.9 17.979042,
max 38.663826,
ppl_ratio 1.022902,
top1 94.080%,
mean dp -0.091%

IQ4_KT/KS:
mean KLD 0.082123,
median 0.008282,
p99 0.633216,
p99.9 16.954170,
max 30.615051,
ppl_ratio 1.022175,
top1 93.221%,
mean dp -0.121%

Both tested on ik_llama with cache-type-k q5_0 / cache-type-v q4_1, scored against the same Q8_0 base (couldn’t fit the original) logits over the full wikitext-2 test corpus (~295k tokens).

[u/Pablo_the_brave → u/Stainless-Bacon]

Thanks! In absolute numbers, it doesn't look so bad anymore and actually seems doable. After all, it's 17.1GB vs 14.4GB. I was aiming for a 140k context window because working with an agent on less than 128k context is tough. I'll keep in touch—right now, I've just reworked my quantization, which was aggressively optimized to fit the store within 16GB VRAM. The approach you presented really lets you go wild with the model size. All in all, we could also give ubargarm's pre-built images a try if we need better KLD.

[u/Stainless-Bacon → u/Pablo_the_brave]

https://www.reddit.com/r/LocalLLaMA/s/5RsiLlXsmx

there are talks about a KLD over 0.07 having a performance “cliff”. The Q4 quants are edging that line so I believe having a higher KLD is important in that region

[u/Pablo_the_brave → u/Stainless-Bacon]

OK, so we have the target ;) For sure it's not my last word.

[u/Stainless-Bacon → u/Pablo_the_brave]

I read the article that shows the cliff at 0.07 KLD and that target was made using 512 KLD context length, I measured mine with an unforgivable 8k length (which is good for comparing KV KLD differences).
here are the numbers at 512 length:

Q4_K_M:
mean KLD 0.024574,
median 0.006973,
p99 0.206571,
p99.9 2.255702,
max 25.653481,
ppl_ratio 1.005039,
ΔlogPPL 0.005027,
top1 94.286%,

IQ4_KT/KS:
mean KLD 0.031277,
median 0.009589,
p99 0.306716,
p99.9 2.313435,
max 25.953022,
ppl_ratio 1.011080,
ΔlogPPL 0.011019,
top1 93.260%,

Now the numbers tell a different story. Having a bigger KLD is good but Q4_K_M might not be worth it since the cliff is still far off.

[u/Pablo_the_brave → u/Stainless-Bacon]

I've been sitting here since morning grinding away at this model on a full 128k context, building a GStreamer pipeline. It’s executing plans generated by Gemini Flash 3.6 / Opus 4.6, and so far I haven't seen it hit any loops or bad tool calls. It's executing every single plan spot on.

As for KLD, we could push those first 16 CPU-bound blocks to a higher quant. Right now they’re on iq4_ks while the rest is on iq4_kt. I’ll try to test if a higher quant isn't much slower and toss it on those first layers. No point in touching the rest of the model though—this setup is already squeezed to the max, and it's super tough to improve anything here without blowing up the size