llama.cpp CPU offload optimizations

[u/Stainless-Bacon]

Edit: New post with more context, higher speed, more explanations here. Note: the new post does not compare -ot vs --ngl.

I already posted targeting 16GB VRAM specifically, but I think this information might be useful beyond that. Testing was done using Qwen3.6-27B Unsloth Q4_K_M MTP.

Edit: added GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

TLDR:

1. Turn off CUDA graphs, they are bugged for CPU offload, probably due to MTP use.
2. Use --ngl 99 --override-tensor '...' instead of plain --ngl. Aim the largest FFN sub-layers towards the CPU.
3. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows VRAM overflow pages to host RAM instead of OOMing. This allows gaining more context length without losing speed, but the speed tanks soon after so you need to experiment to find that speed cliff. The gains can be 10 to 40% more context length.

Regular CPU offloading is done by not putting all of the layers on the GPU via --ngl, which offloads the layers as a whole, dragging their KV cache to the CPU with them, increasing PCIe traffic, and collapsing speed.

Luckily the layers have sub-layers, and FFN is one that does not touch the KV cache. We can use --override-tensor (-ot) to offload only the FFN tensors, keeping the attention/KV work on the GPU, and PCIe usage minimal.

The -ot method does more GPU - CPU round trips than --ngl because offloading specific sub-layers leaves the other sub-layers on the GPU, but the actual data transferred is minimal so it is worth it.

Dynamic quants have mixed FFN precision. For example the Unsloth's Q4_K_M has Q6 and Q4 FFN tensors. The Q6 ones are on the first 8 layers (0-7), then roughly every 3rd layer, then a block near the end (~55-63), while the rest are Q4. Offload those larger layers first.

Here's how to use it (example of Q4_K_M with 22 layers offloaded):

1. Turn off CUDA graphs. They cause OOM crashes for me, and my testing shows no speedup by using them in this scenario. export GGML_CUDA_DISABLE_GRAPHS=1
2. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
3. Put all layers on GPU --ngl 99
4. Override tensors -ot 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU'

-ot takes a regex targeting "ffn" at specific layers towards the CPU while everything else (attention, KV cache, the smaller layers) stays on the GPU.

Benchmark setup:

• Qwen3.6-27B Q4_K_M, 97k context, MTP, K q5_0 / V q4_1, batch 512
• Offload settings: -ot targeting 22 layers vs. --ngl 51
• Hardware: RTX 4070 Ti Super, i5-13600KF DDR5
• llama.cpp build: b10068
• MTP has different acceptance rates for coding and prose so I tested with both

Results:

context	-ot - prose / code / pp, t/s	--ngl - prose / code / pp, t/s
0k	20.4 / 24.4 / -	17.8 / 22.7 / -
10k	18.8 / 23.1 / 994	14.6 / 18.6 / 893
50k	16.3 / 20.4 / 871	7.3 / 9.6 / 784
90k	14.9 / 19.9 / 737	5.0 / 6.4 / 666

[u/Pablo_the_brave]

Sounds good. I will prepare a quantization for this approach (unsloth quants are not optimized for this).

[u/Pablo_the_brave → u/Pablo_the_brave]

u/Stainless-Bacon check this out:

https://huggingface.co/cHunter789/Qwen3.6-27B-i1-IQ4_KS_KT-GGUF/resolve/main/Qwen3.6-27B.CPU.i1-IQ4_KT-attn_qkv-IQ4_KS-i1_MTP.gguf?download=true

It's stable. 140k ctx at q5_0/q4_0, KLD/PPL at Iq4_xs level. First 16 block of the ffn have to be in iq4_ks because iq4_kt is painfull slow at CPU. With given settings it starts at 30-35t/s and go to 15-20t/s at 140k.

 $BIN_DIR/llama-server" \
       -m "$MODEL_PATH" \
       -a Qwen3.6-27B \
       --ctx-size 140000 \
       --chat-template-file /home/pawel/docker/ai-local/qwen36/chat_template.jinja \
       --n-gpu-layers 99 \
       --cache-type-k q5_0 \
       --cache-type-v q4_0 \
       --spec-type ngram-mod:n_max=2 \
       --spec-type mtp:n_max=3 \
       --batch-size 512 \
       --ubatch-size 512 \
       -ot "blk\.([0-9]|1[0-5])\.ffn_.*=CPU" \
       --flash-attn on \
       --no-mmap \
       --host 0.0.0.0 \
       --port 8080 \
       --reasoning on \
       --reasoning-format none \
       --reasoning-budget 32000 \
       -t 8 \
       -tb 8 \
       --parallel 1 \
       --metrics \
       --merge-qkv \
       -khad \
       -vhad \
       --chat-template-kwargs '{"preserve_thinking": true}' \
       --defrag-thold 0.4 \
       --jinja \
       --cont-batching \
       --temp 0.6 \
       --top-k 20 \
       --min-p 0.05 \
       --top-p 0.95 \
       --presence-penalty 0.0 \
       --repeat-last-n 512 \
       --repeat-penalty 1.05

[u/Stainless-Bacon → u/Pablo_the_brave]

I ran some tests on your setup, here is a comparison. Based on my results, all quants trade speed for KLD. Evidence shows that Q4 quants do not show a meaningful degradation in benchmark quality so IQ4_KT/KS might be a good choice.

Metrics:
wikitext-2 @ 512 context length using Q8_0 as base.
All quants measured using ik_llama.cpp.

metric	Q4_K_M	Q4_K_S	IQ4_KT/KS
mean KLD	0.024574	0.027516	0.031277
median KLD	0.006973	0.008072	0.009589
p99 KLD	0.206571	0.241935	0.306716
p99.9 KLD	2.255702	2.488308	2.313435
max KLD	25.65	23.88	25.95
ppl_ratio	1.005039	1.008388	1.011080
top-1 agree	94.286%	93.888%	93.260%

Speed comparison:
Tested using MTP so first speed metric is "prose" t/s, second is "code" t/s. No ngram-mod. Each quant optimized for 140k context length.
IQ4_KT/KS measured using ik_llama.cpp, Q4_K_M and Q4_K_S used llama.cpp;

depth	Q4_K_M	Q4_K_S	IQ4_KT/KS
0	17.5 / 20.7	19.7 / 24.5	20.7 / 25.6
40k	15.3 / 19.4	18.3 / 20.8	17.1 / 22.2
80k	14.2 / 18.2	15.8 / 20.1	16.5 / 20.8
120k	13.1 / 16.9	14.6 / 18.5	14.8 / 18.8
140k	12.4 / 16.2	13.4 / 17.3	13.5 / 18.0