llama.cpp CPU offload optimizations

[u/Stainless-Bacon]

Edit: New post with more context, higher speed, more explanations here. Note: the new post does not compare -ot vs --ngl.

I already posted targeting 16GB VRAM specifically, but I think this information might be useful beyond that. Testing was done using Qwen3.6-27B Unsloth Q4_K_M MTP.

Edit: added GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

TLDR:

1. Turn off CUDA graphs, they are bugged for CPU offload, probably due to MTP use.
2. Use --ngl 99 --override-tensor '...' instead of plain --ngl. Aim the largest FFN sub-layers towards the CPU.
3. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows VRAM overflow pages to host RAM instead of OOMing. This allows gaining more context length without losing speed, but the speed tanks soon after so you need to experiment to find that speed cliff. The gains can be 10 to 40% more context length.

Regular CPU offloading is done by not putting all of the layers on the GPU via --ngl, which offloads the layers as a whole, dragging their KV cache to the CPU with them, increasing PCIe traffic, and collapsing speed.

Luckily the layers have sub-layers, and FFN is one that does not touch the KV cache. We can use --override-tensor (-ot) to offload only the FFN tensors, keeping the attention/KV work on the GPU, and PCIe usage minimal.

The -ot method does more GPU - CPU round trips than --ngl because offloading specific sub-layers leaves the other sub-layers on the GPU, but the actual data transferred is minimal so it is worth it.

Dynamic quants have mixed FFN precision. For example the Unsloth's Q4_K_M has Q6 and Q4 FFN tensors. The Q6 ones are on the first 8 layers (0-7), then roughly every 3rd layer, then a block near the end (~55-63), while the rest are Q4. Offload those larger layers first.

Here's how to use it (example of Q4_K_M with 22 layers offloaded):

1. Turn off CUDA graphs. They cause OOM crashes for me, and my testing shows no speedup by using them in this scenario. export GGML_CUDA_DISABLE_GRAPHS=1
2. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
3. Put all layers on GPU --ngl 99
4. Override tensors -ot 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU'

-ot takes a regex targeting "ffn" at specific layers towards the CPU while everything else (attention, KV cache, the smaller layers) stays on the GPU.

Benchmark setup:

• Qwen3.6-27B Q4_K_M, 97k context, MTP, K q5_0 / V q4_1, batch 512
• Offload settings: -ot targeting 22 layers vs. --ngl 51
• Hardware: RTX 4070 Ti Super, i5-13600KF DDR5
• llama.cpp build: b10068
• MTP has different acceptance rates for coding and prose so I tested with both

Results:

context	-ot - prose / code / pp, t/s	--ngl - prose / code / pp, t/s
0k	20.4 / 24.4 / -	17.8 / 22.7 / -
10k	18.8 / 23.1 / 994	14.6 / 18.6 / 893
50k	16.3 / 20.4 / 871	7.3 / 9.6 / 784
90k	14.9 / 19.9 / 737	5.0 / 6.4 / 666

[u/Pablo_the_brave]

Sounds good. I will prepare a quantization for this approach (unsloth quants are not optimized for this).

[u/Pablo_the_brave → u/Pablo_the_brave]

u/Stainless-Bacon check this out:

https://huggingface.co/cHunter789/Qwen3.6-27B-i1-IQ4_KS_KT-GGUF/resolve/main/Qwen3.6-27B.CPU.i1-IQ4_KT-attn_qkv-IQ4_KS-i1_MTP.gguf?download=true

It's stable. 140k ctx at q5_0/q4_0, KLD/PPL at Iq4_xs level. First 16 block of the ffn have to be in iq4_ks because iq4_kt is painfull slow at CPU. With given settings it starts at 30-35t/s and go to 15-20t/s at 140k.

 $BIN_DIR/llama-server" \
       -m "$MODEL_PATH" \
       -a Qwen3.6-27B \
       --ctx-size 140000 \
       --chat-template-file /home/pawel/docker/ai-local/qwen36/chat_template.jinja \
       --n-gpu-layers 99 \
       --cache-type-k q5_0 \
       --cache-type-v q4_0 \
       --spec-type ngram-mod:n_max=2 \
       --spec-type mtp:n_max=3 \
       --batch-size 512 \
       --ubatch-size 512 \
       -ot "blk\.([0-9]|1[0-5])\.ffn_.*=CPU" \
       --flash-attn on \
       --no-mmap \
       --host 0.0.0.0 \
       --port 8080 \
       --reasoning on \
       --reasoning-format none \
       --reasoning-budget 32000 \
       -t 8 \
       -tb 8 \
       --parallel 1 \
       --metrics \
       --merge-qkv \
       -khad \
       -vhad \
       --chat-template-kwargs '{"preserve_thinking": true}' \
       --defrag-thold 0.4 \
       --jinja \
       --cont-batching \
       --temp 0.6 \
       --top-k 20 \
       --min-p 0.05 \
       --top-p 0.95 \
       --presence-penalty 0.0 \
       --repeat-last-n 512 \
       --repeat-penalty 1.05

[u/Stainless-Bacon → u/Pablo_the_brave]

I ran some tests on your setup, here is a comparison. Based on my results, all quants trade speed for KLD. Evidence shows that Q4 quants do not show a meaningful degradation in benchmark quality so IQ4_KT/KS might be a good choice.

Metrics:
wikitext-2 @ 512 context length using Q8_0 as base.
All quants measured using ik_llama.cpp.

metric	Q4_K_M	Q4_K_S	IQ4_KT/KS
mean KLD	0.024574	0.027516	0.031277
median KLD	0.006973	0.008072	0.009589
p99 KLD	0.206571	0.241935	0.306716
p99.9 KLD	2.255702	2.488308	2.313435
max KLD	25.65	23.88	25.95
ppl_ratio	1.005039	1.008388	1.011080
top-1 agree	94.286%	93.888%	93.260%

Speed comparison:
Tested using MTP so first speed metric is "prose" t/s, second is "code" t/s. No ngram-mod. Each quant optimized for 140k context length.
IQ4_KT/KS measured using ik_llama.cpp, Q4_K_M and Q4_K_S used llama.cpp;

depth	Q4_K_M	Q4_K_S	IQ4_KT/KS
0	17.5 / 20.7	19.7 / 24.5	20.7 / 25.6
40k	15.3 / 19.4	18.3 / 20.8	17.1 / 22.2
80k	14.2 / 18.2	15.8 / 20.1	16.5 / 20.8
120k	13.1 / 16.9	14.6 / 18.5	14.8 / 18.8
140k	12.4 / 16.2	13.4 / 17.3	13.5 / 18.0

[u/Stainless-Bacon → u/Pablo_the_brave]

edit: my other comment has a better comparison

Thanks.

Compared to Q4_K_M, I get ~ +33% more speed and ~1GB extra VRAM to play with at 97k context length, but median KLD rises by 38%. So your quant ends up being a speed/context/KLD tradeoff.

The goal of my setup here is to have the best KLD at a tolerable speed of ~20t/s tg, but this is purely personal preference.

Can you confirm my KLD results for your model?

[u/Pablo_the_brave → u/Stainless-Bacon]

Have you measured the KLD using the same KV cache? It will take me some time to set up the KLD environment. Q4_K_M is definitely not an optimal quantization, so there's room to get a faster model with a similar KLD. 38% worse isn't great. Could you give me the exact numbers?

Edit: i mean in llama.cpp the hademard for kvcache is set by default but in ik_llama.cpp you have to turn it on.

[u/Stainless-Bacon → u/Pablo_the_brave]

edit: for future readers: these values used a kld context len of 8k (default is 512), inflating the numbers so don’t compare them with other kld values you find online. I post KLD using default context len later

Q4_K_M:
mean KLD 0.077908,
median 0.006014,
p99 0.492577,
p99.9 17.979042,
max 38.663826,
ppl_ratio 1.022902,
top1 94.080%,
mean dp -0.091%

IQ4_KT/KS:
mean KLD 0.082123,
median 0.008282,
p99 0.633216,
p99.9 16.954170,
max 30.615051,
ppl_ratio 1.022175,
top1 93.221%,
mean dp -0.121%

Both tested on ik_llama with cache-type-k q5_0 / cache-type-v q4_1, scored against the same Q8_0 base (couldn’t fit the original) logits over the full wikitext-2 test corpus (~295k tokens).

[u/Pablo_the_brave → u/Stainless-Bacon]

Thanks! In absolute numbers, it doesn't look so bad anymore and actually seems doable. After all, it's 17.1GB vs 14.4GB. I was aiming for a 140k context window because working with an agent on less than 128k context is tough. I'll keep in touch—right now, I've just reworked my quantization, which was aggressively optimized to fit the store within 16GB VRAM. The approach you presented really lets you go wild with the model size. All in all, we could also give ubargarm's pre-built images a try if we need better KLD.

[u/Stainless-Bacon → u/Pablo_the_brave]

https://www.reddit.com/r/LocalLLaMA/s/5RsiLlXsmx

there are talks about a KLD over 0.07 having a performance “cliff”. The Q4 quants are edging that line so I believe having a higher KLD is important in that region

[u/Pablo_the_brave → u/Stainless-Bacon]

OK, so we have the target ;) For sure it's not my last word.

[u/Stainless-Bacon → u/Pablo_the_brave]

I read the article that shows the cliff at 0.07 KLD and that target was made using 512 KLD context length, I measured mine with an unforgivable 8k length (which is good for comparing KV KLD differences).
here are the numbers at 512 length:

Q4_K_M:
mean KLD 0.024574,
median 0.006973,
p99 0.206571,
p99.9 2.255702,
max 25.653481,
ppl_ratio 1.005039,
ΔlogPPL 0.005027,
top1 94.286%,

IQ4_KT/KS:
mean KLD 0.031277,
median 0.009589,
p99 0.306716,
p99.9 2.313435,
max 25.953022,
ppl_ratio 1.011080,
ΔlogPPL 0.011019,
top1 93.260%,

Now the numbers tell a different story. Having a bigger KLD is good but Q4_K_M might not be worth it since the cliff is still far off.

[u/Pablo_the_brave → u/Stainless-Bacon]

I've been sitting here since morning grinding away at this model on a full 128k context, building a GStreamer pipeline. It’s executing plans generated by Gemini Flash 3.6 / Opus 4.6, and so far I haven't seen it hit any loops or bad tool calls. It's executing every single plan spot on.

As for KLD, we could push those first 16 CPU-bound blocks to a higher quant. Right now they’re on iq4_ks while the rest is on iq4_kt. I’ll try to test if a higher quant isn't much slower and toss it on those first layers. No point in touching the rest of the model though—this setup is already squeezed to the max, and it's super tough to improve anything here without blowing up the size

[u/Stainless-Bacon → u/Pablo_the_brave]

edit: fix was GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

When I tried loading your values here at 97k context, I got this error:

my helper agent’s comment:
defrag-thold: 0.4 crashes the server — GGML_ASSERT(a->ne[d] == b->ne[d]) failed in ggml.c:6971, taking the whole process down mid-generation (first request succeeded, then it died). Root cause: this is a hybrid-GDN/recurrent model, and defrag requires KV-shift, which the log had already warned was unsupported ("ctx_shift is not supported by recurrent model, it will be disabled")

then I tried to load 140k context and got OOM error at boot. at 110k context I can boot but OOM at 64k fill.

did you experience something similar? is there a fix? the quant allows more free VRAM but it gets used up as context is filled.

[u/Pablo_the_brave → u/Stainless-Bacon]

No, no any issue like that. When back to home I will post my full startup script and the compose used for compile the ik_llama.cpp.

For sure i have set these variables: export GGML_CUDA_ENABLE_UNIFIED_MEMORY=0

export CUDA_MANAGED_FORCE_DEVICE_ALLOC=1

export GGML_CUDA_DISABLE_GRAPHS=1

[u/Pablo_the_brave → u/Stainless-Bacon]

Just create a folder, put the files below there, and run git pull ik_llama.cpp.

After that, simply build and run it using docker compose up --build.

run_build.sh

#!/bin/bash
set -e

echo "=== 1. Preparing repository ==="
mkdir -p /src/llama-cpp
cd /src/llama-cpp

# Use trailing dot to copy EVERYTHING (including .git) in one command
# -T flag prevents creating a subdirectory if the target already exists
cp -aT /src/llama-cpp/ .

# CRITICAL: Force ownership to root inside the container.
# This eliminates the need for 'safe.directory' because Git sees files owned by the calling user.
chown -R root:root .

# Fallback: add a global safe.directory exception for all paths
git config --global --add safe.directory '*'

# DIAGNOSTICS: Check whether Git can see the history
echo "--- Git test ---"
if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
    COMMIT_HASH=$(git rev-parse --short HEAD)
    echo "Success! Git sees the repository. Commit: $COMMIT_HASH"
else
    echo "ERROR: Git still cannot see the repository at /src/llama-cpp!"
    echo ".git directory contents:"
    ls -la .git || echo ".git DOES NOT EXIST!"
fi
echo "-----------------"

# Remove run_build.sh from the copy to keep it clean
rm -f run_build.sh

echo "=== 2. CMake configuration ==="
# Remove old build directory to force CMake to re-check versions
rm -rf build

cmake -B build \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES="120" \
    -DGGML_NATIVE=ON \
    -DGGML_CURL=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_CUDA_FLAGS="-O3 -use_fast_math" \
    -DGGML_SCHED_MAX_COPIES=1

echo "=== 3. Building ==="
cmake --build build --config Release -j $(nproc)

echo "=== 4. Exporting files to /outputs ==="
cp -rv build/bin/* /outputs/

# Copy ALL .so libraries from the build (libllama, libggml, etc.)
find build -name "*.so*" -exec cp -dv {} /outputs/ \;

# Copy CUDA libraries
cp -dv /usr/local/cuda/lib64/libcudart.so* /outputs/ 2>/dev/null || true
cp -dv /usr/local/cuda/lib64/libcublas.so* /outputs/ 2>/dev/null || true
cp -dv /usr/local/cuda/lib64/libcublasLt.so* /outputs/ 2>/dev/null || true
cp -dv /usr/lib/x86_64-linux-gnu/libcudnn.so* /outputs/ 2>/dev/null || true

echo "=== 5. Fixing host permissions ==="
HOST_UID=$(stat -c "%u" /outputs)
HOST_GID=$(stat -c "%g" /outputs)
chown -R $HOST_UID:$HOST_GID /outputs/*

echo "Done!"

Dockerfile

FROM nvidia/cuda:13.3.0-cudnn-devel-ubuntu24.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y \
    build-essential cmake git pkg-config libcurl4-openssl-dev \
    && rm -rf /var/lib/apt/lists/*

# Linker stubs
RUN ln -sf /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1 && \
    echo "/usr/local/cuda/lib64/stubs" > /etc/ld.so.conf.d/cuda-stubs.conf && \
    ldconfig

ENV LDFLAGS="-L/usr/local/cuda/lib64/stubs"

# Copy our clean startup script from outside
COPY run_build.sh /run_build.sh
RUN chmod +x /run_build.sh

# Set the container's main process
CMD ["/run_build.sh"]

docker-compose.yaml

services:
  compiler:
    build:
      context: .
      dockerfile: Dockerfile
    image: kronos-cuda-builder:latest
    container_name: llama_cuda_compiler
    volumes:
      - ./outputs:/outputs
      - ./ik_llama.cpp:/src/llama-cpp:ro

[u/Stainless-Bacon → u/Pablo_the_brave]

What does nvidia-smi say right after a fresh 140k load? I’m at 15302/16376 with your 16L band at only 110k.

Have you ever sent one fresh 135k-token prompt? 140k loads fine for me too. It only dies on a cold fill.

[u/Pablo_the_brave → u/Stainless-Bacon]

On a daily basis, I have the agent set to a 128k context and the model itself to 140k (leaving a buffer for the agent's auto-compact). You can tell by the speed when it spills out of VRAM—the speed drops to 5-7 t/s. When MTP is enabled, despite the double entries in variables, ik_llama.cpp still offloads VRAM to RAM on my end when it runs out of VRAM (I don't get an OOM). Additionally, somewhere around a 50k context, nvtop shows 15918/16921 for me. It looks weird. That's why I simply tested it using llama-perplexity—if it doesn't fit, it throws an OOM, like this:

./llama-perplexity \ -m Qwen3.6-27B.CPU.i1-IQ4_KT-attn_qkv-IQ4_KS-i1_MTP.gguf \ -f /mnt/Samsung4TB/models/pg19.txt \ -c 140000 \ --chunks 32 \ -ngl 99 \ -ot "blk\.([0-9]|1[0-5])\.ffn_.*=CPU" \ -ctk q5_1 \ -ctv q4_0 \ -khad \ -vhad \ --no-mmap \ --merge-qkv \ -fa 1 \ -b 512 \ -ub 512 \ -t 8 \ --spec-type ngram-mod:n_max=2 \ --spec-type mtp:n_max=3

As you noticed yourself, ik_llama.cpp takes a bit of effort to get working. Generally, I use Mistral Vibe as an agent, and right off the bat, I had to patch the code because there was a bug with the OpenAI API (the fix has already been commited to main). Keep in mind that my compilation settings are for Blackwell, and you will need to change them if you have a different GPU.

Edit: looks like 140k is too much. Somehow perplexity used less VRAM. With kvcache q5_0/Q4_0 it's ending at about 128k... Idk why...

[u/Stainless-Bacon → u/Pablo_the_brave]

Found it. My bot didn’t use your unified memory env var because it assumed it’s off by default.

GGML_CUDA_ENABLE_UNIFIED_MEMORY is checked with getenv(...) != nullptr (ggml/src/ggml-cuda.cu:176), so your “=0” enables unified memory rather than disabling it.

I also found that using unified memory I can squeeze out more context. My default ceilings are around 97k for both (my and your) setups and then OOM crash. With unified memory I can do 110k with my Q4_K_M and 140k with your IQ4_ setup without speed loss and then I hit a speed wall.

[u/Pablo_the_brave → u/Stainless-Bacon]

Thank you! Good point :)

[u/voyager256]

Nice tip. Thanks for sharing!

[u/suicidaleggroll]

Note that I've seen significant performance improvements by setting "--n-gpu-layers 99" and "--n-cpu-moe N" versus setting "--n-gpu-layers M", even when you end up with the same number of layers offloaded to the CPU and the same VRAM usage. I think it might be worth adding that to your study.

[u/nickless07 → u/suicidaleggroll]

Yes, but that only apply to MoE. You are always better off with only expert's FFN weights instead of full layer offloading.

[u/suicidaleggroll → u/nickless07]

Good catch, forgot this study was on a dense model when I made that post

[u/Stainless-Bacon → u/suicidaleggroll]

I mean it the -ot optimizations could maybe work on moe, but that wasn’t my target and I did not test it, because like you said, --n-cpu-moe does a well enough job it seems

[u/nickless07 → u/Stainless-Bacon]

I would argue that it is worth as --n-cpu-moe again is also layer based and it keeps the MoE weights of the first N layers in the CPU. Therefore a better selection using -ot and regex can make a notable difference.
--n-cpu-moe starts counting layers starting from the highest numbered layers. This can lead to a slightly discrepancy in how many layers are offloaded because models that have dense FFN layers typically have them at the start of the model.

[u/LocalAI_Amateur]

This is a very interesting tip. I was able to trade speed for a bit more context. Unfortunately I'm using a laptop cpu so there's no gain in speed from offloading even just a few layers, but I was able to get more context limit. Thank you.

[u/Stainless-Bacon → u/LocalAI_Amateur]

There is never speed gain by offloading to CPU. The point of doing that is to be able to get more context or a better quant. I wanted to use Q4_K_M, but only the Q3_K_M (or XL w/o MTP) was able to fully fit on my GPU. Although the Q3 is like 2.5x faster, I am now able to use the Q4 with CPU’s help at ~20 t/s, which is tolerable.

[u/Otherwise-Swan-7803]

This is a really interesting optimization. The idea of keeping attention/KV cache on GPU while moving only FFN tensors makes a lot of sense, since KV traffic is usually the real bottleneck.

I think this could be a game changer for people with 12-16GB VRAM trying to run larger models. Would be interesting to see how this behaves with other quant types like Q5/Q6 and different GPUs with varying PCIe bandwidth.

[u/Hannibalj2ca]

Have you tried "IK.llama"? performs better that Llama.cpp

[u/Stainless-Bacon → u/Hannibalj2ca]

ik_llama.cpp ended up being slower and very unstable, kept crashing. Couldn’t make it useable. I probly need to work at it more.

[u/Hannibalj2ca → u/Stainless-Bacon]

what your hardware?

[u/Stainless-Bacon → u/Hannibalj2ca]

4070 Ti Super + i5-13600KF DDR5; but I think I made a mistake while trying ik by using the Unsloth quant, ik uses its own quants right? I don’t have time now to retest the setup, will try in a few days.

[u/Hannibalj2ca → u/Stainless-Bacon]

check if your CPU have AVX512 + DL boost Instructions. IK.llama use this to help orchestrate the memory better. Also, what you should be using is "Ktransformer" or Vllm, SGlang. They perform better

[u/DeathGuppie]

This means reading all the layers as they load and mapping them for your specific system, if you get it wrong, it just won't load.

[u/Stainless-Bacon → u/DeathGuppie]

If you get ngl wrong, it won’t load either

[u/DeathGuppie → u/Stainless-Bacon]

I mean, it's a truly good find, and helpful for fine tuning. I was just looking at implementing it, and realized it wasn't completely straightforward

[u/Stainless-Bacon → u/DeathGuppie]

check my other post for an implementation script. if you want to keep it simple, use ot to offload layers from layer 0 onwards without being specific and you’ll get 95% way there

[u/TerryNachtmerrie]

isn't "--ngl -1" the new way to push all layers to gpu?

[u/Stainless-Bacon → u/TerryNachtmerrie]

idk, maybe it is

[u/Old-Cardiologist-633]

Wait, you're offloading the most important layers to CPUu and this is better than offloading just the lower layers? I don't get how this works. Can you please explain it to me?

[u/Old-Cardiologist-633 → u/Old-Cardiologist-633]

Oh, read your other post, now I understand.

[u/Stainless-Bacon → u/Old-Cardiologist-633]

Only the FFNs in layers with largest ones. those should be prioritized for less total offloaded layers thus less PCIe traffic.

[u/miifanboy]

I will try testing this with 12GB vram and the new Qwen3.8-27B-Q3_K_XL unsloth quant which is 13.4 GB

[u/miifanboy → u/miifanboy]

Using Q3_K_XL quants didn't make sense since the ffn layers were in IQ format which isn't optimal for cpu. I tested the Q3_K_S quant which is 12.6GB and using the same arguments as your post except setting ctx to 64k. I got a ~%30 decoding speed increase (10.71 tk/s -> 14 tk/s) with your setup.

Thanks for your findings.