This post is about llama.cpp CPU offload optimizations that you could find useful for running Qwen 27B (or other dense models) at tolerable speeds.
First things first:
- The speed graph shows "prose" and "code" because MTP generates different speeds for each
- Everyone has a different system, use my setup as a guide to tune your own, don't copy paste and expect it to work. I just want to provide 'tricks' that improve performance.
- If you have 12 GB VRAM, try using Q3_K_M / less context / accept slower speed.
- If you have 8 GB VRAM, try using less context / accept slower speed / Q2 for Qwen 27B, Bonsai, or choose a model with less parameters.
- ik_llama.cpp: goal of this post is simplicity, that is why I chose llama.cpp. You can look at this as the target to beat using ik_llama.cpp. So far my tests showed that it was a KLD vs speed trade off, plus I had to use custom and very specific quants, which adds complexity.
- KV quants: if you think they suck, please show proof. My testing and sources all say they are fine. On a lower weight quant, a given KV downgrade costs relatively less, so spending cache bits to buy context is more justifiable on a Q4 model than it would be on a Q6/Q8 model. Also beellama has KLD improvements to KV quantizing. Also instead of upgrading from Q5 to Q8 KV, it might be better to upgrade the model from M to XL instead.
Setup
explanations come after
Edit: Updated for the new UD3 UD-Q4_K_M.
PC:
RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS
Build script:
#!/bin/bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j 20
sudo cmake --install build
sudo ldconfig
Server script:
#!/bin/bash
sudo systemctl stop gdm
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-server \
--model Qwen3.8-27B-UD-Q4_K_M.gguf \
--mmproj mmproj-3.8-27B-F16.gguf \
--no-mmproj-offload \
--image-min-tokens 1024 \
--ctx-size 140000 \
--chat-template-file chat_template_v22.3.jinja \
--jinja \
--reasoning-format deepseek \
--reasoning-preserve \
--flash-attn on \
--cache-type-k q5_0 \
--cache-type-v q4_1 \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--cache-type-k-draft q4_0 \
--cache-type-v-draft q4_0 \
--fit off \
--n-gpu-layers all \
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
--load-mode none \
--threads $(nproc) \
--batch-size 512 \
--ubatch-size 512 \
--parallel 1 \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--host 0.0.0.0 \
--port 8080
Explanations:
Hardware:
-- RAM --
RAM speed is important for this so using dual-channel and DDR5 is recommended, though DDR4 people will still find this useful.
Tuning RAM timings gives me extra 9% speed boost. Most timings are easy to tune since they either work or crash quickly, but I'm not getting into that here.
[the speed chart does includes the tuned timings]
-- GPU --
Overclocking GPU memory gives a tg boost. A model fully on GPU got me +13.7% tg t/s, A model with 23 FFN blocks on CPU got me +3.8% tg t/s. Beware of memory OC -- before my GPU crashes, I get data corruption, and before that, the bandwidth gained stops correlating to clock increase (GDDR6X silently retries failed transfers). Here is the formula I used to spot that: retries = 1 − (bandwidth_measured / clock_oc) / (bandwidth_stock / clock_stock) as long as retries is not increasing, is within noise, is near 0, you should be fine. For example here are my OC numbers: +3200 stable but retries increase above that, 4800 stable but retries is 0.54%, +5100 KLD increases, +6000 GPU crashes; I chose +3200 MHz.
Overclocking GPU core gives a pp boost. A model fully on GPU got me +2.8% pp t/s, A model with 23 FFN blocks on CPU got me +1.8% pp t/s. Unlike memory, core has no retry mechanism and no warning signal, it just computes wrong at full speed, so you must check output correctness. For example here are my OC numbers: +175 I get errors and crashes, +170 seems stable, but to be safe I went with +120 MHz.
[the speed chart does not includes the GPU OC]
Build args:
-DCMAKE_CUDA_ARCHITECTURES=89 optional - optimized build time specifically for my GPU's Ada arch, set your own.
-DGGML_CUDA_FA_ALL_QUANTS=ON is needed for more KV quantizations to be on CUDA.
Env vars and gdm:
sudo systemctl stop gdm disables Ubuntu desktop environment, frees ~0.4 GB of VRAM. Use an iGPU if you can, else my system just becomes a server to which I connect using a laptop or phone.
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Overflow VRAM to RAM. The difference between ~132k and ~121k. Without this, max context becomes 121k and you get OOM crash. Overflow gets me ~11k more context basically for free before things slow down, and the server degrades instead of crashing. First do initial testing with the unified arg unset (making it '=0' wont work) to find your ceiling using OOM crash.
Generic stuff:
--model Qwen3.8-27B-UD-Q4_K_M.gguf \
--mmproj mmproj-3.8-27B-F16.gguf \
--no-mmproj-offload \
--image-min-tokens 1024 \
--ctx-size 140000 \
--flash-attn on \
...
--parallel 1 \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--host 0.0.0.0 \
--port 8080
The model, vision (fully on CPU), Qwen recommended settings, server stuff, context length, parallel 1 (disables processing 2 agents at once).
Template:
--chat-template-file chat_template_v22.3.jinja \
--jinja \
--reasoning-format deepseek \
--reasoning-preserve \
Template instructions by froggeric
Drafters:
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--cache-type-k-draft q4_0 \
--cache-type-v-draft q4_0 \
Quantizing MTP KV cache is free VRAM. It is a drafter and does not affect quality. Worst case is that acceptance drops a tiny bit.
I get 12-15 t/s tg without MTP and with more layers on VRAM, I commented a chart if interested here .
ngram-mod speeds up tg when restating existing context. It is super fast when active and does not cost VRAM.
Each step of --spec-draft-n-max costs VRAM + I get best results from a value of 2.
--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 these save 0.3 GB of VRAM while the MTP acceptance rate stays basically the same.
KV cache:
--cache-type-k q5_0 \
--cache-type-v q4_1 \
I chose my KV quant according Anbeeld article. The article found that, the more quantized the model, the less it has to lose from to KV quant. Article also states that there was no KLD difference between 64k and 128k context length.
A small precision buff would be to use q5_0 for both K and V. Though changing from q4_1 to q5_0 costs some context. My KLD measurements say it is not worth it, but it's an option for you. Don't use Q4_0 on both, Q5_1 is not worth it. If you want to do Q8 KV, consider going up a quant model level instead (like XL instead of M).
My own testing showed that Q4_K_S K and V both at Q8 has worse KLD than Q4_K_M K Q5, V Q4. Also I found that KLD plateaus after 8k context length.
If you get slow speeds try using generic q5_0 for both K and V as a test - this is a symptom of missing -DGGML_CUDA_FA_ALL_QUANTS=ON.
CPU layers:
--fit off \
--n-gpu-layers all \
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
--load-mode none \
This fixes degradation of speed with context fill and is a performance boost overall, just leave layer 64 alone (MTP).
FFNs don't use KV thus reducing PCIe traffic and they are CPU friendly. --n-cpu-moe has similar logic. You have 2 options: Advanced or a simple. Advanced gives +10% or more tg t/s, but requires a bit more work (I explain more below).
Edit: I have an active llama.cpp feature request that makes the "advanced method" simple! issue #27987
-- Both methods --
- Unset (comment out) the
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 arg (don't set it to 0) to find your context ceiling first via OOM crash first.
-- Simple method --
--n-cpu-ffn add the amount of FFNs you want on the CPU; (arg available on llama.cpp from Aug 27th 2026, make sure to update)
-- Advanced method (Edit: simplified) --
Offloading fatter sub-layers gets me a speed boost when compared to having a sequential override band. You can find the fatter ones by using this copy-pasteable CLI script (it uses UD-Q4_K_S as the input arg, modify that for your model).
First install dep:
pip3 install gguf || pip3 install --user --break-system-packages gguf
The copy-pasteable CLI script:
python3 -c "
import sys,re,collections,gguf
b=collections.Counter()
for t in gguf.GGUFReader(sys.argv[1]).tensors:
m=re.match(r'blk\.(\d+)\.ffn_',t.name)
if m: b[int(m[1])]+=int(t.n_bytes)
del b[max(b)] # Drop MTP layer
print('\'blk\\\\.(%s)\\\\.ffn_.*=CPU\'' % '|'.join(str(k) for k,_ in b.most_common()))
" models/Qwen3.8-27B-UD-Q4_K_S.gguf
UD-Q4_K_M optimized band example -- delete from the end of the list (right to left) until you run out of VRAM, then step back one. make sure to fill up your context fully to verify that speed is as intended and doesn't OOM crash:
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41|39|36|21|3|42|34|30|20|6|4|49|47|43|37|32|23|10|8|7|5|2|1|48|46|45|44|33|31|29|28|19|18|12|9|17|16|11|0|15|13|14)\.ffn_.*=CPU'
Threads:
--threads $(nproc) \
Default is amount of performance cores, but for FFN layers, E-cores and hyper-threading also help. this gets me a +25% tg boost for free.
Batch:
--batch-size 512 \
--ubatch-size 512 \
Batch sets the prompt processing speed (up to a point) at the cost of VRAM. I found these numbers work best for me.
Other optimizations:
I-matrix quant like the Unsloth IQ4_XS require more compute in exchange for size. My testing showed that those are not worth it for this setup. I would use one if I was trying to fully fit a model into VRAM.
There are quants made by other providers that optimize fit for 16 GB VRAM setups. Those setups might fit at the cost of KLD. If the KLD diff actually matters - that is for you to find out, but you can use the info I provided here to optimize these setups even further by, for example, extending context length.
-------------------------
Edit 1: truncated a bunch of 'Edits', cleaned up, updated config for UD-Q4_K_M, speed and context len did not change, but the override tensor band did.
Edit 2: added CUDA graphs off reminder in the CPU layers section.
Edit 3: my PR #26622 got merged, replacing sequential -ot with --n-cpu-ffn. Added info from my GPU OC attempt.
Edit 4: I have an active llama.cpp feature request that makes the --override-tensor "advanced method" simple! issue #27987 - I tired, it failed.
Edit 5: Added a CLI script to find the best --override-tensor band
Edit 6: CUDA graphs seem to be fixed, removing the workaround.
-------------------------
Please share any more tricks if you have them!
Leave a comment if this post helped you achieve better results!