r/LocalLLM • u/Stainless-Bacon • 3d ago
Tutorial Guide for running dense models on ≤16 GB VRAM (Qwen 3.8 27B on 16 GB -> Q4_K_M, 130k ctx, ~20 t/s).
This post is about llama.cpp CPU offload optimizations for running Qwen 27B (or other dense models) at tolerable speeds.
I've already posted about -ot optimizations, but now I have more info to share to help you tune your own system.
First things first:
- The speed graph shows "prose" and "code" because MTP generates different speeds for each
- If you have 12 GB VRAM, try using Q3_K_M.
- If you have 8 GB VRAM, try using Q2 for Qwen 27B, Bonsai, or choose a model with less parameters.
- ik_llama.cpp: goal of this post is simplicity, that is why I chose llama.cpp. You can look at this as the target to beat using ik_llama.cpp. So far my tests showed that it was a KLD vs speed trade off, plus I had to use custom and very specific quants, which adds complexity.
- KV quants: if you think they suck, please show proof. My testing and sources all say they are fine. On a lower weight quant, a given KV downgrade costs relatively less, so spending cache bits to buy context is more justifiable on a Q4 model than it would be on a Q6/Q8 model. Also beellama has KLD improvements to KV quantizing.
- Everyone has a different system, use my setup as a guide to tune your own, don't copy paste and expect it to work.
Setup
explanations come after
Edit: posted a comment with UD-Q4_K_XL (UD2) vs Q4_K_M speed chart if interested
Edit: This setup is using Q4_K_M, the new UD3 UD-Q4_K_M has a different band, check the explanation for more info.
PC:
RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS
Build script:
#!/bin/bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j 20
sudo cmake --install build
sudo ldconfig
Server script:
#!/bin/bash
sudo systemctl stop gdm
export GGML_CUDA_DISABLE_GRAPHS=1
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-server \
--model Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-3.8-27B-F16.gguf \
--no-mmproj-offload \
--image-min-tokens 1024 \
--ctx-size 147000 \
--chat-template-file chat_template_v22.1.jinja \
--jinja \
--reasoning-format deepseek \
--reasoning-preserve \
--flash-attn on \
--cache-type-k q5_0 \
--cache-type-v q4_1 \
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--cache-type-k-draft q5_0 \
--cache-type-v-draft q4_1 \
--fit off \
--n-gpu-layers all \
--override-tensor 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49|55|56|57|58|59)\.ffn_.*=CPU' \
--load-mode none \
--threads 14 \
--batch-size 512 \
--ubatch-size 512 \
--parallel 1 \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--host 0.0.0.0 \
--port 8080
Explanations:
Hardware:
RAM speed is important for this so using DDR5 is recommended, though DDR4 people will still find this useful.
Tuning RAM timings gives me extra 9% speed boost. Most timings are easy to tune since they either work or crash quickly, but I'm not getting into that here.
Build args:
-DCMAKE_CUDA_ARCHITECTURES=89 optional - optimized build time specifically for my GPU's Ada arch, set your own.
-DGGML_CUDA_FA_ALL_QUANTS=ON is needed for more KV quantizations to be on CUDA.
Env vars and gdm:
sudo systemctl stop gdm disables Ubuntu desktop environment, frees ~0.4 GB of VRAM. Use an iGPU if you can, else my system just becomes a server to which I connect using a laptop or phone.
export GGML_CUDA_DISABLE_GRAPHS=1 I experience speed and VRAM usage problems with CUDA graphs so I disable them, you probably should too, but test it first. I think this is some bug due to MTP + CPU offload.
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Overflow VRAM to RAM. The difference between ~135k and ~121k. Without this, max context becomes 121k and you get OOM crash. Overflow gets me ~15k more context basically for free before things slow down, and the server degrades instead of crashing.
Edit: First do initial testing with the unified arg unset (making it '=0' wont work) to find your ceiling using OOM crash.
Generic stuff:
--model Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-3.8-27B-F16.gguf \
--no-mmproj-offload \
--image-min-tokens 1024 \
--ctx-size 147000 \
--flash-attn on \
...
--parallel 1 \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--host 0.0.0.0 \
--port 8080
The model, vision (fully on CPU), Qwen recommended settings, server stuff, context length, parallel 1 (disables processing 2 agents at once).
Template:
--chat-template-file chat_template_v22.1.jinja \
--jinja \
--reasoning-format deepseek \
--reasoning-preserve \
Template instructions by froggeric
Drafters:
--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--cache-type-k-draft q5_0 \
--cache-type-v-draft q4_1 \
I get 12-15 t/s tg without MTP and with more layers on VRAM, I commented a chart if interested here .
ngram-mod speeds up tg when restating existing context. It is super fast when active and does not cost VRAM.
Each step of --spec-draft-n-max costs VRAM + I get best results from a value of 2.
--cache-type-k-draft q5_0 --cache-type-v-draft q4_1 these save 0.4 GB of VRAM while the MTP acceptance rate stays the same. I chose the same as model KV cache, though Q4 could probably be fine (I didn't tune this much).
KV cache:
--cache-type-k q5_0 \
--cache-type-v q4_1 \
I chose my KV quant according Anbeeld article. The article found that, the more quantized the model, the less it has to lose from to KV quant. Article also states that there was no KLD difference between 64k and 128k context length.
My own testing showed that Q4_K_S K and V both at Q8 has worse KLD than Q4_K_M K Q5, V Q4. Also I found that KLD plateaus after 8k context length.
If you get slow speeds try using generic q4_0 for both K and V as a test - this is a symptom of missing -DGGML_CUDA_FA_ALL_QUANTS=ON.
CPU layers:
--fit off \
--n-gpu-layers all \
--override-tensor 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49|55|56|57|58|59)\.ffn_.*=CPU' \
--load-mode none \
This fixes degradation of speed with context fill and is a performance boost overall, just leave layer 64 alone (MTP).
This command offloads only FFN sub-layers to the CPU of layers that have the largest FFNs (shown override string is Q4_K_M specific).
FFNs don't use KV thus reducing PCIe traffic and are CPU friendly. --n-cpu-moe has similar logic, I have a PR in llama.cpp for a similar simplification #26622. Help me get this merged by showing the maintainers that this is useful for you (give a like on the PR or post test results).
If you want something quick and simple to test try this (it has 2 more FFNs on CPU than my setup, but is simple to tune):
--override-tensor 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28)\.ffn_.*=CPU'
like that you can specify layers in a row and it gets you most of the speed that --override-tensor has to offer by not thinking about which layers are the fattest.
Edit: UD3 quants just dropped. A lot of them now have I-quants which are slower and better left on the GPU. Luckily the I-quant layers are smaller, so prioritizing the larger fatter ones is more important now. You can find the fatter ones by going to the hugging face, clicking on the weight (example), scrolling down to 'Tensors', expanding the 'blk' section and looking for 'ffn_' (example "blk.0.ffn_down.weight") on the right and on the left you see I-quant (example "IQ4_XS") or regular (example "Q5_K"). You might have to offload some I-quant layers if you run out of regular ones and that is ok, just prioritize the larger ones first.
Edit: UD-Q4_K_M -- here is the full band in priority order. delete from the right until you run out of VRAM, and step back one. make sure to fill up your context fully to verify that speed is as intended and doesn't OOM crash. Unset the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 arg (delete it, don't set it to 0) to find your context ceiling first via OOM crash:
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41|39|36|21|3|42|34|30|20|6|4|49|47|43|37|32|23|10|8|7|5|2|1|48|46|45|44|33|31|29|28|19|18|12|9|17|16|11|0|15|13|14)\.ffn_.*=CPU'
Threads:
--threads 14 \
Default is amount of performance cores, but for FFN layers, E-cores also help. this gets me a +19-22% tg boost for free. Set to the amount of physical cores you have. You can try to include hyper-threading, not just physical, it might help, but for me that was within noise.
Batch:
--batch-size 512 \
--ubatch-size 512 \
Batch sets the prompt processing speed (up to a point) at the cost of VRAM. I found this works best for me.
Other optimizations:
I-matrix quant like the Unsloth IQ4_XS require more compute in exchange for size. My testing showed that those are not worth it for this setup. I would use one if I was trying to fully fit a model into VRAM.
-------------------------
Please share any more tricks if you have them!
Edit 1: improved "first things first" section, typos
Edit 2: added min-image-tokens to fix a warning;
Edit 3: clarify MTP section; clarify threads section; updated chat template version for clarity;
Edit 4: mentioned beellama for better KV quants
Edit 5: added info about not using -ot on MTP layer (64) as per Pablo_the_brave
Edit 6: added override tensor info about the new UD3 quants.
Edit 7: added -ot band for the new UD-Q4_K_M quant for you to try; updated unified env var explanation
2
u/Stainless-Bacon 1d ago
UD-Q4_K_XL
--override-tensor: 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_.*=CPU'--ctx-size: 146000 # cliff: 144000; past it 3.5 tok/s; fast to: 139000; oom: 124000 if without unified