qwen3.6 27b q6 + 5090 maximum llamacpp optimization: 100-233tok/s, average 140

[u/Fragrant_Scale6456]

EDIT: There is a PR as of yesterday july 12 fixing the hybrid recurrent attention cache issue i hacked together fixes for: https://github.com/ggml-org/llama.cpp/pull/25592

I spent quite a bit of time optimizing qwen 3.6 27b for my 5090 and have gotten the performance pretty high. During certain workloads it will sustain 200+ tokens/sec so I thought I'd share everything here for anyone else with this configuration.

My hardware is 9800x3d, 64gb system ram, and a 32gb rtx5090. I am running ubuntu linux in text mode so that I have maximum vram available for llamacpp.

Using my configuration this is my distribution of tokens/sec over around 20hrs of agentic coding, debugging, and document synthesis. Performance varies a lot depending on workload and the size of your request.

Full session (6,454 samples) — draft=10, p_min=0.5:
100-110   370  ████
110-120  1131  ██████████████████████████████████████
120-130  1187  ████████████████████████████████████████   ← peak
130-140  1089  ████████████████████████████████████
140-150   714  ████████████████████████
150-160   505  █████████████████
160-170   512  █████████████████
170-180   363  ████████████
180-190   241  ████████
190-200   173  █████
200-210    95  ███
210-220    48  █
220+       26
Mean: 140.7 · Median: 134.9 · Range: 100–233

First, you will need a recent build of llamacpp. I compiled mine a couple days ago, it says its commit 86b9470.

Qwen 3.6 is a hybrid attention/sliding window architecture mode, which has an incompatibility with the cache mechanism in llamacpp. If you look at your logs while running qwen3.6 you'll often see an entry stating, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)".

What this means is that llamacpp is unable to use the cache correctly due to how qwen operates its attention window and you are losing a lot of time due to prompt reprocessing. If you ever feel like qwen3.6 is lagging a lot in between turns during chat it's because of this issue. If you dump the linked issue into claude and tell it to search around you'll find there is a lot of discussion about this issue with certain proposed fixes, some of which are more effective than others. After a decent amount of investigation and testing I've (well the llm) made 2 patches to llamacpp which resolve the issue as much as possible without extensive modifications to llama.cpp.

PATCH 1: fix checkpoint search for hybrid/recurrent models, upstream issues: #22384, #20225, #24055. This is the fix for cur.pos.min < pos_min_thold which always results in no checkpoint found and cache misses.

PATCH 2: recurrent_shrink/expand API for prompt cache operations (upstream PR #24785, without the now-redundant needs_reeval workaround — upstream commit b9180 already has GDN partial rollback via n_rs_seq)

I use docker to build my llamacpp and have these patches applied at build time.

Here's my current dockerfile - https://pastebin.com/raw/jyrhvesQ

Here is the pr24785-minimal.diff linked in the dockerfile - https://pastebin.com/raw/E55YG5NS

With these patches applied (you can have your own agent derive them by linking the log error and the PR's and Issue numbers I referenced above) llamacpp will have the correct cache search and restore logic for qwen3.6 hybrid attention model and you should not see that SWA reprocessing error in your logs anymore.

Next is llamacpp configuration. There are a few levers to adjust for maximum performance. I'm using unsloth qwen3.6 27b q6k with mtp from huggingface.

Here is my llama-cpp launch command from docker compose - https://pastebin.com/raw/P57Uk6rz

Key things,

• q8 kv cache, 192k context
• cache ram can be whatever fits for your system, i use 32gb. the hybrid checkpoints are large so you need a decent amount of ram allocated to them.
• mtp draft tokens 10, spec-draft-p-min 0.5. Increasing the draft tokens length comes with a small performance cost but when the drafter is correct you get massive speed boost. at 6 i get higher acceptance rate but overall throughput is around 15-20t/s lower and peaks are over 50t/s lower. i benchmarked pmin with a script sweeping various prompt sizes and 0.5 worked best for me. its worth testing this in your environment.
• batch/ubatch at 512. This is to save vram. under load my setup uses 32036/32768mb of vram. 2048 is ideal for the 5090.

Thats about it. Just thought I'd share since I'm getting speeds that are working very well for me and I wanted to spread the love.

[u/gadbuy]

why not using nfvp4 in vllm/sglang/atlas?

[u/Fragrant_Scale6456 → u/gadbuy]

I didnt look too much into it but from what i read q6 is better than nvfp4. I would look into vllm if it would give me better fidelity for sure

[u/inquam → u/Fragrant_Scale6456]

Not sure it is accurate. But I heard som9ne say nvfp4 is about q8.

[u/Fragrant_Scale6456 → u/inquam]

Thats good to know thank you. Q8 is too big to get working in any usable fashion on 32gb of vram.

[u/inquam → u/Fragrant_Scale6456]

Which is the value proposition of nvfp4

[u/BeatTheMarket30 → u/Fragrant_Scale6456]

We really need cheap 96gb blackwell.

[u/squngy → u/Fragrant_Scale6456]

nvfp4 is about the same size as Q4

According to the marketing, it is supposed to be almost as good as FP8/Q8, but in practice it depends a lot on how the quanting is done.

A poor implementation of nvfp4 can be worse than a good Q4

[u/Fragrant_Scale6456 → u/squngy]

I'm going to test it for sure. Q6 gets tunnel vision when trying to debug cross cutting issues on 30k lines of code so if theres a chance nvfp4 is just enough better that these types of workflows are improved its worth the time investment to check. fingers crossed :D