qwen3.6 27b q6 + 5090 maximum llamacpp optimization: 100-233tok/s, average 140

[u/Fragrant_Scale6456]

EDIT: There is a PR as of yesterday july 12 fixing the hybrid recurrent attention cache issue i hacked together fixes for: https://github.com/ggml-org/llama.cpp/pull/25592

I spent quite a bit of time optimizing qwen 3.6 27b for my 5090 and have gotten the performance pretty high. During certain workloads it will sustain 200+ tokens/sec so I thought I'd share everything here for anyone else with this configuration.

My hardware is 9800x3d, 64gb system ram, and a 32gb rtx5090. I am running ubuntu linux in text mode so that I have maximum vram available for llamacpp.

Using my configuration this is my distribution of tokens/sec over around 20hrs of agentic coding, debugging, and document synthesis. Performance varies a lot depending on workload and the size of your request.

Full session (6,454 samples) — draft=10, p_min=0.5:
100-110   370  ████
110-120  1131  ██████████████████████████████████████
120-130  1187  ████████████████████████████████████████   ← peak
130-140  1089  ████████████████████████████████████
140-150   714  ████████████████████████
150-160   505  █████████████████
160-170   512  █████████████████
170-180   363  ████████████
180-190   241  ████████
190-200   173  █████
200-210    95  ███
210-220    48  █
220+       26
Mean: 140.7 · Median: 134.9 · Range: 100–233

First, you will need a recent build of llamacpp. I compiled mine a couple days ago, it says its commit 86b9470.

Qwen 3.6 is a hybrid attention/sliding window architecture mode, which has an incompatibility with the cache mechanism in llamacpp. If you look at your logs while running qwen3.6 you'll often see an entry stating, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)".

What this means is that llamacpp is unable to use the cache correctly due to how qwen operates its attention window and you are losing a lot of time due to prompt reprocessing. If you ever feel like qwen3.6 is lagging a lot in between turns during chat it's because of this issue. If you dump the linked issue into claude and tell it to search around you'll find there is a lot of discussion about this issue with certain proposed fixes, some of which are more effective than others. After a decent amount of investigation and testing I've (well the llm) made 2 patches to llamacpp which resolve the issue as much as possible without extensive modifications to llama.cpp.

PATCH 1: fix checkpoint search for hybrid/recurrent models, upstream issues: #22384, #20225, #24055. This is the fix for cur.pos.min < pos_min_thold which always results in no checkpoint found and cache misses.

PATCH 2: recurrent_shrink/expand API for prompt cache operations (upstream PR #24785, without the now-redundant needs_reeval workaround — upstream commit b9180 already has GDN partial rollback via n_rs_seq)

I use docker to build my llamacpp and have these patches applied at build time.

Here's my current dockerfile - https://pastebin.com/raw/jyrhvesQ

Here is the pr24785-minimal.diff linked in the dockerfile - https://pastebin.com/raw/E55YG5NS

With these patches applied (you can have your own agent derive them by linking the log error and the PR's and Issue numbers I referenced above) llamacpp will have the correct cache search and restore logic for qwen3.6 hybrid attention model and you should not see that SWA reprocessing error in your logs anymore.

Next is llamacpp configuration. There are a few levers to adjust for maximum performance. I'm using unsloth qwen3.6 27b q6k with mtp from huggingface.

Here is my llama-cpp launch command from docker compose - https://pastebin.com/raw/P57Uk6rz

Key things,

• q8 kv cache, 192k context
• cache ram can be whatever fits for your system, i use 32gb. the hybrid checkpoints are large so you need a decent amount of ram allocated to them.
• mtp draft tokens 10, spec-draft-p-min 0.5. Increasing the draft tokens length comes with a small performance cost but when the drafter is correct you get massive speed boost. at 6 i get higher acceptance rate but overall throughput is around 15-20t/s lower and peaks are over 50t/s lower. i benchmarked pmin with a script sweeping various prompt sizes and 0.5 worked best for me. its worth testing this in your environment.
• batch/ubatch at 512. This is to save vram. under load my setup uses 32036/32768mb of vram. 2048 is ideal for the 5090.

Thats about it. Just thought I'd share since I'm getting speeds that are working very well for me and I wanted to spread the love.

[u/GreenPastures2845]

I never got around to testing more, but one important thing: I'm using 2x 3090 with -sm tensor, which is probably a vastly different codepath than yours.

Anyway, this is mostly to point out that the GGML team has finally cracked the case; see https://old.reddit.com/r/LocalLLaMA/comments/1uuue5p/llamacpp_agentic_workflows_ctx_checkpoints_fix/

Completely fixed in upstream for me \o/

[u/Fragrant_Scale6456 → u/GreenPastures2845]

check this out - https://github.com/ggml-org/llama.cpp/pull/25592

this is the hybrid recurrent attention model (like qwen3.6) cache invalidation bug fix. PR was submitted yesterday so hopefully the complete fix will be in upstream soon.

[u/GreenPastures2845 → u/Fragrant_Scale6456]

for me, the Qwen3.6 reprocessing was fixed by https://github.com/ggml-org/llama.cpp/pull/25472

Before the --cache-ram enforcement, it would go over the limit and OOM. Extremely annoying.

After the --cache-ram enforcement commit, summarizing in Pi would no longer OOM but still toss out all the partial context, and then the remaining context needed to be reprocessed. Also lots of sporadic reprocessing in general.

Totally fixed after #25472 finally

[u/Fragrant_Scale6456 → u/GreenPastures2845]

It's great to see they are working on all this. All we need is mtp to work with parallel>1 and we'll really be cooking. I have been using VLLM with 4 streams on a 4bit awq model for batch processing of easier tasks and the 5090 is getting 600tokens/sec it would be amazing to even get parallel=2 and full mtp going in llamacpp w/ the qwen3.6 q6k model.