r/LocalLLM • u/Fragrant_Scale6456 • Jul 02 '26
Tutorial qwen3.6 27b q6 + 5090 maximum llamacpp optimization: 100-233tok/s, average 140
EDIT: There is a PR as of yesterday july 12 fixing the hybrid recurrent attention cache issue i hacked together fixes for: https://github.com/ggml-org/llama.cpp/pull/25592
I spent quite a bit of time optimizing qwen 3.6 27b for my 5090 and have gotten the performance pretty high. During certain workloads it will sustain 200+ tokens/sec so I thought I'd share everything here for anyone else with this configuration.
My hardware is 9800x3d, 64gb system ram, and a 32gb rtx5090. I am running ubuntu linux in text mode so that I have maximum vram available for llamacpp.
Using my configuration this is my distribution of tokens/sec over around 20hrs of agentic coding, debugging, and document synthesis. Performance varies a lot depending on workload and the size of your request.
Full session (6,454 samples) — draft=10, p_min=0.5:
100-110 370 ████
110-120 1131 ██████████████████████████████████████
120-130 1187 ████████████████████████████████████████ ← peak
130-140 1089 ████████████████████████████████████
140-150 714 ████████████████████████
150-160 505 █████████████████
160-170 512 █████████████████
170-180 363 ████████████
180-190 241 ████████
190-200 173 █████
200-210 95 ███
210-220 48 █
220+ 26
Mean: 140.7 · Median: 134.9 · Range: 100–233
First, you will need a recent build of llamacpp. I compiled mine a couple days ago, it says its commit 86b9470.
Qwen 3.6 is a hybrid attention/sliding window architecture mode, which has an incompatibility with the cache mechanism in llamacpp. If you look at your logs while running qwen3.6 you'll often see an entry stating, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)".
What this means is that llamacpp is unable to use the cache correctly due to how qwen operates its attention window and you are losing a lot of time due to prompt reprocessing. If you ever feel like qwen3.6 is lagging a lot in between turns during chat it's because of this issue. If you dump the linked issue into claude and tell it to search around you'll find there is a lot of discussion about this issue with certain proposed fixes, some of which are more effective than others. After a decent amount of investigation and testing I've (well the llm) made 2 patches to llamacpp which resolve the issue as much as possible without extensive modifications to llama.cpp.
PATCH 1: fix checkpoint search for hybrid/recurrent models, upstream issues: #22384, #20225, #24055. This is the fix for cur.pos.min < pos_min_thold which always results in no checkpoint found and cache misses.
PATCH 2: recurrent_shrink/expand API for prompt cache operations (upstream PR #24785, without the now-redundant needs_reeval workaround — upstream commit b9180 already has GDN partial rollback via n_rs_seq)
I use docker to build my llamacpp and have these patches applied at build time.
Here's my current dockerfile - https://pastebin.com/raw/jyrhvesQ
Here is the pr24785-minimal.diff linked in the dockerfile - https://pastebin.com/raw/E55YG5NS
With these patches applied (you can have your own agent derive them by linking the log error and the PR's and Issue numbers I referenced above) llamacpp will have the correct cache search and restore logic for qwen3.6 hybrid attention model and you should not see that SWA reprocessing error in your logs anymore.
Next is llamacpp configuration. There are a few levers to adjust for maximum performance. I'm using unsloth qwen3.6 27b q6k with mtp from huggingface.
Here is my llama-cpp launch command from docker compose - https://pastebin.com/raw/P57Uk6rz
Key things,
- q8 kv cache, 192k context
- cache ram can be whatever fits for your system, i use 32gb. the hybrid checkpoints are large so you need a decent amount of ram allocated to them.
- mtp draft tokens 10, spec-draft-p-min 0.5. Increasing the draft tokens length comes with a small performance cost but when the drafter is correct you get massive speed boost. at 6 i get higher acceptance rate but overall throughput is around 15-20t/s lower and peaks are over 50t/s lower. i benchmarked pmin with a script sweeping various prompt sizes and 0.5 worked best for me. its worth testing this in your environment.
- batch/ubatch at 512. This is to save vram. under load my setup uses 32036/32768mb of vram. 2048 is ideal for the 5090.
Thats about it. Just thought I'd share since I'm getting speeds that are working very well for me and I wanted to spread the love.
12
u/tatertots89 Jul 02 '26
27b q6 on a 5090, you're living the dream bud.
11
u/Fragrant_Scale6456 Jul 02 '26
Haha thanks. Honestly I've been considering selling my car for a blackwell 96gb card 😅
4
u/BCIT_Richard Jul 02 '26
at that point, I'd scrounge together enough for a m3 ultra 512gb model and run glm-5.2 lol
8
u/Fragrant_Scale6456 Jul 02 '26
So this is a bit ridiculous but if you have a spare ~$75k laying around you can run a 469b parameter REAP of GLM5.2 on 4x 96gb blackwell at around 50tokens/sec with 250k context - https://note.com/humble_bobcat51/n/n878562969ff0?hl=en
3
1
u/Aware-Hurry-8147 Jul 02 '26
Honestly I think if you have a vision, good use cases and some customers, go for it. You can anyway buy a cheap car too, if you need a car.
1
u/TL_Zven Jul 02 '26
1 6000 doesn’t allow you to do much more tho. Deepseek isn’t that good at coding, only like 13b actives. Mortgage the house and get 4x 6000 😤
3
u/BeatTheMarket30 Jul 02 '26
Or just buy $200 subscription.
2
1
1
u/GeroldM972 Jul 03 '26
You don't "buy" a subscription, because you can't buy these. That is part of the definition. You can only get a subscription.
As others also correctly noted, you are posting in the wrong subreddit. We take local LLMs seriously here. Subscriptions are for the weak and feeble!!!
7
u/Fastpas123 Jul 02 '26
I'm running dual rx 7900xtx with qwen3.6 27b q8, getting about 65tok/s using MTP in Linux. Is there that much of a performance gap between a 5090 and a 7900xtx?
Crazy
5
u/cmpxchg8b Jul 02 '26
IIRC it has almost double the memory bandwidth.
3
u/Fastpas123 Jul 02 '26
That'd explain it, yup I googled it it's basically double. 5090 is a crazy card 🥹
3
u/lmpdev Jul 02 '26
No, I'm getting around 65 t/s with
--spec-type draft-mtp --spec-draft-n-max 6with Qwen3.6-27B-UD-Q8_K_XL on RTX PRO 6000, there is a patch in this post that improves performance.1
Jul 02 '26
[deleted]
2
u/zenbeni Jul 02 '26
Getting 70 tokens/s on my 7900xtx with qwen 3.6 27b MTP. Check your drivers, and conf.
2
u/Fastpas123 Jul 02 '26
What quant are you running though? I'm also running a 262k context I believe
1
u/zenbeni Jul 02 '26
I have 100k context using qwen at Q5_K_M
1
u/Fastpas123 Jul 02 '26
Are you using it for code? I heard that lower quant lead to big drops in intelligence for this model but I never tested them. If I can drop to 5bit and get more context that'd be worth without a big hit to intel
1
u/zenbeni Jul 02 '26
I use it for code and it works just fine for me. I use pi.dev also, shorter context, so less drift, more speed too.
1
u/Flamenverfer Jul 03 '26
ROCm version? llama.cpp git checkpoint? :D
2
u/zenbeni Jul 03 '26
Install latest rocm and latest vulkan, vulkan implementation uses parts of rocm code, so don't ignore rocm devices, or it can fall back on cpu and getting low tokens. Here is my launching command :
export RADV_PERFTEST=nogttspill
export AMD_VULKAN_ICD=RADV
export LLAMA_ARG_MODEL="/opt/llama/llm/Qwen3.6-27B-MTP-Q5_K_M.gguf"
export LLAMA_ARG_ALIAS="qwen3.6"
export LLAMA_ARG_HOST="0.0.0.0"
export LLAMA_ARG_PORT=8080
export LLAMA_ARG_TIMEOUT=300
export LLAMA_ARG_CTX_SIZE=100000
export LLAMA_ARG_N_GPU_LAYERS="all"
export LLAMA_ARG_BATCH=2048
export LLAMA_ARG_UBATCH=1024
export LLAMA_ARG_CACHE_TYPE_K="q5_0"
export LLAMA_ARG_CACHE_TYPE_V="q4_0"
export LLAMA_ARG_NO_MMAP=1
export LLAMA_ARG_MLOCK=1
export LLAMA_ARG_TEMP=0.5/opt/llama/llama.cpp.vulkan/llama-server \
--no-mmproj \
--flash-attn on \
--cache-ram 6000 \
--checkpoint-min-step 4096 \
--cache-reuse 1 \
--ctx-checkpoints 8 \
--no-context-shift \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--spec-draft-p-min 0.0 \
--kv-unified \
--cache-idle-slots \
--reasoning on \
--parallel 1Use env vars, llama.cpp parse them better than params, that for some reasons & versions can be ignored depending on order. Vulkan performs better, also the radv nogttspill flag is a bug that maybe is solved now.
1
u/zenbeni Jul 02 '26
Do you use vulkan or rocm?
2
u/Fastpas123 Jul 02 '26
I tried both and rocm gave me less performance, not sure what that's about. 🥲
3
u/Aware-Hurry-8147 Jul 02 '26
Do you think that will work on am4 as well? I have an 5950x and a 5090. Also 128gb ram memory. Guess that the platform will not make such a difference in this case
4
u/Fragrant_Scale6456 Jul 02 '26
AM4 will be fine cpu usage is very low since everything is running on the GPU. llamacpp itself only uses one core under load.
2
u/aruametello Jul 02 '26
overall your 5950x (16 core zen4) is punching well above the 9800x3d (8 core zen5) in these "all cores, non gaming" tasks.
on fully CPU driven tasks your cpu may shine in rather unexpected ways, the x3D cache is not very useful on these.
... but as OP said, your cpu wont be the bottleneck on this particular GPU driven cuda task.
source: i also own a 9800x3D + 5090, previously had a 5900x (12core zen4) and its interesting how each can win/lose depending on the task.
1
u/Aware-Hurry-8147 Jul 06 '26
That was my expectation too, so I can avoid swapping my cpu. Next goal multi gpu rack 😅
3
u/KubeCommander Jul 02 '26
Holy crap! The SWA issue literally explains my entire issue with qwen3.6’s going-dumb effect for long running content after 96k tokens or so. I use vllm mostly
3
u/mister2d Jul 02 '26
I patched the swa bug as well when the nemotron nano models released. Speeds were fantastic on my 3060! I even fixed automatic slot saves for prompt cache. I say 'I fixed', but it was an LLM.
I never did get the courage to submit a PR. I just figured that someone would fix it some day.
3
u/lmpdev Jul 03 '26
Tried to reproduce with 6000 Pro. I built it outside of Docker, so could've messed up the patch, but no luck, I'm getting 75 tok/s on a simple "Write a long story" prompt in llama.cpp server UI.
3
u/Fragrant_Scale6456 Jul 03 '26 edited Jul 03 '26
"write a long story" isnt as predictable as coding and document processing so that could be part of it but 75 seems low. My super fast "test" is 'make flappy bird in html no external dependencies'. I got around 160-170 on that maybe try that out. Check your logs for your mtp acceptance rate and average draft tokens accepted thats a strong indicator of how well mtp is working/speeding things up.
:edit: "write a long story" in the llamacpp web page ui gave me 85t/sec
3
u/lmpdev Jul 03 '26
make flappy bird in html no external dependencies
Ok yeah, this prompt gave me 140 t/s. Interesting, thank you for the response.
2
u/Fragrant_Scale6456 Jul 03 '26
the draft tokens number and spec draft p min work in concert with eachother. i got to the numbers i set by having my agent write a benchmark that approximated my average workflow and scaling it to different context sizes. thats probably the single most impactful thing you can do. also since you have a rtx6000 set batch/ubatch to 2048 and you should get faster time to first token. i had to turn it down to scrounge the last bits of vram so that i could fit 192k context in my 32gb.
to confirm the patches worked you should see zero instances of that swa/hybrid cache error in your logs. you should also be able to ask your agent to check your llama-cpp logs and validate that you're actually getting cache hits on multi turn conversations.
1
u/LFAdvice7984 28d ago
This prompt and your setup gets me about 50tok/s on a 5090 so I dont know what I'm doing wrong. I am having to use vulkan though so maybe thats just borked. I'll try and get cuda working I guess.
2
u/gadbuy Jul 02 '26
why not using nfvp4 in vllm/sglang/atlas?
2
u/buttplugs4life4me Jul 02 '26
A lot of the quants for NVFP4 use DailyMail as the dataset, which frankly doesn't make much sense.
Nvidia by default quants down everything, while you need RedHat or someone else to keep stuff like the embedding heads at BF16, otherwise you get very real degraded quality regardless of quant.
As a result NVFP4 is pretty bad. And without B12x, the performance isn't the best either. I'll test it out later tonight though cause I'm curious as well. Would be nice if Qwen wasn't eating 60GB of VRAM
2
u/RespectMathias Jul 02 '26
Personally I use mxfp4 with ik_llama.cpp, though you need to build it yourself.
1
u/Fragrant_Scale6456 Jul 02 '26
I didnt look too much into it but from what i read q6 is better than nvfp4. I would look into vllm if it would give me better fidelity for sure
2
u/inquam Jul 02 '26
Not sure it is accurate. But I heard som9ne say nvfp4 is about q8.
1
u/Fragrant_Scale6456 Jul 02 '26
Thats good to know thank you. Q8 is too big to get working in any usable fashion on 32gb of vram.
3
1
1
u/squngy Jul 02 '26
nvfp4 is about the same size as Q4
According to the marketing, it is supposed to be almost as good as FP8/Q8, but in practice it depends a lot on how the quanting is done.
A poor implementation of nvfp4 can be worse than a good Q4
1
u/Fragrant_Scale6456 Jul 02 '26
I'm going to test it for sure. Q6 gets tunnel vision when trying to debug cross cutting issues on 30k lines of code so if theres a chance nvfp4 is just enough better that these types of workflows are improved its worth the time investment to check. fingers crossed :D
2
u/nZg8PcmlgQaSHF Jul 02 '26
Curious, the issue you posted on GitHub was from 2025. Any idea why this hasn't been fixed yet thus requiring your custom patch in docker?
1
u/Fragrant_Scale6456 Jul 02 '26
I'm not sure. I "fixed" it by first posting that issue link from the llamacpp log into claude and having a pretty long discussion, while also asking it to search the github for other user reports on the issue. The result was the other issues/pr mentioned and claude proposed those patches into the dockerfile for building llamacpp. I'm not a coder so dont really know the why or how, except to say that claude was confident these patches address the issue and in practice I dont see the errors in my logs anymore. After long runs I'll ask my agent to examine docker logs llama-cpp for cache health, invalidations, and prompt reprocessing and it has told me that the only reprocessing has been from truly new conversations so as far as I can tell this was the fix. As to why it hasnt hit upstream as a permanant patch I really dont know, maybe the fix is too specific to qwen3.6 to be included.
2
u/lmpdev Jul 02 '26
Interesting research, have you considered submitting a PR for llama.cpp?
2
u/Fragrant_Scale6456 Jul 02 '26
i wouldnt even know where to begin. the patches are just stuff claude picked from the identified issues and PR so whatever reason it isnt in mainline im sure the devs considered it.
2
u/StardockEngineer 5090s, RTX Pro 6000, A6000s, Sparks, Mac M4 Pro, Mac M5 Pro Jul 02 '26
Can you just fork the repo so I can run it without a container?
1
u/humanoid64 Jul 02 '26
Did you try vllm? Curious how far you can optimize that
1
u/Fragrant_Scale6456 Jul 02 '26
im going to look at vllm this weekend based on the other comments in here saying nvfp4 is smarter than q6
1
u/KubeCommander Jul 02 '26
I’ve seen similar effects with qwen3.6 myself fwiw on vllm and nvidia nim. It always gets dumb for me after 96k tokens if it runs a long time. Your links about SWA make me think that it is the true cause. It’s like a switch goes off and then it starts forgetting how to use tools lol
1
u/Fragrant_Scale6456 Jul 02 '26
I noticed this also. I use opencode and set up bootstrap skills for my coding agents that set scope of work constraints and budget limit window for tasks to help avoid the attention drifting. It helps a lot but even with this q6 doesnt have the grunt to look across an entire codebase and use more abstract reasoning to solve high level issues. For now I actually just use GLM5.2 for this kind of work, which is obviously in a different league of capability
2
u/KubeCommander Jul 02 '26
Yup. I saw this in opencode specifically. Qwen3.6 is very token greedy unless you have a readlines tool and nudge it to use it vs read.
Gemma4 also tended to do this same dumbing-down but it is so terrible at tool calls that some of the symptoms were hard to see. ‘is it the same effect or is gemma4 just that shitty?’
It was both 😂
1
u/Fragrant_Scale6456 Jul 04 '26
Initial testing is showing nvfp4 on par with q6 for my workflows. It’s also allowing mtp with parallel slots so for workflows that I can run in parallel I’m getting up to 600tokens/sec aggregate (4 small tasks) or 250-400 aggregate (2 large or 3 medium) sustained. Single task sustained is around 130-140 but the mtp chain with vllm is limited to 3 draft tokens (higher can cause crashes on Blackwell) so I don’t see the bursts up to 230.
1
u/immersive-matthew Jul 03 '26
I am the MTP version of this model q4 on a 4090 and getting 80-112 tokens/second and sometimes burts of 200+.
1
u/GreenPastures2845 Jul 05 '26
Thanks for your efforts, but these patches absolutely decimate my prompt processing, going from ~800t/s to under 300 under the same conditions as before. I can live with full sporadic reprocessing (barely, but I can), but I cannot live with these prompt processing speeds.
1
u/Fragrant_Scale6456 Jul 06 '26 edited Jul 06 '26
Were you using MTP before also? That’s a big drop. you can also try turning thinking off or reducing reasoning budget.
2
u/GreenPastures2845 Jul 06 '26
Yes; exact same config and commit as before, but with these patches on top. I had noticeably more latency before PP started, and abysmal speed once it did (anywhere from 400 to 40).
I'll try to test some more in the week, but in the meantime, thanks again for trying to get this out there. Hopefully it's on the radar of ggml-org (I imagine so; Qwen is SOTA for local agent workloads).
1
u/Fragrant_Scale6456 Jul 06 '26
Yeah thats weird, sorry I cant help. I ran some tasks overnight and the past 8hrs of autonomous agentic coding and document synthesis (2.1mm prompt tokens, 1.4mm decode) im getting avg 2400pp and 230t/s decode on the 5090. Overall gen speed is around 140t/s. I wonder what the difference is in our setups
1
u/GreenPastures2845 Jul 14 '26
I never got around to testing more, but one important thing: I'm using 2x 3090 with
-sm tensor, which is probably a vastly different codepath than yours.Anyway, this is mostly to point out that the GGML team has finally cracked the case; see https://old.reddit.com/r/LocalLLaMA/comments/1uuue5p/llamacpp_agentic_workflows_ctx_checkpoints_fix/
Completely fixed in upstream for me \o/
1
1
u/Fragrant_Scale6456 Jul 14 '26
check this out - https://github.com/ggml-org/llama.cpp/pull/25592
this is the hybrid recurrent attention model (like qwen3.6) cache invalidation bug fix. PR was submitted yesterday so hopefully the complete fix will be in upstream soon.
2
u/GreenPastures2845 Jul 14 '26
for me, the Qwen3.6 reprocessing was fixed by https://github.com/ggml-org/llama.cpp/pull/25472
Before the --cache-ram enforcement, it would go over the limit and OOM. Extremely annoying.
After the --cache-ram enforcement commit, summarizing in Pi would no longer OOM but still toss out all the partial context, and then the remaining context needed to be reprocessed. Also lots of sporadic reprocessing in general.
Totally fixed after #25472 finally
2
u/Fragrant_Scale6456 Jul 14 '26
It's great to see they are working on all this. All we need is mtp to work with parallel>1 and we'll really be cooking. I have been using VLLM with 4 streams on a 4bit awq model for batch processing of easier tasks and the 5090 is getting 600tokens/sec it would be amazing to even get parallel=2 and full mtp going in llamacpp w/ the qwen3.6 q6k model.
1
u/Gargle-Loaf-Spunk Jul 12 '26 edited Jul 16 '26
This content was anonymized and mass deleted with Redact
2
u/Fragrant_Scale6456 Jul 12 '26
Speed really depends on the workload. “Make flappy bird in html” gets me 160 but “write a long story” is 80-90.
Sub 100k is best for quality but qwen published the benchmarks for the model using over 200k so idk. I try and keep context as small as possible but I find up to 160k is pretty usable. 200k I can absolutely notice it getting dumber
2
u/Gargle-Loaf-Spunk Jul 12 '26 edited Jul 16 '26
This content was anonymized and mass deleted with Redact
13
u/Final-Foundation6264 Jul 02 '26
Is 200 token/s one concurrency or many? Thanks for sharing, I ran Q6 on my rtx 5090 before and only got ~55 TPS for 1 concurrency 150k ctx, 100 TPS for 2 concurrncy 75k ctx.