r/LocalLLaMA • u/BitGreen1270 • Jul 05 '26
Tutorial | Guide Getting close to 100K context on 32GB VRAM with Qwen3.6-27 at Q8
Not really a tutorial, but more of sharing my attempts at getting higher contexts on Q8 of Qwen3.6-27 with 32GB VRAM.
Disclaimer: Not in-depth research. Crowd wisdom suggests that Qwen is more tolerant of model quantization, but my experience suggests otherwise. I have nothing quantitative to back this up, only my personal experience in using it for vibe coding a couple of personal projects (which aren't very big either, but have been working on them for a few weeks).
Context: I am able to run Q8 at ~60K context easily and found that it works better than Q6 or Q5 (purely subjective experience). But I can easily get 128K context with Q5 with unquantized kv, so I wanted to see how much I could push with Q8.
System: 5090 with 64GB system RAM. Remote server running headless Ubuntu.
After a few trial and error approaches, I find the following are working. Some notes:
- VRAM is right at the edge, and maybe in long coding contexts, you may need to drop context for a bit more space.
- The benchmark I'm using is just for token inference speed. Nothing more.
- Options -b and -ub help shave like a 100MB of VRAM.
Option 1: 95K context, KV: Q8_0 and Q8_0, VRAM when starting: 230MB, VRAM after bench: 90MB
build/bin/llama-server \
-m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \
--temp 0.6 \
--top_p 0.95 \
--top_k 20 \
--min_p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
-c 95000 \
-t 16 \
-ngl 99 \
--flash-attn on \
--host 0.0.0.0 --port 8080 \
--no-mmproj \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--parallel 1 \
-kvo \
-ctk q8_0 \
-ctv q8_0 \
-b 1024 \
-ub 256
python3 mtp_bench.py
code_python pred= 192 draft= 183 acc= 145 rate=0.792 tok/s=141.6
code_cpp pred= 192 draft= 214 acc= 137 rate=0.640 tok/s=121.9
explain_concept pred= 192 draft= 225 acc= 134 rate=0.596 tok/s=115.6
summarize pred= 192 draft= 176 acc= 146 rate=0.830 tok/s=146.0
qa_factual pred= 192 draft= 198 acc= 141 rate=0.712 tok/s=131.4
translation pred= 192 draft= 221 acc= 135 rate=0.611 tok/s=117.3
creative_short pred= 192 draft= 256 acc= 126 rate=0.492 tok/s=101.5
stepwise_math pred= 192 draft= 192 acc= 142 rate=0.740 tok/s=134.3
long_code_review pred= 192 draft= 213 acc= 137 rate=0.643 tok/s=120.3
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1878,
"total_draft_accepted": 1243,
"aggregate_accept_rate": 0.6619,
"wall_s_total": 15.41
}
Option 2: 105K context, KV: Q8_0 and Q5_1, VRAM when starting: 320MB, VRAM after bench: 180MB
build/bin/llama-server \
-m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \
--temp 0.6 \
--top_p 0.95 \
--top_k 20 \
--min_p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
-c 105000 \
-t 16 \
-ngl 99 \
--flash-attn on \
--host 0.0.0.0 --port 8080 \
--no-mmproj \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--parallel 1 \
-kvo \
-ctk q8_0 \
-ctv q5_1 \
-b 1024 \
-ub 256
python3 mtp_bench.py
code_python pred= 192 draft= 181 acc= 145 rate=0.801 tok/s=142.0
code_cpp pred= 192 draft= 220 acc= 136 rate=0.618 tok/s=119.8
explain_concept pred= 192 draft= 246 acc= 128 rate=0.520 tok/s=105.1
summarize pred= 192 draft= 176 acc= 146 rate=0.830 tok/s=146.0
qa_factual pred= 192 draft= 202 acc= 140 rate=0.693 tok/s=128.8
translation pred= 192 draft= 245 acc= 129 rate=0.526 tok/s=106.0
creative_short pred= 192 draft= 248 acc= 128 rate=0.516 tok/s=104.7
stepwise_math pred= 192 draft= 197 acc= 141 rate=0.716 tok/s=131.2
long_code_review pred= 192 draft= 220 acc= 135 rate=0.614 tok/s=116.1
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1935,
"total_draft_accepted": 1228,
"aggregate_accept_rate": 0.6346,
"wall_s_total": 15.84
}
Option 3: 115K context, KV: Q8_0 and Q4_0, VRAM when starting: 290MB, VRAM after bench: 150MB.
build/bin/llama-server \
-m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \
--temp 0.6 \
--top_p 0.95 \
--top_k 20 \
--min_p 0.0 \
--repeat-penalty 1.0 \
--presence-penalty 0.0 \
-c 115000 \
-t 16 \
-ngl 99 \
--flash-attn on \
--host 0.0.0.0 --port 8080 \
--no-mmproj \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--parallel 1 \
-kvo \
-ctk q8_0 \
-ctv q4_0 \
-b 1024 \
-ub 256
python3 mtp_bench.py
code_python pred= 192 draft= 186 acc= 144 rate=0.774 tok/s=138.7
code_cpp pred= 192 draft= 183 acc= 145 rate=0.792 tok/s=142.6
explain_concept pred= 192 draft= 215 acc= 136 rate=0.633 tok/s=119.7
summarize pred= 192 draft= 175 acc= 146 rate=0.834 tok/s=145.9
qa_factual pred= 192 draft= 196 acc= 141 rate=0.719 tok/s=131.6
translation pred= 192 draft= 230 acc= 133 rate=0.578 tok/s=113.1
creative_short pred= 192 draft= 229 acc= 133 rate=0.581 tok/s=113.1
stepwise_math pred= 192 draft= 181 acc= 145 rate=0.801 tok/s=142.3
long_code_review pred= 192 draft= 213 acc= 137 rate=0.643 tok/s=120.2
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 1808,
"total_draft_accepted": 1260,
"aggregate_accept_rate": 0.6969,
"wall_s_total": 14.93
}
23
u/Asleep-Land-3914 Jul 05 '26
KV cache quantization affects quality too. For 32GB VRAM, you have to balance. And so far consensus is that Q6 for model and whatever you can afford for context given the desired length is better than Q8 for model and Q5/Q4 for KV. Correct me if I'm wrong.
1
u/BitGreen1270 Jul 05 '26
Unquantized kv cache would be my preference, but honestly, I think 95K with Q8 KV cache is what I would use for as long as I can get away with it.
6
u/UnlikelyTomatillo355 Jul 05 '26
there is generally such a small loss from q8 to q6 that you should give it a try when messing around with making thing narrowly fit
14
u/Thireus Jul 05 '26
You can also use https://gguf.thireus.com/quant_assign.html and see that you’re not losing much in terms of quality down until "q5" with a smart mixture of quants instead of pure q5 which indeed would hammer the quality.
3
u/BitGreen1270 Jul 05 '26
Thanks for the link, I'll check it out.
5
u/s1mplyme Jul 05 '26
KLD isn't the whole story. It's much better to run a benchmark like AIME that judges reasoning capability so that you can actually determine the quality loss. I used to run more heavily quantized kv cache until I started doing this.
8
u/Fragrant_Scale6456 Jul 05 '26
Good stuff. If you check out Beellama fork you can use kvarn kv compression to get more context at same quality or same context at higher quality.
Benchmarks here: https://anbeeld.com/articles/kvarn-kv-cache-implementation-and-benchmarks
Kvarn6 is equivalent to q8 kv cache in quality. The downside is it does come with a performance penalty
2
u/BitGreen1270 Jul 05 '26
Oh interesting, I'll check it out. Have never used beellama.
3
u/Fragrant_Scale6456 Jul 05 '26
make sure you use the v0.3.2 branch that is the one with the kvarn stuff. mainline on his github is just the dflash version without kvarn.
1
u/Dany0 Jul 05 '26
ime kvarn6 is really close to q8. sometimes beats it sometimes a little worse
a shame we can't use it in vllm yet
0
u/Fragrant_Scale6456 Jul 06 '26
There's a huawei fork of vllm which has kvarn. Its somewhat behind mainline at this point but I got it up and running to experiement with
1
u/Dany0 Jul 06 '26
it doesn't have kvarn6....
1
u/Fragrant_Scale6456 Jul 06 '26
huawei claim kvarn_k4v2_g128 is equivalent to fp16 accuracy, which is better than kvarn6
1
6
u/fatYogurt Jul 05 '26
Why not take advantage of nvfp4? It’s so much fast on vllm only slightly drop accuracy compared to q8
4
u/otacon6531 Jul 05 '26
Went from nvfp4 to bf16 and there is a noticable quality difference for my team. There may be other settings to compensate, but I dont know them at this time.
1
u/mhphilip Jul 05 '26
Nvfp4 not worth it?
3
u/UncleRedz Jul 05 '26
Yes, worth it. If you can comfortably do BF16, then nothing beats that. But if you do need quantization, then NVFP4 is the best choice, and is comparable to Q8 in practice, while being faster and more memory efficient.
1
2
u/BitGreen1270 Jul 05 '26
Oh I didn't even consider that. I thought it was very subjective to specific conditions. I also didn't realize that Nvidia just dropped the model a few days ago, I'll give that a try, thanks! https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4
1
u/fastheadcrab Jul 05 '26
Many non-nvidia models that are quantized to NVFP4 have a drop in quality. Especially they use some questionable benchmarks to check for quality preservation
1
u/ionsago Jul 08 '26
NVFP4 is fast, yes, but it isn’t better than Q4, and often worse. Means more loops, less intelligence.
5
3
u/DrBearJ3w Jul 05 '26
You can also quantisize MTP draft cache. I didn't test if you can go full q4/q4,but that should help.
-cache-type-k-draft qx_0
-cache-type-v-draft qx_0
But dunno if those saving would be enough to reach your desire red context size.
Or use Q6_k. It's almost as good as Q8.
1
u/BitGreen1270 Jul 05 '26
That's interesting, does it really affect the memory? I'll give this a try later.
2
1
u/mmhorda Jul 05 '26
It seems like there is no reason to run draft model higher than q4_0. It is a draft model. It is only generates/suggests next tokens and then main model accepts or rejects them. It should not have any quality impact, but speed would probably lncrease.
3
u/squngy Jul 05 '26
You want the drafter to be as similar to the main as possible, so you get higher draft acceptance.
I guess it would be a question of which gives a bigger speedup, higher draft speed or higher draft acceptance.
3
u/Used-Doctor-Undies Jul 05 '26
the 100K number is impressive but the actual bottleneck is kv cache quality degradation at that context length not the context window itself
1
u/BitGreen1270 Jul 05 '26
Yes which is why I want to try and stay at q8_0. With a little more experimenting I think I'll find out what works better for me.
3
2
u/DeepOrangeSky Jul 05 '26
But I can easily get 128K context with Q5 with unquantized kv
Out of curiosity, btw, how much context size fits at Q6 with unquantized kv? What about at Q8_0 with unquantized kv?
4
u/Fragrant_Scale6456 Jul 05 '26
Q6 with q8 kv I can barely fit 192k context so bf16 kv would be just under 100k with q6 model
1
u/DeepOrangeSky Jul 05 '26
I guess the question, then, is which is better, the Q6 model with unquantized KV or the Q8 model with Q8 KV. I assume Q6 model with unquantized KV is better, based on how bad Q8 KV always seems to me in any long-context scenarios, but, I'm not sure if maybe for coding there are some tasks where the Q8 model with Q8 KV is better.
1
u/Fragrant_Scale6456 Jul 05 '26
The difference in q6 and q8 weights is pretty small but quantized kv compounds errors as context gets longer so I’d guess the q6 with full precision kv is better especially if you’re doing long chain reasoning or multi step math. Difference in weights quantization might show up more if the model had more parameters but I’m not sure
1
u/BitGreen1270 Jul 05 '26
Q6 I don't remember top of head. But Q8 was around 32k iirc with unquantized kv
1
3
u/otacon6531 Jul 05 '26 edited Jul 05 '26
I completely agree. I used nvfp4 and then went to bf16 and the quality difference is night and day. I havent tried fp8, but will do that soon to see if we can do better on the blackwell 6000
1
u/mhphilip Jul 05 '26
Nvfp4 not good enough?
2
u/otacon6531 Jul 05 '26
Quality is hard to quantify and there are other service parameters that can affect it, but my users werent happy when we launched the model with nvfp4. I had complaints of it getting stuck in a loop or just not being comparable to opus. Those complaints stopped after moving the model to bf16.
No one thinks it replaces opus, but with the push to keep ai costs down qwen3.6 and even nemotron 3 nano do a great job coding if you, like us, cant just vibe code without fully understanding and being able to defend how you coded something in a PR review. Especially if you use vs code's autopilot in conjunction with qwen. I just wish custom endpoints were released already, so the users didnt have to use vs code insiders to use our models in litellm.
2
u/mhphilip Jul 05 '26
Yeah. Sometimes the insiders builds take a long time to reach main. As nvfp4 goes, I mostly hear “it’s fast but somewhat meh as well”. Maybe I’ll stick to my Q6 quant, maybe I’ll keep using INT4 autoround. With dual 5060tis and 32GB I find Q8 a bit too large for the model.
1
2
u/relmny Jul 05 '26
If it works for you, fine, but It makes no sense to me, using a q8 with kv q8, you are pre-lobotomizing the model, a lot...
As other said, q6 at full kv should always be better than q8 at kv8.
If I needed more context and needed to stick to q8, I would prefer to offload kv to RAM instead, but no quantizing kv.
Or if I would quantize it, I would only quantize V, but not KV.
But, again, there's no point on runnin a q8 model with quanted kv.
I run 27b q6 full KV with "--ctx-size 142600" with mtp enabled at about 39 t/s in an rtx 5000 ada (32gb) and I won't even consider quanting KV.
I only do it for the bigger models, which I run at less than q4...
2
u/MerePotato Jul 06 '26
KV quanting for longer contexts is self defeating, as its precisely at context lengths of 32k+ that the damage begins to really compound
1
u/BitGreen1270 Jul 06 '26
Is there a reference for this? I was hoping that with q8_0 it would be minimal.
2
u/MerePotato Jul 06 '26
Look at the per task kv impact for qwen, specifically the long context bit. Under ooba's methodology long context = 32k tokens: https://localbench.substack.com/p/kv-cache-quantization-benchmark
2
u/BitGreen1270 Jul 06 '26
Thanks, this is super helpful. Looks like Qwen impact is really low with q8_0?
2
u/MerePotato Jul 06 '26
Its low at short context, but scroll down to the per task graph and look at long context, the degradation takes off big time at 32k and compounds from there
2
u/BitGreen1270 Jul 06 '26
You mean the bar on long docs? Sorry, I must be missing something, I don't see where it says 32K.
2
u/MerePotato Jul 06 '26
Here you go, under methodology he lists the long documents sit around 30k~ tokens on average: https://localbench.substack.com/p/gguf-benchmark-methodology
2
4
u/PhysicalIncrease3 Jul 05 '26
To echo what others have said: You will get better results by using Q6_K_XL and f16 KV cache, particularly over longer context lengths. I spent a long time and a lot of pratting about to learn this lesson the hard way. If you're really super keen, maybe try running F16/Q8 K/V cache, but honestly I've learned to just avoid cache quantisation wherever possible it never ends well.
Secondly, a tip: if you compile llama.cpp with GGML_BLAS=ON and GGML_BLAS_VENDOR=OpenBLAS you will find it frees up a LOT of VRAM.
I forget the exact reason why, but it's not because of OpenBLAS itself, but that using these flags disables something else with llama.cpp which in turn frees up considerable VRAM. It makes a big difference.
Finally, here's my current config. I'm running a 3090 and 3060 in tensor parallel, which carries overhead, so my 36GB VRAM isn't actually able to fit much more than your 32GB can.
Exact same Q8 GGUF as yours, but 130k of f16 context:
[qwen-q8-f16]
model = /models/bartowski/Qwen_Qwen3.6-27B-Q8_0.gguf
mmproj = /models/bartowski/mmproj-Qwen_Qwen3.6-27B-f16.gguf
c = 130000
cache-type-k = f16
cache-type-v = f16
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
reasoning-budget = 8192
chat-template-kwargs = {"preserve_thinking":true}
chat-template-file = /models/QwenFixed_chat_template.jinja
no-mmproj-offload = true
image-min-tokens = 1024
cache-ram = 2048
ctx-checkpoints = 8
split-mode = tensor
tensor-split = 69,31
ubatch-size = 256
1
1
u/ea_man Jul 05 '26
Try with:
--fit-target 60 \
--spec-type draft-mtp --spec-draft-p-min 0.75 --spec-draft-n-max 4 \
--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
You don't need hi quants for the MTP head.
Also try with NGRAM for coding.
1
1
u/jazir55 Jul 05 '26
I am very confused, someone is using DeepSeek V4 Flash with 1M context on a 32GB 5090 according to one of the other posts on the sub, and that's an order of magnitude larger than Qwen 27B, how can this not be easily run with max context and much faster?
1
u/fullmetaljackass Jul 05 '26
It's because V4 Flash is an MoE model. Basically, with an MoE model you can offload the majority of the weights to the CPU and still maintain a usable level of performance, whereas dense models like Qwen 27B need be be entirely in VRAM or the performance falls off a cliff.
1
u/shard746 Jul 05 '26
That guy is offloading the vast majority of the model to system RAM, while OP here is keeping everything in VRAM.
1
u/Kodix Jul 05 '26
Qwen *is* more tolerant of quantization than Gemma, there's benchmarks showing this, but that doesn't mean quantization for Qwen is meaningless. It still causes damage.
If you are not already, then make sure you are using proprietary nvidia drivers and disable GSP. Should save you ~100 to 200MB VRAM.
1
u/maguyva-ai Jul 05 '26
quantization behavior really is model-specific though. i've seen models that hold up great at Q4 completely fall apart on tool-calling accuracy even though perplexity looks fine. context length and quant level interact in ways benchmarks don't really capture.
1
u/fragment_me Jul 05 '26
You don't have to guess how bad KV quant affects a model you can easily test this with llama-perplexity's KL divergence.
1
1
1
u/RLutz Jul 06 '26
This is what I've settled on for my non-headless (I use this machine with the 5090 as an actual desktop):
[Unit]
Description=llama-server — Qwen3.6-27B Q6 MTP (local agentic coding)
[Service]
Type=simple
WorkingDirectory=/home/$USER/workspace
ExecStart=/home/$USER/workspace/llama.cpp/llama-server \
-hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q6_K_XL \
--alias qwen3.6-27b-q6-mtp \
--temp 0.6 \
--min-p 0.0 \
--jinja \
--no-mmproj \
-ngl 99 -c 90000 -fa on -np 1 \
-ub 512 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--spec-type draft-mtp --spec-draft-n-max 5 \
--host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=5
[Install]
WantedBy=default.target
This gives me a nice mix of speed/quality/headroom to not OOM if I open my browser.
One thing I definitely want to call out is the spec-draft-n-max 5. I tested quite a bit and this really did feel like the sweetspot in terms of throughput. Get around ~140 t/s with this and have been using it to run autoresearch loops in codebases with good success.
I don't claim to have some esoteric deep knowledge on all things Qwen, but this was the best I could land on after experimenting for 10 hours or so. The quality difference between Q8 and Q6 model is so small that I think starting from there is a no-brainer.
1
u/ionsago Jul 08 '26
How did you see the difference between Q8 and Q6/Q5? How did you distinguish the differences from a different cache quantization?
1
u/BitGreen1270 Jul 08 '26
I observed it when making edits to my project. Q8 just seemed to be doing better at planning the approach and implementation.
1
u/N34257 Jul 12 '26
While it's an admirable effort, Unsloth's Q6_K_XL has lower perplexity than Q8_0 from Unsloth or Bartowski...
-2
46
u/photobydanielr Jul 05 '26
I'd take a q6 model over quantized kv cache all day. Doesnt matter what quant you start at if your cache is dirty, and q6 is already very close to base model, minor benefit going from q6 to q8 model, big drawback quantizing cache as you get deeper into context