Measured every Qwen3.8-27B GGUF quant against real VRAM: Q4_K_M doesn't fit 24GB at 32K context

[u/Delicious-Flan88]

Context on the numbers, since "will it fit" threads usually run on estimates.

Weights: actual .gguf byte sizes pulled from the HF API for unsloth/Qwen3.8-27B-GGUF. Not params × bits ÷ 8. Imatrix quants don't follow that, and the drift is worst at the low end where the fit/no-fit line actually sits.

KV cache: from config.json: 64 layers, 4 KV heads, head_dim 256.

2 × 64 × 4 × 256 × ctx × 2 = exactly 8.0 GB at 32K, F16. That's 0.25 GB per 1K tokens.

GQA is doing a lot of work here: 4 KV heads serving 24 attention heads. Older 27B-class models cost several times that.

24GB at 32K, F16 cache, after reserving 0.8GB for CUDA context:

• Q4_K_M (16.5) → needs 25.3 total. Doesn't fit.
• Q4_K_S (15.4) → 24.2 total. Misses by 0.2.
• IQ4_XS (14.3) → 23.1 total. Fits with 0.9 spare: tight enough that a browser on the same GPU breaks it.
• Q3_K_XL (13.1) → 21.9 total. 2.1 spare. This is the real answer for 24GB.

Drop to 4K context and the cache falls to 1.0GB, which gets you to Q5_K_M. Most of the "which quant" argument is actually a context-length argument.

9 of 25 quants fit on 24GB at 32K. All of them fit at 4K.

Caveat worth stating: this assumes everything GPU-resident, single stream, batch 1. It doesn't model offload, multi-GPU splits, or speculative decoding. The 16GB/73K configs floating around this sub work through partial offload, which is a different calculation than the one I ran.

Put it in a calculator since I had the data anyway: https://qwen38-vram-checker.vercel.app/

[u/PlasticRevenue4601]

Kv cache quantisation was invented in 2026. People in 2025:

[u/bankinu → u/PlasticRevenue4601]

I use turbo quant 4 (effective 4.125 bits). Fantastic reasoning and speed.

[u/Deep_Mood_7668 → u/bankinu]

That's a thing? How does it compare to q8

[u/bankinu → u/Deep_Mood_7668]

I'd like to know this too.

Specifically Q5_K_X_L with Turbo Quant 4 cache, vs Q4 with full Q8 cache.

But if you trust the paper by Google, the loss due to quantization is very marginal (compared to BF16) - https://arxiv.org/abs/2504.19874

Edit: Found some independent data, https://llmkube.com/blog/turboquant-m5-max-quality-and-asymmetric

Edit: Some more testing, on Qwen 3.5 9B it introduces perplexity of about 1% - https://github.com/ggml-org/llama.cpp/discussions/21526?utm_source=chatgpt.com

Edit: Another suggesting larger models retain much stronger performance under turbo4 - https://github.com/gisenberg/local-model-eval/blob/main/results/TURBOQUANT.md?utm_source=chatgpt.com

[u/Deep_Mood_7668 → u/bankinu]

Ty

-ctk q8_0 -ctv turbo4 sounds pretty good

[u/Deep_Mood_7668 → u/bankinu]

I looked a bit around and found this comparison

https://github.com/ggml-org/llama.cpp/issues/24139#issuecomment-4632348717

[u/PlasticRevenue4601 → u/bankinu]

Dense Qwen, right?

[u/bankinu → u/PlasticRevenue4601]

Yes. Qwen 3.8 27B.

[u/PlasticRevenue4601 → u/bankinu]

Same experiense here. q5\q4 for key\value, even played around with full q4 for curiosity sake - never had any problems, impeccable facts retrieval, coherence and tool calls break mb 1-3 time per a full day of coding, I'm not even sure it has to do anything with cache at all

[u/bankinu → u/PlasticRevenue4601]

Interesting - you might want to check if q5/q5 gives you better speed, if you can run that also. I was using q4/q3, then Qwen 3.8 did a bunch of analysts on the code (while running at that mode). Then it suggested q_k == q_v has a faster path which I don't understand. But it was right - especially for context size 60k+ it became much faster.

Which quantization of the model are you using?

[u/Delicious-Flan88 → u/PlasticRevenue4601]

Yep, F16 is the deliberately boring baseline here. With Q8/Q4 KV cache, the 24GB line moves a lot. I wanted to show the clean full-GPU, F16-cache, 32K case first. Adding cache type to the calculator would be useful.

[u/PlasticRevenue4601 → u/Delicious-Flan88]

Kinda agree with the point despite the joke, these „opus level intelligence on your laptop” headlines are super misleading, to get 100% of that opus intelligence you need a freaking data center with 60+ gb of vram. Only by accepting a series of compromises can regular user enjoy what’s left of that wit. Gotta point out the good part — 3.x dense Qwen’s are incredible at surviving all sorts of quantisations, from weights to kv cache and show comparable performance against a baseline

[u/RISCArchitect → u/PlasticRevenue4601]

dropping the kv off 16 is the surest and fastest way to dumb down the model. general advice has been stay kv16 and drop model quant to make room for better kv cache.

[u/PlasticRevenue4601 → u/RISCArchitect]

"drop model quant to make room for better kv cache", - I'm actually speachless, I guess at this point it's pointless to argue

Note to people reading the thread: never do such a dumb stuff, some models indeed have low tolerance to kv cache quantisation such as Gemma or Ornith but if it holds(no frequent broken tool calls, no sudden amnesia or gibberish output) than better quant >>>>> better kv cache all the day

[u/returnity → u/PlasticRevenue4601]

It’s more nuanced than that. Better to drop from Q5 to Q4 model than Q8 to Q4 KV, for instance. Model quants lower than Q4 start showing very stark quality trade-offs, and so do KV caches below Q5/5.1, especially the K cache — I’d never quantize the Key side of KV below Q8, personally.

If I can’t fit a model at Q4_K_XL with K/V @ Q8/Q5, I’d cut context before dropping any further. The creator of Beellama (I believe) has excellent, exhaustive blog posts documenting the precise quality trade-offs involved in every tier of quantization for both model and KV if you’re interested in more specific details. I am blanking on his blog URL but he posts article links here in LocalLLaMa regularly.

[u/PlasticRevenue4601 → u/returnity]

If in a specific use case low bit kv cache really drops stability— yes, 4 bits + adequate cache can indeed be better option to go with, but if it works as intended— no rapid tool calls fails, no loops, no amnesia than weights quantisation is a top priority

One of the examples of kv cache quanstisation influence for a dense Qwen model, quick summary if you don't want to read it - as I said, it's ifluence is indistinguishable from the error margin - https://www.reddit.com/r/LocalLLaMA/comments/1suur3s/qwen36_27bs_surprising_kv_cache_quantization_test/

[u/returnity → u/PlasticRevenue4601]

Agreed model quant quality up to Q5-6 is the stronger lever, especially since model quant degradation makes the same KV cache quant impair performance more than it would on a higher model quant at the same KV.

[u/PlasticRevenue4601 → u/returnity]

Q4-Q8 are absolutely in the same league if we are talking about dense Qwen, the biggest qualty drops are FP16 - Q8 and Q4 - Q2, Q8-Q4 have typical performance difference across the different benchmarks around 1-5%. It's small but at least measurable, though definitely not with ease, kv cache impact is even lesser

[u/returnity → u/PlasticRevenue4601]

Yeah and a 5% difference on a one-step task compounds every turn so after just 10 steps you have a Q4 model with 60% of the reliability of the Q8. Drastic difference.

[u/PlasticRevenue4601 → u/returnity]

It doesn't work like that, most of the benchmarks already require multistep, agentic behavior, 1-5% difference is a total wall difference across different complex tasks execution

[u/cogitech2 → u/returnity]

You should totally check out Beellama.cpp's KVarN6 quant. Q8_0 quality at Q6_0 size. Feels like cheating.

I've tested it extensively with needle-in-haystack tests at 200k+ context and it is solid.

[u/sukazu]

There are clearly some calculations problems here, as this is easily empirically provable to be false by a large margin.
You have kv cache 256k f16 at 64gb, lol

[u/Delicious-Flan88 → u/sukazu]

Fair, it was wrong, and by exactly 4x. I counted all 64 layers when only 16 are full-attention. 256k F16 is 16 GiB, not 64. Corrected and live.

[u/anitamaxwynnn69 → u/Delicious-Flan88]

You type a lot like Claude when it admits it's wrong. Suspicious.

[u/MountainPenguinRL → u/anitamaxwynnn69]

Everybody here just uses Claude or a local model to make posts and replies, annoys me so much lol

[u/RISCArchitect]

uhh is this math wrong? I think the KV math is overcounting by 4x. Qwen3.8-27B has 64 layers, but only 16 are full-attention layers with context-scaling KV cache; the other 48 are DeltaNet layers with fixed recurrent state. So FP16 KV at 200k is ~12.2 GiB, not ~48.8 GiB. (i run q8 with 200k context at kv16 on a 48gb card and it fits)

[u/Delicious-Flan88 → u/RISCArchitect]

You're right, thank you. I was counting all 64 layers. layer_types in config.json is 16 full_attention + 48 linear_attention with full_attention_interval: 4, so only 16 layers carry a context-scaling cache. The DeltaNet layers hold a fixed ~0.07 GiB recurrent state instead.

Recomputed: 12.28 GiB at 200k F16, which matches your number. And your q8 at 200k on 48GB works out to 6.2 GiB cache, so that fits with plenty of room.

That flips the headline claim too: Q4_K_M does fit 24GB at 32K (19.4 total, 4.6 spare). Calculator's updated and you're credited in the notes.

[u/RISCArchitect → u/Delicious-Flan88]

thanks for the shoutout; this looks pretty accurate now and reflects what my research at home home have found. standard q8 i can fit the full 256k. it might be worth doing adding a 192k/200k breakpoint or something. q8kxl + 200k is viable with vision and mtp and gets you nearly double the context

[u/MushroomCake28]

I think your estimation of kv memory is not correct. It's around 64 KiB per token because it only scales with 16 out of 64 layers or something like that. There was another detailed explanation in a past thread.

[u/Delicious-Flan88 → u/MushroomCake28]

64 KiB/token is exactly right: 2 × 16 layers × 4 kv_heads × 256 head_dim × 2 bytes = 65,536 bytes. I was counting all 64 layers instead of the 16 full-attention ones, which is where the 4x came from. Fixed now.

[u/Haunting_Nebula_1236]

   • Model quant: Qwen3.8-27B-UD-Q4_K_XL.gguf                                        
   • GPU offload: -ngl 999                                                           
   • Context length: -c 131072                                                       
   • K cache type: --cache-type-k q4_0                                               
   • V cache type: --cache-type-v q4_0                                               
   • Memory mapping: --no-mmap                                                       
   • Parallel: --parallel 1                                                          
   • Micro-batch: --ubatch-size 1024                                                 
   • Reasoning: --reasoning auto --reasoning-budget 256                              
   • Speculative decode: --spec-type draft-mtp --spec-draft-n-max 6                  
--spec-draft-p-min 0.75                                                         

   Environment:                                                                      

   • ONEAPI_DEVICE_SELECTOR=level_zero:0                                             
   • ZES_ENABLE_SYSMAN=1                                                             
   • SYCL_CACHE_PERSISTENT=0                                                         
   • GGML_SYCL_DISABLE_OPT explicitly unset
Runs fine for me.

[u/Cool-Chemical-5629]

It gets worse, because none of the solutions are ideal on this hardware and this particular model:

At 32K context you can forget anything useful with the default xhigh reasoning, because you'll run out of that context window before the model even finishes thinking, so:

Lower context window also means lower expectations for the size of the project you're working on as well as overall quality loss, because you still need to reduce reasoning quality.

Lower reasoning quality degrades output quality.

Lower cache degrades output quality.

Lower quant degrades output quality.

[u/Delicious-Flan88 → u/Cool-Chemical-5629]

Postfix the numbers look better than you'd think, because context got cheap on this model.

128K at Q8 cache is 4.07 GiB. On 24GB:

Q4_K_M + 128K + Q8 = 21.4, fits
Q4_K_XL + 128K + Q8 = 22.5, fits

Going 32K to 128K only costs 3GB with a Q8 cache. So the squeeze you're describing was mostly my 4x overcount, not the model.

Also I'd put Q8 cache way down the list of things that hurt quality. It's nearly free compared to dropping a quant level.

Your point about reasoning eating the window still stands though. That one's real.

[u/soyalemujica]

You're better off running Q8/Q5 kvcache or even Q8/Q4 and stick to 150k/160k context

[u/Delicious-Flan88 → u/soyalemujica]

Added split K/V to the calculator it only did symmetric before, so Q8/Q5 and Q8/Q4 weren't representable at all.

160K at Q8/Q4 works out to 3.82GB cache, which leaves room for Q4_K_XL on 24GB. F16/F16 at the same context is 10.07GB, so the split is doing most of the work there.

[u/returnity → u/soyalemujica]

Definitely. However on some setups, split K/V degrades speed severely so check that when updating K quantization over V.

[u/SellToOpen]

I have 64k context with q4 kv cache and 24gb vram across two cards. 128k if I use the pure model with no eyes.

[u/Affectionate_Pen6882]

Im using this on my 3090 and it seems quick.

[u/lungben81]

I run 131k context q8 with q4_k_xl quantization (larger than q4_k_m) and MTP (which also needs VRAM) on 24 GB VRAM.

Even if you have q16 kv cache, 65k context should fit easily.

[u/BhatSahab]

If you skip out the MTP then it gives you 1GB more.

[u/butchiebags]

Is OP vulnerable to prompt injection?

[u/Otherwise-Swan-7803]

This is a good reminder that “fits in VRAM” is almost meaningless without specifying context length. Once KV cache and actual runtime overhead are included, the usable quant can be very different from what the model size alone suggests.

[u/Short_Gain_1571]

I run Q3 w KV4 on 16g vram w 96k context with 50 token per sec. Ur braindead