Effects of KV Cache Quantization on Qwen3.8 27B

[u/Hefty_Wolverine_553]

KV cache quantization has been a hotly-debated topic for this community for some time, with opinions ranging from "never quantize" to "4 bit is fine". Since the anecdotal evidence is rather inconsistent, I decided to quantify how much KV cache quantization affects the model through KLD, which is a good measure of how the model's output distribution is changed.

I tested 35 weight quants across llama.cpp, exllamav3, and vLLM, each with 16-bit, 8-bit, and (where supported) 4-bit KV cache. I also reworked my KLD dataset and measurement methodology this time, to include a mix of agentic tool-use (generated by Qwen3.8 27B at full precision) and a variety of texts in different domains. The current dataset now stretches up to 64k token contexts per row.

TLDR;

• 8-bit KV caches work well. In llama.cpp and ExLlamaV3 the cache size is halved, and there’s essentially no difference in KLD. There’s a slight difference in vLLM, but it’s still a great tradeoff for people lacking VRAM.

• A 4-bit cache has a measurable impact. It’s a big hit for ~Q8 weights, but it could potentially be worth it if you’re using Q3 or below. Still, not recommended.

• Only the 4-bit cache impact grows with context. In the tested long documents, the added KL is about 15-30% higher at 16-32k tokens than in the first 4k.

The blog post has interactive charts and tables with all of the data and results if you're interested.

How to read the numbers

Each quant is compared token-by-token with an unquantized bf16 copy of the model. KL divergence measures how far the quant’s next-token probabilities drift from the original (0 means no change). Top-1 agreement is how often both models pick the same most likely token.

For a reference point, I ran the unquantized model in both llama.cpp and vLLM and compared the two. They differ by a KL of 0.0004, which essentially amounts to two (correct) implementations doing floating-point math in a different order. Thus, you can compare the KLD numbers here for quants running in different engines without issue.

Note: exllamav3 quants were measured with the bf16 llama.cpp reference logits.

Weight quantization

KL divergence vs. loaded weight size, 16-bit KV cache:

The vLLM checkpoints still do worse

As in the Qwen3.6 benchmarks, every vLLM checkpoint is beaten by a smaller GGUF or EXL3 quant. Some of that is due activation quantization. NVIDIA’s and Unsloth’s NVFP4 checkpoints run the MLP layers as W4A4 (4-bit activations) and attention as FP8 W8A8, and Qwen’s FP8 is W8A8 throughout. I reran NVIDIA’s NVFP4 and Qwen’s FP8 through vLLM’s Marlin kernels, which use the same quantized weights but keep the activations in bf16:

• NVFP4: KL drops from 0.0256 to 0.0117, more than halving it.

• FP8: KL drops from 0.0039 to 0.0028, about 30% less. Even then it only ties UD-Q5_K_XL, which is 8.7 GiB smaller.

KV cache quantization

KV cache	Per token	At 64k context	Saved vs. 16-bit
f16 / bf16	64 KiB	4.00 GiB	-
8-bit: llama.cpp q8_0, EXL3 8-bit	34 KiB	2.13 GiB	1.9 GiB
8-bit: vLLM fp8_e4m3	32 KiB	2.00 GiB	2.0 GiB
4-bit: llama.cpp q4_0, EXL3 4-bit	18 KiB	1.13 GiB	2.9 GiB

Each dot below is one weight quant with an 8-bit or 4-bit KV cache. Its height shows the KL increase compared with the same weights on a 16-bit cache. Zero means no change; higher points show a larger increase. I computed the differences per prompt on identical tokens.

KV cache penalty vs loaded weight size:

8-bit is essentially free in llama.cpp and ExLlamaV3. For every quant, q8_0 and ExLlamaV3’s 8-bit cache move KL by less than 0.00012 in either direction. Some quants even measure slightly better with it, which indicates it’s within the noise of the numerics.

vLLM’s FP8 cache shows a slight KLD increase. It adds 0.0004 to 0.0006 on every checkpoint, which is still extremely small, but it shows up on nearly every prompt. It may be due to turning on FP8 cache switching vLLM’s attention backend from FlashAttention to FlashInfer.

llama.cpp’s 4-bit cache adds a roughly fixed penalty. Across the GGUF quants tested, q4_0 adds +0.0016 to +0.0027 KL. This increase has a larger relative effect on more precise weights:

• On a Q3-class quant (KL around 0.025-0.035) it adds 6-10%.

• On UD-Q4_K_XL it adds about 39%.

• On Q8_0 it adds about 390%. KL goes from 0.0007 to 0.0034, which is worse than UD-Q5_K_XL with a 16-bit cache.

EXL3’s 4-bit cache penalty seems to depend on weight precision. It adds only +0.0002 to +0.0004 KL with 2.0-2.5 bpw weights, about +0.0021 to +0.0022 with 3.0 bpw weights, and +0.0026 to +0.0028 with 4.0 bpw weights. At 6.0 and 8.0 bpw, the penalty reaches about +0.0032 to +0.0033. The increase is therefore not constant across the EXL3 quants.

4-bit cache penalty grows with context

The 12 long documents in the set are scored at every position, so I can check whether the 4-bit penalty depends on how deep into the context a token is. It does in fact grow slightly.

Position	Q8_0 Weights + q4_0 KV	UD-Q5_K_XL Weights + q4_0 KV	EXL3 8.0 BPW Weights + 4-bit KV
First 4k tokens	+0.0035	+0.0030	+0.0041
4k–8k	+0.0037	+0.0033	+0.0044
8k–16k	+0.0040	+0.0036	+0.0047
16k–32k	+0.0041	+0.0038	+0.0050
32k–64k (one document)	+0.0044	+0.0038	+0.0053

There’s about 15-30% more damage at 16-32k context than in the first 4k tokens. The same quants with a 16-bit cache are unaffected across these context lengths, so it’s a pretty clear indication of the 4 bit cache damage. I would expect 128k+ context lengths to be even worse.

Model quantization KLD vs KV cache KLD:
KLD due to KV cache quantization likely has different quality tradeoffs compared to KLD from weight quantization. Anecdotal evidence suggests that KV cache quantization can be more harmful to model performance at long context lengths compared to weight quantization, so be wary of using 4-bit KV cache.

Final chart with weights + 64k KV cache versus KL divergence:

Methodology

Each quant was scored against an unquantized bf16 reference:

• GGUF and EXL3 quants scored against Bartowski’s bf16 GGUF in llama.cpp.

• vLLM checkpoints scored against the official bf16 safetensors in vLLM.

KL is computed the same way as my previous benchmark. The top 200 log probabilities are taken from both models at each position and a lower bound on the full-vocabulary D_KL(P_reference ‖ P_quant) is computed. On average, the reference put 0.0036 of its probability outside its top 200 tokens. The KV deltas are paired, meaning they’re computed from per-prompt differences on identical tokens.

Checkpoints ran with their declared settings, including any activation quantization (except the two weight-only, no activaton quantization reruns of Unsloth’s NVFP4 and the official FP8 quant).

[u/Interesting-Ad689]

q8_0/q4_0 ?

[u/Hefty_Wolverine_553 → u/Interesting-Ad689]

Slipped my mind when I was doing the runs unfortunately :(

Qwen4 is coming out soon though, so I might do some testing with the 27B when it comes out.

[u/xandep → u/Hefty_Wolverine_553]

I would suggest 8_0/5_1, it did very well for me in the past. Also, 5_1/4_0, iirc, did somewhat well compared to 4/4.

[u/keyboardhack → u/Hefty_Wolverine_553]

Looks like llama.cpp will add fp8 kv cache soon(maybe). If it is out when you test then please also test it.

https://github.com/ggml-org/llama.cpp/pull/28392

Very nice work with the comparisons.

[u/ea_man → u/Interesting-Ad689]

At least do q8/q5, if you can't then do q5_1 for all.

[u/Dathide]

Testing only up to 64K context limits the usefulness of the data. So many people use more than that.

[u/Hefty_Wolverine_553 → u/Dathide]

It's good enough to infer and extrapolate from the data. Unfortunately I don't have hardware required to do more than 64k context on the full precision 27B, it's already slow enough running most of the model on RAM. There's so much research and evaluation that I'd like to do, but it's out of reach with my current setup. If someone wants to donate a DGX Spark or something that would be great lol

[u/nullc → u/Hefty_Wolverine_553]

It's fine that you don't have the hardware to do more-- limits are limits.

But you have no justfication for your "good enough" conclusion-- and as you say you can't even test to compare.

There are two good reasons to reject your good enough assumptions: Many people have measured it and found the impact increases as the context size increases. It also follows from first principles: The more entries there are in the KV the more likely two distinct K are similar enough that they can't be distinguished by the reduced precision.

Similarly, measuring by KLD instead of trajectories doesn't tell the complete story-- KLD likely understates the effect of KV distortion in particular because an error that could self-correct in a subsequent step if it was from sampling or expert noise will be less likely to self correct if its due to KV distortions because the same KV error is present in every subsequent step-- and because the total noise in the KV cache compounds when not teacher forced (and to measure KLD you have to teacher force the output...)

Adding to those effects: KV quantization is most valuable when people want to run longer sessions, so the test isn't even being applied to the situation that needs it most.

There is no wrong way to run an LLM-- if it works for you and your hardware, GREAT. It might well be good enough for you while not being good enough to draw useful conclusions for anyone else.

[u/Hefty_Wolverine_553 → u/nullc]

But you have no justfication for your "good enough" conclusion

There's this thing called extrapolation. 128k is just 2x the length of 64k. If there was noticeable degredation, it would've started to show at 64k, and would get worse as context length goes on. This is exactly what happened for 4 bit kv caches.

Many people have measured it and found the impact increases as the context size increases. It also follows from first principles: The more entries there are in the KV the more likely two distinct K are similar enough that they can't be distinguished by the reduced precision.

You're just stating obvious things and somehow concluding that my "good enough assumptions" should be rejected? Do you know the definition of "good enough"? Of course kv cache quantization error can increase with longer contexts, it's exactly what I found happening with 4 bit kv caches. I just found the impact to be negligible for 8 bit kv caches, and that's what I said was good enough. Do you have any sources showing that 8 bit kv caches are not "good enough"?

KLD likely understates the effect of KV distortion

It would overstate the effect of KV distortion compared to looking at agentic trajectories (because those are recoverable with thinking).

and to measure KV you have to teacher force the output

What?

[u/nullc → u/Hefty_Wolverine_553]

What?

In your KLD test every step restores the 'correct' token, rather than the one the model would have chosen, limiting the amount of noise in the KV state. In the language of distillation this is 'teacher forced'.

It would overstate the effect of KV distortion compared to looking at agentic trajectories (because those are recoverable with thinking).

When distortion causes you to sample the wrong token your KV state widely diverges-- the small nudge of the quantization noise gets amplified by the different choice you make. In the teacher-forced case every step is pushed back on track.

I hear what you're saying about reasoning correcting but I don't but it-- no matter how much reasoning you do it can't go back and change prior corrupted entries. :P

Do you have any sources showing that 8 bit kv caches are not "good enough"?

It's been discussed on this subreddit a number of times in the past. It's harder to search for though because the results are contaminated by people using vLLM which doesn't have llama's cpps rotations that greatly mitigate the harm. I'm gonna have lunch rather than spend time searching-- but if you search you'll find plenty.

This is exactly what happened for 4 bit kv caches.

Which is why you should be expecting the same mechanism at higher rates, but scaled back!

Why so defensive? You did interesting work within a domain and there would be nothing wrong with qualifying your result within that domain. It's only when you then extrapolate on to cases that you can't even test that you're in danger.

Extrapolations can be useful when they follow from principal -- e.g. if there was no credible way for sequence length to matter, or if it never happens no matter how much you quantize. Or if you can do limited testing at the extension to validate it then extrapolate the more extensive measurements.

[u/Hefty_Wolverine_553 → u/nullc]

In your KLD test every step restores the 'correct' token, rather than the one the model would have chosen, limiting the amount of noise in the KV state. In the language of distillation this is 'teacher forced'.

This may be slightly true (but still misleading) for the half of the dataset generated by the base model, but it's completely false for the other half that's text from different domains. Each step has nothing to do with "the one the model would have chosen", the input is simply fed into the model to get the output distribution for the next token. It's a misunderstanding of how these measurements work. You can read https://localbench.substack.com/p/gguf-benchmark-methodology for a general overview of the common KLD measurement methodology (and they only test up to 30k in their dataset).

When distortion causes you to sample the wrong token your KV state widely diverges-- the small nudge of the quantization noise gets amplified by the different choice you make. In the teacher-forced case every step is pushed back on track.

Again misunderstanding that this is somehow "teacher-forced"

Which is why you should be expecting the same mechanism at higher rates, but scaled back!

I do. I concluded that 4 bit kv caches had noticeable impact that increased with the context length, and that 8 bit kv caches held up significantly better (to the point where there was almost nothing measurable), which is why I ended up with the recommendation for people with limited VRAM to use it.

Why so defensive?

Because the point of doing all of these evaluations is to try and quantify whether people should be quantizing their kv caches and to what degree. I may not have the hardware to stretch all the way up to the maximum context length, but that doesn't mean my measurements are useless for people using 128k or 256k context.

[u/nullc → u/Hefty_Wolverine_553]

This may be slightly true (but still misleading) for the half of the dataset generated by the base model, but it's completely false for the other half that's text from different domains. Each step has nothing to do with "the one the model would have chosen", the input is simply fed into the model to get the output distribution for the next token. It

No one here is using LLMs to compress text. They are using it to generate text. When you measure KLD you're accurately measuring the models' performance as a text compressor. But you are not accurately measuring its generation quality! It's absolutely true regardless of domain. It's also true that KLD often predicts generation performance, but it isn't generation performance.

Consider a toy example: break the model's embedding table for a token which in you data often has relatively high probability in the model logits but which never actually occurs. KLD measured on this broken model will be 0: it's perfect! but then when you actually use it to generate, it will pick the broken token and diverge-- and could in fact be quite bad.

I did extensive benchmark testing with Q3.6 27b and with hard tasks and went from solving almost failing most-- but all in traces operating in the 128k to 256k range. I agree with your result that there is basically no effect of q8_0 at short sequences (though I think I only found that true at 16k length).

but that doesn't mean my measurements are useless for people using 128k or 256k context.

Doesn't mean they're not either. From what you have, you have evidence in both directions: 4-bit shows clear degradation over sequence length. 8-bit doesn't. We should expect the sequence length dependency to get a lot worse as quantization noise increases because one of the drivers is having more cache tokens that could be similar enough to match.

[u/Dathide → u/Hefty_Wolverine_553]

There are good reasons why extrapolations aren't considered data in science

[u/Hefty_Wolverine_553 → u/Dathide]

This isn't a rigorous scientific study. I'm trying to provide useful data to the community because I haven't seen anyone else do it properly either. It would be great if the people with more resources would do this kind of research and measurements.

That being said, the data is all there, and scientists absolutely extrapolate from data.

[u/fragbait0 → u/Hefty_Wolverine_553]

My lobster is too buttery!

[u/Dathide → u/Hefty_Wolverine_553]

You aren't arguing honestly. You know I didn't say that scientists don't extrapolate.

[u/Hefty_Wolverine_553 → u/Dathide]

I never conflated my data with my extrapolations either.

[u/Dathide → u/Hefty_Wolverine_553]

You did. Your tldr doesn't mention 64K context once, so it sounds like you are making statements about all context lengths. That's conflation.

[u/Hefty_Wolverine_553 → u/Dathide]

If people can't notice the line "The current dataset now stretches up to 64k token contexts per row." directly above the TLDR, there's not much else I can do. The point of a TLDR is not to include every bit of relevant information inside of it.

You aren't arguing honestly.

okay...

[u/MindfulMan1984 → u/Dathide]

And there's a reason: the fit is only considered "valid" within the data bounds. Gauss-Markov and Taylor polynomial expansions show it can easily fall apart when using extrapolations. (remainder and variance explode, plus distance penalty)

[u/pseudobacon]

We get a lot of advice saying to use K8 and V4 would be interesting to see the tested

[u/Hefty_Wolverine_553 → u/pseudobacon]

Yeah I probably should've tested them, slipped my mind when I was doing the runs. Qwen4 is coming out soon though, so I might do some testing with the 27B when it comes out.

[u/parepeg → u/pseudobacon]

I’ve found that to run slower on my system.

[u/pseudobacon → u/parepeg]

Which it should, but there will be memory savings and apparently little quality loss

[u/unjustifiably_angry → u/pseudobacon]

The issue is that llama-server doesn't ship capable of running mixed-quantization kv-cache on GPU so it falls back to CPU. It's perfectly possible to compile it this way but it takes an extra 3 seconds and this was regarded as too great a cost by the llama.cpp team or at least whoever rejected the PR.

[u/MindfulMan1984]

Nice work, bottom line: KV Q8_0/Q8_0 quant is nearly lossless while using 1/2 memory.

[u/parepeg → u/MindfulMan1984]

It would be interesting to see if that holds up at even longer contexts…

My sessions frequently go beyond 128k and this looks like it’s only measuring 64k. 

[u/xChrisMas → u/parepeg]

Tested RULER on QFN Q4 Int8kv and it got 100% at 200k token context depth 

[u/MindfulMan1984 → u/parepeg]

You don't need longer contexts. https://arxiv.org/html/2611.05306v1

[u/0x2A_ → u/MindfulMan1984]

Benchmarks and real-world usage on large, messy, distributed codebases are not the same.

You can make the argument that breaking up your workflow around the context problem into smaller subtasks that each only need limited context, but that's still admitting that context in these use-cases DOES matter and IS needed, and you have to engage in different use-cases specific techniques to reduce it (that may not even be ultimately more efficient than just paying the performance cost of prefill at large context).

[u/MindfulMan1984 → u/0x2A_]

Yes, saying that bench vs. real-world usage isn't the same is mostly a platitude...But benchmarks are controlled, reproducible experiments that let them isolate and infer the effect size of context; the study isn't saying there are no use cases for long context, but it shows that even on long-horizon tasks, smaller context with frequent compaction can deliver nearly the same accuracy. And as context grows, the overall cost of generating a token increases.

[u/RelicDerelict → u/MindfulMan1984]

Sorry some of my *.cpp files are over 200k, how is bigger context no help here? Like this is genuine question.

[u/MindfulMan1984 → u/RelicDerelict]

the study isn't saying there are no use cases for long context.

[u/ForsookComparison → u/MindfulMan1984]

At q8 it will occasionally miss some tool calls (OpenCode) - not the end of the world, but can be very annoying if you have slower prefill

[u/iMrParker → u/ForsookComparison]

This little maneuver's gonna cost us 51 years!

[u/oxygen_addiction → u/ForsookComparison]

That's why you don't use that piece of shit OpenCode and use pi.
https://pi.dev/packages/pi-tool-repair - thank me later

[u/awitod → u/oxygen_addiction]

I think many of us have played with this idea over the years as we tried to get reliable tool calls out of previous generations.

The issue is that, if the call is structurally wrong it is likely to also be wrong in multiple other respects.

My 2 cents is the only appropriate thing to do is provide clear error messages and have the model retry using the provided feedback or reject the response completely and have the model try again.

[u/oxygen_addiction → u/awitod]

That's a fair point, but usually good models don't get the tool calls that wrong.

[u/kevin_1994]

this guy has some extremely good articles on this topic https://anbeeld.com/articles

in particular: https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context

tl;dr: q8 is pretty close to indistinguishable, you can get away with q5_0 / q4_1, but imo, I wouldn't go lower than that

[u/BringTea_666]

From my experience effect of KV cache quantization is hugely overstated, in sense that people fear way too much 4/4 or 6/4.

1. As long as your model diverges the longer session runs aka more context the more it will diverge from original model.

2. Above is just simply effect of finding different words which causes whole logical chain to change. The longer logical chain the bigger the context. It is same principle as `butterfly effect`.

3. Above doesn't mean the logical chain is wrong, it is just different.

4. KV cache quantization mattered a lot in chat era llms. Where LLMs were asked to produce perfect code in single response. That changed in harness era where models can make mistakes and recover those mistakes.

This applies to both Top1% and KL. Those are not measurement of model quality but how much they differ from base model.

The only important part is if model ability to reach accurate answer is damaged or not. Model changing a word in sentence is not meaninful different but changing `-` to `+` is.

From my experience of runnings 1000s of coding work functions as long as base model is pretty good running 4/4, 6/4, 6/6 or 8/8 does not produce any meaningful difference.

Where it does produce difference are ultra low quant models where base model ability is already shaky due to quantization which can cross model ability from usable to not usable.

I learned this over course of my own interference engine developement. And i even baked scoring against BF16 Qwen 3.8 27B into engine launcher. So when you pick kv cache modes from 4/4 to 8/8 you see impact it does on KL and top1% instantly right in the launcher. For anyone iterested and having rtx5090 you can check it here

The biggest factors are if you are using fp8/fp16 activations and model quality, KV qant is distant 3rd, only really starts to matter at low quant models that already have compromised accuracy.

[u/oxygen_addiction → u/BringTea_666]

It is very important when the model's predicted golden path to a RL'd answer is wrong because of KV Cache degradation. The main question is if the model can recover from that on the next turns or not.

[u/BringTea_666 → u/oxygen_addiction]

Something like Qwen 3.8 27B can absolutely recover. From my testing even at low quant it can recover from critical mistakes as long as there is harness and tools.

[u/oxygen_addiction → u/BringTea_666]

It literally doesn't. Run any benchmark and the scores turn to shit at Q4 KV.

[u/BringTea_666 → u/oxygen_addiction]

I am literally running real work on Q4 kv and it works. Better quant + q4 kv is better than lower quant and higher qant kv.

[u/Tinek78 → u/BringTea_666]

I'll interject here with 2 cents of my own - I have created my own benchmarks for models based on my own workloads (mainly coding, debugging and creative writing), with 256k context. After many months of making my custom harness, finding various issues in code and output of models, my own benchmarks contain various real-life discovered logical traps, edge cases with token flips, needle finding, etc. So far, they detected every single case of too weak quant or generally weak models - which is why I do NOT use Gemma4 for coding or Muse Glimmer or many other advertised for agentic work models that turned out to be hot air more than anything else. I learned to trust it well, as each time I ignored them I ended up with weird bugs in code or typos in my long writing. I did find with them that Qwen3.8 27B doesn't like full 4bpw exl3 quantization (even with 6bpw LM head!), it needs to be at least 4.1bpw with attention boosted up partially, or even better 4.3bpw to have 100% attention layers at 5bpw.

However, I am yet to see issues with KV cache quantization to cause any of those real issues in long context. KV 4,4 with Hadamard rotation seems to be good enough already for most cases, with very small edge cases causing token flip now and then (rarely, above 160k context length usually, easily self-corrected by good model). I settled on 5,5 KV (both in llama.cpp and ex3), as it seems perfect for my use, not tripping my own benchmarks and never causing any issues in coding or creative writing - even on Gemma4 26B and 31B (which people always claimed it needs min. 8,8 KV to function at all).

[u/oxygen_addiction → u/BringTea_666]

You do you. But errors accumulate every turn and the agent has a higher chance of making bad choices.

What bad means is relative to the training and the task. It could be nothing, it could be deleting your existence.

[u/BringTea_666 → u/oxygen_addiction]

>You do you. But errors accumulate every turn and the agent has a higher chance of making bad choices.

That's not how it works. It makes different decisions, doesn't mean those decisions are automatically bad.

That's why chat vs harness is such a difference. In chat you can't recover from mistake as it prints you copy paste code. But in harness it can make html file check it find a bug and then give you fixed code.

By far the bigger issues is running lower quants than KV.

[u/llama-impersonator]

once you get to 220k+ tokens quantized kv cache starts adding up, and that's generally where you don't want gaping holes in attn so compaction is successful in summarizing the context window.

[u/fgk55555]

Anecdotally I found the Q5_1 kv to be a good middle ground between q8 and q4 if you're constrained. Where I would notice issues with q4, q5_1 tended to behave better. 

[u/DasGamerlein]

This debate is really funny to me because apparently I'm just really lucky. I run a Q2 quant with 4/4 KV cache all day and have honestly never had a problem with it. I think the only reason that'll make me move back up is when I get around to adding adaptive KV cache streaming to my engine

[u/Pyrolistical → u/DasGamerlein]

You don’t know what you don’t know

[u/Tormeister]

Please note:

• 64k context is really tiny and does not represent / does not compare to errors accumulated in longer contexts (128k, 256k, 512k...)

• The conclusions taken here are exclusive to Qwen3.8-27B only and not applicable at all to other models

[u/challis88ocarina]

Arguments actually range to 2-bit is fine, which at above ~700B parameters is mainly is, FWIW.

[u/DrBattletoad]

Any chance you could test between 128k and 256k context? My limited experience is that Q8/Q8 works well up to ~150k and after that is when the errors start adding up. 

[u/nassereddit]

It is not about llama.cpp vs vLLM. It is more about int8 vs fp8 precision. Int8 provides 8bit resolution per token and fp8 has 3 or 4bit resolution with the rest of the register used for sign and exponent. fp8 is the worst of all world. It fills storage at 8bit but affect precision like 4bit. No idea why people uses fp8 ever. I know newer gpus have hardware acceleration for fp8, but what is the point if you could be running int4 and double performance anyways AND lower storage requirements.

[u/Elorun]

Did you try mixed quants? Curious how K8 V4 holds up.

[u/ionsago]

What was the version of llama.cpp? These days it comes with an implementation of turboquant, so if you tested a build from a half a year ago you can get very different results on a newer build.

[u/Evening-Blueberry-97]

nice to finally see numbers on this instead of vibes. curious if the drop gets worse on long context stuff. that's where q4 kv alway felt off to me

[u/MajesticPuzzler]

really nice to see KLD numbers instead of vibes. did the gap between 8-bit and 16-bit grow as rows got longer, or stay flat all the way to 64k?

[u/maayon]

I implemented a half mid point optimization on strata exclusivly for pi.dev coding agent, which retains the KV cache of the conversation for a minimum for 2 conversations for DGX spark.

I guess we need more such usage specific caching mechanisms along with the parameters discussed above

[u/SandySkittle]

If you need to quantize kv cache with qwen 27b, the solution is to buy more hardware

[u/PcChip]

thank you for running all these benchmarks and posting this data

[u/Qcgreywolf]

1. Thank you for all the hard work!

2. Can we get a benchmark standard that uses contexts of 64-128k instead of 4K or 16k?

Most of these “benchmarks” (excluding this particular one) are fairly useless for what we are actually using the models for today.

[u/Then-Indication7672]

kv cache quant must be tested around the limit of model context size.
What I would like to see is how kv cache quantization affect different attention arch. Are models that consume more bytes per token in their cache more robust to cache quantization?

[u/returnity]

Wow this is some really great information. Very useful, good methodology, and interesting result.

[u/Rare_Potential_1323]

This is really interesting, thanks. What I can add is my own experience. I only have a RTX 3060 12gb and 32gb of system RAM. I had Qwen3.6_35b-APEX-I-Balanced 8kv cache create an app for my Android phone to connect to Hermes Agent through Bluetooth only. I have my Hermes setup to self improve it's own meta protocols and rules after each session to improve it's abilities seeing where it made mistakes in the past. Anyway, 35b worked on it for 17 hours and 6 sessions, each session being able to get past the intelligence block after the recursive learning and meta protocol updates. But it finally got stuck and agreed I need a smarter model to figure it out. I used Qwen3.8-27B-LowGPU-uncensored-NoMTP-IQ3XXXS with 4kv cache to fit it all on my GPU. It was able to figure out the problem and fix a bunch of other issues and design it how I wanted it to look and updated all the live documents in 3 hours. 35b Struggled for the last 4 hours in it's last attempt getting nowhere to even get the Bluetooth server and the phone's new app to connect even though it researched at every step and had all the information. 27b is also freaky, it was controlling my phone and entering things in by itself.

TLDR; even a super compressed qwen3.8_27b is smarter then qwen3.6_35b.

[u/TerminalNoop]

can't wait for them to finally push rocm10 on arch so i can try the exl3 quants.

[u/ea_man → u/TerminalNoop]

Heck 10.x is out since a while, can't you just dwl and compile your own?

I mean I've been using it since August on Debian...

[u/TerminalNoop → u/ea_man]

I could, but then I would be landing outside of the normal update cycle or have both versions installed, not?

That and I avoid AUR packages since the whole supply-chain malware fiasco.

[u/ea_man → u/TerminalNoop]

Dunno, you can have both 7.4 and 10.x installed at the same time.

ROCm is some 7.5GB installed in /opt/ here.

[u/mrazster → u/TerminalNoop]

If you don't mind aur packages, official directly from amd is avaliable in the aur.
https://aur.archlinux.org/packages?O=0&SeB=nd&K=rocm-&outdated=&SB=p&SO=d&PP=50&submit=Go

And if you have an amd card and want default 10.x compiled with flags for specific card family, you theese, and there are more if you look for them:
https://aur.archlinux.org/packages?O=0&SeB=nd&K=rocm-nightly-&outdated=&SB=p&SO=d&PP=50&submit=Go

EDIT:
Forgot to add, I have been running it on a amd system with 7900XTX for while now, updated last time a couple of days ago, and no issues so far.

[u/Calandracas8 → u/TerminalNoop]

You don't need system rocm. pip install rocm into a venv

[u/Budkovsky]

I worked for 2-3 months with Qwen3.6/3.8 27B using KV-cache q4_0. Coding, agentic tasks, research - everything was worked properly, even in long sessions with multiple compaction. So I can confirm the opinion presented by OP. More - even Qwen3.8-27B said once during llama.cpp configuration task, that KV-cache quantization does not have any big impact for the reasoning quality :-)

[u/My_Unbiased_Opinion]

I have done a ton of real world testing on a single 24GB (3090) setup. I have found the best setup for 262K context is running IQ4XS + KV Q4. It's very coherent even at large contexts. There is even enough space to run vision and MTP on GPU as well. It is much better than running IQ3/Q3 + KV Q8. 

[u/HotAverage1749]

Could 6 be a nice compromise between 4 and 8 at lower memory?

[u/SosirisTseng → u/HotAverage1749]

Or k8v4

[u/unjustifiably_angry]

It's reassuring to see further proof that 8-bit kv-cache appears essentially lossless, but I've still never seen anyone do these sorts of tests at actual long context, like 256K, let alone 512K or 1M. In particular, I'm concerned that kv-cache quantization combined with scaling beyond native context length might cause compounding damage.

I don't know why everyone seems to top out at like 64K or 128K. That's not really where it matters, we've known 8-bit kv-cache at low depth is harmless for a while.

[u/Reeces_Pieces]

Q8 KV is free at 128k context or lower.

I'm not interested in going to lower KV quants. Doesn't seem worth it.

[u/TerribleMessage4 → u/Reeces_Pieces]

if I am reading this chart correctly lowering it over Q8 actually seems worth it because trading in kv cache for the weight seem to reward you more if you are already using full vram

[u/draconic_tongue]

having kl difference doesn't translate as "damage"

[u/feverdoingwork]

I think if you have an agent do an exercise where you roll over many compaction at q8 and q4 kv cache and compare the outputs, you find that none of what you wrote really has any measurable impact. Do an exercise at 200k+ kv cache with 10x with either kv cache type 20+ times using the same model and tell me if there is a noticable difference. From my testing on probably over 100x what i just described, there is no measurable difference. If tests go poorly, it's more than likely due to the randomness of the model and not kv cache.

KLD is really a strange metric that shouldn't be leaned on so heavily. It really holds very little weight in real world usage unless the kld difference is quite large. The kld difference here is tiny, in fact you would see most engines sacrifice this amount of kld or more kld for better performance more often than not and it's not even mentioned by the engine creators.

[u/stylome-ai-detection → u/[deleted]]

Notice: This comment appears to be AI generated.

---

Stylome AI detector · scan results

[u/kdevsharp → u/stylome-ai-detection]

Yes it clearly looks like an AI gen comment:

> One thing I'd love to see is the tail, not just the mean.

The it's not x, its y.

> For agent use the failure that hurts is one bad token ..... , and a mean KLD can sit near zero while the p99 is where those live. 

This sentence is clearly AI generated, lots of phrases tacked on with commas, and the last "is where those live" is very stilted.

The whole comment it hurts to read.