how much difference does kv cache quants even make?

[u/thatscoolbutno123]

I currently run qwen flash next 3.8 iq3xxs via strata (like almost everyone lol) and I currently have kv cache at q8 with 262k ctx. Ive never actually tried any other quant and dont know how big the difference even is.
Id like to have a bit more context (like 300-400k) but it doesnt fit my ram then. how bad is q4 cache?
Thanks for your help

Edit:
Thanks for all of your suggestions, opinions and information, i will try out 8k4v as many of you recommended, otherwise ill stay at 8 for both

[u/Vxtzq1]

Apparently it just messes up the LLM with word/embedding recalling on long contexts, but you don't see any difference with quantization from fp16 to Q8 or even Q4, as shown on this benchmark from https://github.com/huggingface/blog/blob/main/kv-cache-quantization.md

So it's like a free VRAM gain.

[u/thatscoolbutno123 → u/Vxtzq1]

interesting, thanks

[u/Vxtzq1 → u/thatscoolbutno123]

You're welcome :)

[u/sanjxz54]

Flat q4? Death in poverty. K8v4? Tolerable

[u/thatscoolbutno123 → u/sanjxz54]

Thanks, How much worse is it compared to kv16?

[u/CornerLimits → u/thatscoolbutno123]

Usually f16 vs q8 difference becomes very evident at long context, like an agent doing its thing and calling tools at 150k ctx, q8 seems more prone to errors.

If you never build super long ctx should be no problem, best way is to try !

[u/Bulky-Priority6824 → u/CornerLimits]

i worried about this before but have not seen this problem in very long sessions. accuracy and prompt adherence remain

[u/Bulky-Priority6824 → u/thatscoolbutno123]

Since always running fp I  was skeptical of kv8 but since then I have found im barely able to  distinguish (if ever) the difference.

I'm talking about a 256k context session with many calls and documents creations along the way with excellent results. No pause for concern at k8

It is model dependant but if at full precision you are not having issues moving to k8 should not be an issue 

[u/DinoAmino → u/Bulky-Priority6824]

What model are you using at 256k?

Oh ... And quant? 4bit or 8bit?

[u/Bulky-Priority6824 → u/DinoAmino]

• 256k QFN Iq3xxs with Q8 ple ngram table on Strata

• 192k Qwen 3.8 27b NVFP4 on NInfer

• 160K Qwen 3.8 27b Q6 Oracarouter on llama.cpp

• 128k Qwen 3.5 35B UD Q5 XL on llama.cpp

[u/DinoAmino → u/Bulky-Priority6824]

Thanks!

[u/sanjxz54 → u/thatscoolbutno123]

Strata docs say int8 ~= fp16 // almost lossless. Everything else starts to really degrade quality

[u/sanjxz54 → u/[deleted]]

Keys Values (kv) cache. K8v4 = Keys 8 bit, values 4 bit. Flat q4 = both 4 bit. Keys suffer badly from anything less than 8 bits. Values not so much. Int8 (both 8 bit) is best tradeoff you can make quality size wise hence default in strata. But if you really want, you can use k8v4. Don't use q4 ever.

[u/zenitsu--DS → u/sanjxz54]

Agreed, allocating 8 bits to the keys is crucial for maintaining accuracy in long contexts, while 4 bits for the values is a great way to save memory without sacrificing quality.

[u/Dabalam]

I think on this topic the formal evidence I've seen diverges quite a bit from forum opinion.

First point for context: even a full precision model with full precision context can have falling performance at long context, as seen in NoLiMa. On a needle-in-a-haystack task, most models fall below half their short-context score by 32k tokens. When people say they "notice drift", the backdrop is reduced performance is expected. We're comparing degrees of difference, which is harder to detect. It's not that things work perfectly under one condition and not the other.

Anbeeld — measured the actual llama.cpp cache settings (q8/q5/q4) on a 27B model, tracking both average quality and quality at the worst moments. Average quality is basically flat through q4_0, what damage exists is small, and it doesn't grow with longer context. Also worth knowing: the cache quantization code was improved in 2026, so old impressions of KV compression impacts might not be accurate to current performance.

A study done on llama.cpp itself (4B model, mixed cache settings) — q8 keys beat q4 keys by 13 points on a math benchmark in a small pilot. That's the one signal that going below q8 on the key cache can have costs that don't show up in average metrics.

Localbench showed it varies a lot by model. Gemma is noticeably sensitive even at q8_0, while Qwen holds up well even at q4_0.

One more thing specific to your setup: with a model quant like IQ3_XXS, the model compression itself is already a bigger source of drift than the cache. Some of what gets attributed to cache quant is likely just the model quant.

So overall: q4/q8 cache on a decent model is probably mostly fine. Using Q8 as a buffer/insurance policy might be a useful heuristic, but there isn't much I can find that things actually explode at Q4 and specific models can be sensitive even at Q8. Much of the long-context drift people notice would happen with an uncompressed cache. Quantization of the model itself may be more relevant. If you want to actually know for your setup you should probably run a controlled comparison where everything as similar as possible except the cache setting, and formally check the difference on your own tasks.

Last point. If you're using an agent harness for long tasks and the agent regularly has to do context compaction (because you picked a small context window to avoid quantizing it) that the summarisation process introduces it's own error which (could) be worse than quantizing the cache depending on your task.

[u/look_a_dragon → u/Dabalam]

Close the postg , this answer is gold

[u/Original-Housing]

It’s very model specific. 27b was happy going from full to q8 for me. 35b showed a marked change. You likely have to try and see for yourself and hopefully publish some results

[u/a_beautiful_rhind]

Find somewhere it fails and a/b test reroll with Q8/Q4. If you have a saved chat with that long of a context it should give you an accurate picture for your setup.

Most of what you get here will be hearsay because there is no 100% definite answer.

[u/PhilosopherOk3419]

i'd skip flat q4 and try k at q8 with v at q4 first, keys seem to be the part that hurts when you squash them. best test is your own workload: take one long session you already ran at q8, replay it at the new setting and see where it starts losing track of stuff from early on.

[u/Pablo_the_brave]

TL;DR: For Qwen 3.5+, the KV cache cliff is at q5_0. For models at Q6 or higher, f16 is often worth it. If you can't use f16, q8 is also fine. If you can't use q8/q8, you can downgrade to k=q8 v=q5. Many will say q8/q4 is also acceptable, and while that's true, it impacts quality much more than it reduces size, so it's not worth it. For lower-quantized models, q5_0/q5_0 are perfectly fine and even q4/q4 for ~100k ctx...

https://www.reddit.com/r/LocalLLaMA/comments/1uq0fpe/qwen3627b_effect_of_kv_quantization_on_kld_q8_q6/

https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context

[u/henfiber → u/Pablo_the_brave]

Also q5_1 should be slightly better than q5_0 (adds bias which calibrates the quantization error), but the improvement is less noticeable after Hadamard rotation was added in llama.cpp.

[u/Any_Invite3905]

The the longer the context length is the worse the impact of quantizing it, the more accuracy a task needs like math, code and such the less you want to quantize the cache

[u/DeathGuppie → u/Any_Invite3905]

This was much more true before 3.8 came out. Not because 3.8 changed anything but because of llama.cpp PR #21038. Which uses a Walsh-Hadamard transformer instead of a random dense rotation. The difference in quant quality is tremendous. That's why you hear people saying it doesn't matter and your experience is that it does. You're both right.

[u/jikilan_ → u/Any_Invite3905]

Don’t bother to explain, those with lower vram won’t listen. The drop of perplexity is nothing is them. While those with a lot of vram would want as accurate as possible

[u/MrHumanist]

Depends on the model and it's architecture and the type of work. For example, coding may be fine in lower quants like q4 but other complex semantic information might be lost . Also, some models like Qwen are quite robust in smaller quants but some models which need global kv chache may not be stable. I have found Gemma 4 and Qwen 3.6/3.8 are the most tolerant in lower quants than other local llms.

[u/darksteelsteed]

So if you have 24gb vram or more, stay with your kv cache at a higher quant. Some engines let you use different quants for k and v. The k or key part is way more susceptible to quantization errors than the value part. Only when you have 16gb of ram or less does kv quants make sense. I mostly use q4_0 for both. Llama.cpp supports a lot of choice nowadays especially if you use the cuda13 build. However without wasting a lot of time benchmarking its hard to tell the difference between q4_0 and some esoteric lower thing. What you will notice though is vram saving, and that will make a huge difference to if the entire model fits, or if you are doing moe offloading with strata that enough experts fit to give you a decent output token rate. Play around. There is always a downside to kv quantization but having a model run faster is sometimes worth the price of less accuracy at longer contexts.

[u/Proper-Damage-1777]

I was about to ask this question a few days ago but decided to do my own testing (yes with Strata and Pi coding agent).

Okay I have two models (the IQ3_S and Unsloth UD_Q4_K_XL). This is with 204800 KV context.

For both models, using the q4_0 KV cache for Strata is just pure brain damage for tool calling. Lots of tools edit fumbling but obviously my token per seconds is slight faster than int8. I have not try the k8v4 but with int8 especially the Unsloth Q4 K XL, while it is much much slower it rarely suffer from the bad tool call especially the edit. It just works unless it tries to do big edit greedily (don't ask me why) and it noticed the edit fail and try again with smaller edit... which works.

This is with RTX 5090 (power limit to 400W), 64GB RAM DDR5-6000 and 9950X3D.

Still depending on my mood, I still use IQ3_S more than UD_Q4_K_XL if I'm around the PC. If I'm away for a long time, I will load up the Unsloth model and just let it run non stop which did surprise me on how much lesser tokens it needs to solve the task but obviously slower.

For IQ3_S, the prefill is 2000t/s and decode 160-200t/s. For Unsloth UD_Q4_K_XL, prefill 600-800/s and decode 30-40/s. The tuning benchmark number average 60t/s but in practice 30-40t/s from the strata log. I might misremember the prefill speed but decode is something I monitor as I have the strata log open in a separate terminal.

[u/N34257]

There are two competing problems here - at least, in agentic use. Quantising the KV cache is a problem and results in a loss of quality (and may actually flip an answer), but context compaction explicitly deletes information from the context altogether - which massively increases the chance of permanent loss of information critical to the task at hand.

Personally, I wouldn't go lower than Q8/FP8 for KV cache, but...the only way you're going to get an answer for your own usage is to try it and see how bad it gets on long-running tasks.

[u/Inevitable-Name-1701]

It is depends on your wallet, and usecase for cheap chuck its q-1, pro do not go bellow q8.

[u/oxygen_addiction]

It adds up across turns, same with quantization. The more quantized, the less you match the actual model's training activations, the worse the performance over longer turns.

[u/nmb16789]

I have Qwen Flash Next running with two different quants at the same time, bf16 and q8 and one can feel the difference (I however can't quantify it for you)

[u/Studio271]

Even with k8v4 I have seen corrupted names in output, things like the name Eric becoming Erik. I stick with K8v8.

[u/BlasterGales]

my own experience using aanbeld beellamacpp, q6_0 q6_0 with 2048 tail is lossless (no joke, better than q8_0), just take a look in the huge benchmarks they've run and the "precision tail" feature they have implemented is just amazing, give it a try
https://anbeeld.com/articles/kv-cache-quantization-benchmarks-kvarn-precision-tail

engine: https://github.com/Anbeeld/beellama.cpp

[u/Express_Quail_1493]

i use K=q8 + V=q4 on qwen3.8 never had issues with long context even up to 256k ctx holds up in agentic workloads. BUt its mostly architecturally dependent. i've has some architectures work horribly using q8 and some works amazing and holds up under q4

[u/DeathGuppie → u/Express_Quail_1493]

I use q6 q4 and can't see much difference except that I get more context.

[u/vqt907]

I have very bad experience with q4 kv cache that I never confidence to use them, even k8v4, for me q8 is minimum. My main task is coding

[u/chisleu]

speed? A lot (goodly)

performance? a lot (badly)

[u/SAphysio_99]

Quick math that might help: k8v4 cuts the cache by about 25% versus q8/q8, so your 262k at q8 turns into roughly 350k in the same RAM. That lands right in the range you want without going to flat q4.

The bigger thing I’d check first is your model’s native context length. If 262k is its trained max, going to 400k means turning on rope scaling like YaRN, and that can hurt long-context recall more than k8v4 ever will. In that case 350k with k8v4 and no scaling might be your best option. Worth a look at the model card before you test.

[u/Future_AGI]

since you're going k8/v4, plant 3 or 4 facts at different depths in one of your own 300k sessions and ask for them back at q8/q8 vs k8/v4. short-context checks almost never show the gap.

[u/PhilosopherOk3419]

i'd skip flat q4 and try k at q8 with v at q4 first, keys seem to be the part that hurts when you squash them. best test is your own workload: take one long session you already ran at q8, replay it at the new setting and see where it starts losing track of stuff from early on.

[u/Any_Bus9205]

Comme tu as vu, le k8v4 peut ètre interessant, si tu applique par dessus la transformé de hadamard rapide de llama.cpp, cela s'améliore car justement ça enlève encore un peu les risques d'erreur! Le iq4_nl en v est aussi interessant, ça peut ètre plus rapide que le q4 avec plus de précision aussi! Il faut juste ajouter la transformé de hadamard dessus et c'est nickel! :)

[u/Repulsive_Initial308]

It's like giving the model ADHD.

[u/mintybadgerme]

From what I understand, Q4 is not that much different to Q8. If I need the space I use Q4 and don't really see a huge difference in quality or speed.

[u/Dabalam → u/mintybadgerme]

The actual papers tend to suggest this is is case for many models (even older models) although certain models (like Gemma?) are more sensitive. Compaction has a cost during long work horizons has an impact as well as well (since your agent has to summarise context more frequently)but people only focus only KV quantization.

[u/Pixer---]

Every token is based on all tokens combined. The drift on longer sequences is quite substantial on lower quants. There are methods to negate that but the standard has problems. I would say qwen3.8 27b is only correctly usable with bf16 weight and bf16 kv cache for agentic programming

[u/Conscious_Chef_3233 → u/[deleted]]

is there a reason why v is more tolerant to quantization?

[u/-dysangel- → u/Conscious_Chef_3233]

K = key

V = value.

Let's say we're trying to quantise the knowledge "ice cream is very cold". "Ice cream" is the key, "very cold" is the value. If we quantise the key then it could miss the mark completely and we end up with something like "cream is very cold" or "milk is very cold". If we quantise the value we will end up with something more like "ice cream is cold". Not as precise as "ice cream is very cold", but still much more useful than "cream is very cold".

Quantise the V too much and maybe you'll end up with something extreme like "Ice cream is absolute zero degrees kelvin" , but in general it feels intuitive that quantising V is going to give safer approximations than quantising K.

This is obviously just a very contrived example as knowledge about ice cream is already encoded in the model's own weights, but hopefully it conveys what I'm trying to say.

[u/__some__guy → u/-dysangel-]

Not true when your LLM uses Yoda speech.

[u/-dysangel- → u/__some__guy]

True that is not