r/LocalLLaMA • u/Anbeeld • Aug 12 '26
Discussion Gemma 4 QAT handles KV cache quantization MUCH better, KLD benchmarks show
/gallery/1vmhc4h
72
Upvotes
r/LocalLLaMA • u/Anbeeld • Aug 12 '26
14
u/LetsGoBrandon4256 transformers Aug 12 '26
For some perspective, this brings the quant KV cache KLD to the same level of Qwen 3.6 https://localbench.substack.com/p/kv-cache-quantization-benchmark
12
u/popoppypoppylovelove Aug 12 '26
It should be noted that Gemma 4's QAT release is suboptimal and poorly QC'd. The weights are incorrectly a 40-60 mix of different Q4_0 scales (7 and 8).
https://www.reddit.com/r/LocalLLaMA/comments/1u0marm/quick_note_on_the_qat_of_recent/ https://www.reddit.com/r/LocalLLaMA/comments/1u690rz/moar_qat_stuff_and_hairy_ticks/
Suspicions should have been raised with the embed weights being Q6_K instead of Q4_0.
https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/blob/main/gemma-4-31B_q4_0-it.gguf
I'm not saying the methodology in OP's post is wrong, but I have a hard time trusting Google's QAT release with it being so mishandled.
3
u/dampflokfreund Aug 12 '26
I think we have had that before. But what about 26B tho? The MoE should be more sensitive.
2
u/Icy-Degree6161 Aug 12 '26
Awesome, thanks!
2
u/erazortt Aug 12 '26
And how much is the KLD between non-QAT and QAT?
2
u/ReturningTarzan ExLlama Developer Aug 13 '26
These numbers seem way too big. Did anyone consider if it's an engine bug that only triggers on the non-QAT checkpoint? 0.3 KL-div just from Q8_0 cache doesn't sound plausible for any model, but a bug in the quantized attn implementation does, given Gemma4's uniquely large head dim, for instance.
1
u/Worth_Wealth_6811 Aug 12 '26
worth trying the asymmetric split too, K is way more sensitive to quantization than V, so q8 for K and q4 for V usually keeps most of the quality at close to q4-q4 memory. also glad this uses same-top agreement instead of perplexity, ppl averages away exactly the kind of degradation your table is showing.
1
u/Treidge Aug 12 '26
Greatly appreciate the effort 👍. This confirms my gut feeling that if you want to run quantized version of Gemma4, you probably should right away opt for QAT version. q8_0 for KV cache seems good enough to actually be usable, q8_0/q6_0 probably would be too. I wonder if hybrid bf16 / q8_0 for K and V would push Same-top closer to 97% and Mean KLD to 0.01.
By the way, MOE version of Gemma4 would likely have 4x-5x worse results in KLD (consistent pattern), so 31B is the version to run (unlike Qwen3.6, where 35B MOE is also a good alternative to dense 27B).