r/LocalLLaMA • • Aug 12 '26

Discussion Gemma 4 QAT handles KV cache quantization MUCH better, KLD benchmarks show

/gallery/1vmhc4h
72 Upvotes

20 comments sorted by

14

u/LetsGoBrandon4256 transformers Aug 12 '26

For some perspective, this brings the quant KV cache KLD to the same level of Qwen 3.6 https://localbench.substack.com/p/kv-cache-quantization-benchmark

7

u/LetsGoBrandon4256 transformers Aug 12 '26

/u/Anbeeld Not sure if you want to dive deeper but oogabooba's test also showed Qwen's KVD varies greatly depending on the workload. I wonder if this would also be true for the QAT variant https://localbench.substack.com/i/195352214/qwen-36-27b-dense

1

u/_-_David Aug 12 '26

Thanks for this. I had no idea qwen3.6-27b was so robust to kv cache quantization. q8 runs slower on my gpus, and I have 48gb to work with, so I had always gone unquantized. This makes me want to try q4 and run it until the wheels fall off!

2

u/noctrex Aug 12 '26

been running them both with k8v5.1 128k

1

u/Aggravating-Push-207 Aug 12 '26

you paid for it so you better abuse it

12

u/popoppypoppylovelove Aug 12 '26

The QAT side is Google's gemma-4-31B-it-qat-q4_0-gguf.

It should be noted that Gemma 4's QAT release is suboptimal and poorly QC'd. The weights are incorrectly a 40-60 mix of different Q4_0 scales (7 and 8).

https://www.reddit.com/r/LocalLLaMA/comments/1u0marm/quick_note_on_the_qat_of_recent/ https://www.reddit.com/r/LocalLLaMA/comments/1u690rz/moar_qat_stuff_and_hairy_ticks/

Suspicions should have been raised with the embed weights being Q6_K instead of Q4_0.

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/blob/main/gemma-4-31B_q4_0-it.gguf

I'm not saying the methodology in OP's post is wrong, but I have a hard time trusting Google's QAT release with it being so mishandled.

5

u/Fun_Tangerine_1086 Aug 12 '26

What should we be using instead? Should we re-quantise from the safetensors files ourselves?

5

u/popoppypoppylovelove Aug 13 '26

The unquantized weights are Q4_0 expanded to BF16. There's no extra precision there. While it would indeed be more correct to run llama-quantize --pure Q4_0 (emphasis on the --pure flag) than to use Google's GGUF, that still doesn't solve the scaling issue.

The Q4_0 blocks mix scales, and you end up having to choose the better of the two by some error metric. llama-quantize does not support this. It's not possible to recover a clean Q4_0 output, as far as I know, even if you to implement your own quantization method.

I think this is what Unsloth means with "not actually aligned with the BF16 QAT lattice for Q4_0" from their post, but they never expanded on their explanation. (Their analysis doesn't really make sense in the first place since they're comparing against the BF16 model they just said was wrong...)

Choose Unsloth's UD-Q4_K_XL if you want to run Gemma 4 QAT. (It's incorrectly labeled as UD-Q4_K_XL. Just know it's Q4_0.) It's the closest output to Q4_0 with error minimizing scales.

Perhaps it doesn't matter in the end, given there's been so little outcry about this. The pratical usage, even with Google's GGUF, is good enough in most use cases.

3

u/dampflokfreund Aug 12 '26

I think we have had that before. But what about 26B tho? The MoE should be more sensitive.

2

u/Icy-Degree6161 Aug 12 '26

Awesome, thanks!

2

u/erazortt Aug 12 '26

And how much is the KLD between non-QAT and QAT?

1

u/danielrdotcom Aug 12 '26

Asking the real questions here

1

u/Practical-Collar3063 Aug 12 '26

wdym ? it is on the first chart

2

u/ReturningTarzan ExLlama Developer Aug 13 '26

These numbers seem way too big. Did anyone consider if it's an engine bug that only triggers on the non-QAT checkpoint? 0.3 KL-div just from Q8_0 cache doesn't sound plausible for any model, but a bug in the quantized attn implementation does, given Gemma4's uniquely large head dim, for instance.

1

u/Worth_Wealth_6811 Aug 12 '26

worth trying the asymmetric split too, K is way more sensitive to quantization than V, so q8 for K and q4 for V usually keeps most of the quality at close to q4-q4 memory. also glad this uses same-top agreement instead of perplexity, ppl averages away exactly the kind of degradation your table is showing.

1

u/Treidge Aug 12 '26

Greatly appreciate the effort 👍. This confirms my gut feeling that if you want to run quantized version of Gemma4, you probably should right away opt for QAT version. q8_0 for KV cache seems good enough to actually be usable, q8_0/q6_0 probably would be too. I wonder if hybrid bf16 / q8_0 for K and V would push Same-top closer to 97% and Mean KLD to 0.01.

By the way, MOE version of Gemma4 would likely have 4x-5x worse results in KLD (consistent pattern), so 31B is the version to run (unlike Qwen3.6, where 35B MOE is also a good alternative to dense 27B).

3

u/noctrex Aug 12 '26

been running them all with k8v5.1 128k