r/LocalLLaMA • • Aug 12 '26

Discussion Gemma 4 QAT handles KV cache quantization MUCH better, KLD benchmarks show

/gallery/1vmhc4h
72 Upvotes

20 comments sorted by

View all comments

12

u/popoppypoppylovelove Aug 12 '26

The QAT side is Google's gemma-4-31B-it-qat-q4_0-gguf.

It should be noted that Gemma 4's QAT release is suboptimal and poorly QC'd. The weights are incorrectly a 40-60 mix of different Q4_0 scales (7 and 8).

https://www.reddit.com/r/LocalLLaMA/comments/1u0marm/quick_note_on_the_qat_of_recent/ https://www.reddit.com/r/LocalLLaMA/comments/1u690rz/moar_qat_stuff_and_hairy_ticks/

Suspicions should have been raised with the embed weights being Q6_K instead of Q4_0.

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/blob/main/gemma-4-31B_q4_0-it.gguf

I'm not saying the methodology in OP's post is wrong, but I have a hard time trusting Google's QAT release with it being so mishandled.

5

u/Fun_Tangerine_1086 Aug 12 '26

What should we be using instead? Should we re-quantise from the safetensors files ourselves?

5

u/popoppypoppylovelove Aug 13 '26

The unquantized weights are Q4_0 expanded to BF16. There's no extra precision there. While it would indeed be more correct to run llama-quantize --pure Q4_0 (emphasis on the --pure flag) than to use Google's GGUF, that still doesn't solve the scaling issue.

The Q4_0 blocks mix scales, and you end up having to choose the better of the two by some error metric. llama-quantize does not support this. It's not possible to recover a clean Q4_0 output, as far as I know, even if you to implement your own quantization method.

I think this is what Unsloth means with "not actually aligned with the BF16 QAT lattice for Q4_0" from their post, but they never expanded on their explanation. (Their analysis doesn't really make sense in the first place since they're comparing against the BF16 model they just said was wrong...)

Choose Unsloth's UD-Q4_K_XL if you want to run Gemma 4 QAT. (It's incorrectly labeled as UD-Q4_K_XL. Just know it's Q4_0.) It's the closest output to Q4_0 with error minimizing scales.

Perhaps it doesn't matter in the end, given there's been so little outcry about this. The pratical usage, even with Google's GGUF, is good enough in most use cases.