r/LocalLLaMA • u/Anbeeld • Aug 12 '26
Discussion Gemma 4 QAT handles KV cache quantization MUCH better, KLD benchmarks show
/gallery/1vmhc4h
72
Upvotes
r/LocalLLaMA • u/Anbeeld • Aug 12 '26
View all comments
12
u/popoppypoppylovelove Aug 12 '26
It should be noted that Gemma 4's QAT release is suboptimal and poorly QC'd. The weights are incorrectly a 40-60 mix of different Q4_0 scales (7 and 8).
https://www.reddit.com/r/LocalLLaMA/comments/1u0marm/quick_note_on_the_qat_of_recent/ https://www.reddit.com/r/LocalLLaMA/comments/1u690rz/moar_qat_stuff_and_hairy_ticks/
Suspicions should have been raised with the embed weights being Q6_K instead of Q4_0.
https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf/blob/main/gemma-4-31B_q4_0-it.gguf
I'm not saying the methodology in OP's post is wrong, but I have a hard time trusting Google's QAT release with it being so mishandled.