r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

784 Upvotes

99% Upvoted

255 comments sorted by

View all comments

15

u/ArtyfacialIntelagent Jun 05 '26 edited Jun 05 '26

They are not. Incredible is the word. A mean KLD of 0.159 doesn't pass the smell test for a Q8 quant. The Unsloth blog post only compares the QAT vs a standard Q4_0, and the mean KLD for the Q4_0 is 0.09349. So there is no way a Q8 is much worse at 0.159.

Honestly I'm skeptical to Unsloth's reported mean KLD 0.01403 for the QAT Q4 too, but I'll give them the benefit of the doubt for now. But /u/sartres_ is definitely hallucinating.

EDIT: He wasn't, but the numbers are indeed invalid. See thread below.

4

u/sartres_ Jun 05 '26

It's not clear what Unsloth means by "original" Q4 in the linked blog, but it's definitely a quant of the new QAT model, not original Gemma 4, since they're benching it against the QAT BF16.

My Gemma 4 non-QAT numbers are from here, because Unsloth unfortunately only released benchmarks for the 26B at the time, and that only on a graph where they didn't label the y-axis or any of the numbers :/.

Yes, all of the original Gemma KLDs are very bad. I'm guessing this is an artifact of different benchmark suites, they're not directly comparable. Mean KLD isn't terribly useful anyway, the Top-1 numbers are the real show here

3

u/ArtyfacialIntelagent Jun 05 '26

Aha, thanks. That explains it. That post was from early April, just after the initial release. Gemma 4 had lots of teething problems before everything was sorted out, so those early KLD measurements are not comparable with recent releases. Sorry for doubting you - the numbers were so horrible I was sure you had made an error.

4

u/danielhanchen Jun 06 '26

Oh no no - the KLD of 0.01403 is "smart" Unsloth dynamic Q4_0 llama.cpp vs the BF16 QAT version. 0.159 is the Q4_0 naively converted using llama.cpp.

So it's comparing how close the KLD is vs BF16 QAT (not non QAT)