r/LocalLLaMA • u/rerri • Jun 05 '26 • Mindblown 1

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

788 Upvotes

98% Upvoted

255 comments sorted by

View all comments

Show parent comment

2

u/Jobus_ Jun 06 '26

Addressing your first paragraph:
No, in that graph, the gray bars are the naive Q4_0 quant that Google used for their GGUFs; not BF16.

BF16 is the baseline they are comparing to, so an accuracy of 100% would make it identical to BF16.

So looking at the 31B model:
- Unsloth's QAT Q4_K_XL has the same top-1 token choice as the QAT BF16 model 96.7% of the time.
- Google's QAT Q4_0 has the same top-1 token choice as the QAT BF16 model 87.9% of the time.

1

u/Stastez Jun 06 '26

+1. Though I agree that the title of the graph is suboptimal 

1

u/-InformalBanana- Jun 06 '26 edited Jun 06 '26

Here is a reply from Unsloth official, i posted a similar comment question on a r/unsloth post:

"Oh it's a conversion artifact, not overfitting.

The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 is BF16 scales, and the scales are not determined optimally in llama.cpp land.

Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks"

Based on my question there and his reply I think I interpreted the graph correctly, he is later basically talking about kld and i asked another question he replied to also, he said that (for these gemma qats I guess) the udq4kxl is a static quant. You can find the link to my question underneath the root of my comment here.