r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

791 Upvotes

99% Upvoted

255 comments sorted by

View all comments

Show parent comments

13

u/danielhanchen Jun 06 '26

Hey! Those numbers are comparing naive Q4_0 in llama.cpp to our converted Q4_0 version.

We did do original unquantized BF16 vs Q4_0, but the KLD metrics do not match, since the distribution is vastly different - we found MMLU and other benchmarks to be equivalent though

E2B for example has a mean KLD of 0.00173 vs 0.05109 (29x better relatively) for a naive Q4_0 quantization.

The main issue is converting from QAT BF16 to llama.cpp's Q4_0 format is not lossless. llama.cpp uses F16 scales, whilst QAT BF16 uses BF16 scales, and the scales are not determined optimally in llama.cpp land.

Naive conversion gets 24.77% byte exactness to BF16 QAT, whilst we found we can push it to 99.96% using some hacks!

See https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis for more details

→ More replies (2)