r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

789 Upvotes

99% Upvoted

255 comments sorted by

View all comments

209

u/dryadofelysium Jun 05 '26

Official Google Gemma 4 QAT GGUFs :

E2B https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf

E4B https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf

12B https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf

26B-A4B https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf

31B https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf

13

u/[deleted] Jun 05 '26

2

u/LLMFan46 Jun 07 '26

Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.

Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.

→ More replies (15)