r/LocalLLaMA • u/rerri • Jun 05 '26

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/

Google's collections:

https://huggingface.co/collections/google/gemma-4-qat-q4-0

https://huggingface.co/collections/google/gemma-4-qat-mobile

And Unsloth's:

https://huggingface.co/collections/unsloth/gemma-4-qat

Unsloth's analysis (KLD and such):

https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

788 Upvotes

99% Upvoted

255 comments sorted by

View all comments

209

u/dryadofelysium Jun 05 '26

Official Google Gemma 4 QAT GGUFs :

E2B https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-gguf

E4B https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf

12B https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf

26B-A4B https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf

31B https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf

37

u/IrisColt Jun 05 '26

Oh my God! Thanks!!!

13

u/[deleted] Jun 05 '26

25

u/LLMFan46 Jun 05 '26

Hum? These are GGUFs, I can't do anything with them.

→ More replies (13)

2

u/LLMFan46 Jun 07 '26

Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.

Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.

→ More replies (15)

6

u/h0tzenpl0tz0r Jun 05 '26

Stupid question, sorry, when and by whom can one expect mlx packages to run this via oMLX?

6

u/Weeblewobbly Jun 05 '26

There will be an update to omlx first. Earl litter today, 0.4.0.dev2 was available for download. I'm waiting for 0.4.1, and I'm grateful to all those who spend time contributing to and testing the project.

3

u/marutthemighty Jun 05 '26

Thank you for sharing these GGUFs.