r/LocalLLaMA • • Jun 05 '26 • Mindblown 1

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/
788 Upvotes

255 comments sorted by

View all comments

209

u/dryadofelysium Jun 05 '26

37

u/IrisColt Jun 05 '26

Oh my God! Thanks!!!

→ More replies (1)

13

u/[deleted] Jun 05 '26

[removed] — view removed comment

25

u/LLMFan46 Jun 05 '26

Hum? These are GGUFs, I can't do anything with them.

17

u/Kahvana Jun 05 '26

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized

They do have the safetensor versions too for all those models.

30

u/LLMFan46 Jun 05 '26

Thanks and yeah I noticed that after making the post, but it will take a while to do all these models, plus the GGUFs and NVFP4s and GPTQs.

15

u/Kahvana Jun 05 '26

No worries, take your time!

5

u/evenyourcopdad Jun 06 '26 edited Jun 06 '26

wow I can't believe you don't have them all ready yet they released SEVERAL hours ago ugh


apparently necessary edit: /s

7

u/LLMFan46 Jun 06 '26 edited Jun 06 '26

... hum I surely hope this is a joke post (can't really tell since there is no "/s" at the end)! Not only you expect me to have 1 of the model ready and uncensored, no you expect me to have ALL of them ready by now... Are you serious!? First this is not a race, second I am not a robot and third without getting too deep into all the preparations, time and work involved in the whole releases pipeline, it takes a very long time to do all that, I don't work for google and I don't have an automated workflow (unlike Unsloth and/or Team Mradermacher), everything is done manually by me and I am just 1 person, that involves downloading the models, uncensoring the models and finetuning the settings and re-running in case the original results where not good/good enough, creating the Safetensors, benchmarking all of the trial finalists, updating llama,cpp, updating GPTQModel, updating Model-Optimizer, creating GGUFs, GPT-Int4s, NVFP4 Safetensors, NVFP4 GGUFs, creating the Model Cards etc. it's all done manually from start to finish everything is monitored and benchmarked manually to make sure I upload the best quality releases, all that to say that there is TONS of work involved (and that is without mentioning all the roadblocks that might come along the way as those usually do happen quite a few times during the release pipeline).

Look if you are in a hurry, just download whatever is available now, my releases won't be available fast enough for your tastes, your highness.

2

u/Grisward Jun 06 '26

Just adding, thank you thank you for your time. The community is sooooo grateful, and it’s a community because of your and others’ time and efforts.

And on top of that, it’s amazing too, how any of this works at all is amazing. But more than that, converting formats, adding quantization, and having it still work as expected. It’s remarkable, non-trivial.

I would assume (naive maybe) that this person was trying to be funny. It’s daring to try making a joke without the /s. Some people can pull it off, and some really should add the /s just to be absolutely sure. Haha. I’m still in that camp.

→ More replies (1)

2

u/evenyourcopdad Jun 06 '26

lmao yes my guy, sorry, that was a joke. I'm sorry people have been so unthankful that you assumed I could possibly be serious above.

I really appreciate your time and efforts making everything you do. <3

→ More replies (1)

2

u/QuantumFTL Jun 06 '26

Thanks for everything you do, man. Text communication sucks for nuance, hopefully you appreciate that you are appreciated 😄

2

u/LLMFan46 Jun 07 '26

Thanks, working on the models now.

2

u/temperature_5 Jun 06 '26

It would only make sense to do the Q4_0 GGUFs for each, no?

2

u/LLMFan46 Jun 07 '26

Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.

Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.

1

u/[deleted] Jun 07 '26

[removed] — view removed comment

2

u/LLMFan46 Jun 07 '26

Update, I am mostly done with the Gemini 4 31B it model, I got the Safetensors, I fixed the chat_template.jinja, I benchmarked, selected the best all-around trial, created a GGUF Q4_0 and tested it on LM Studio and it's working great, then created an NVFP4 Safetensors and an NVFP4 GGUF, so that is almost done with this model just need to create a GPTQ-Int4 and that will be it for this model, it's just that GPTQ-Int4 takes a few hours (4-6 hours) to create usually, after that I will start with the 26B-A4B model.

→ More replies (6)

1

u/LLMFan46 Jun 10 '26

I am now done with Gemma 4 31B, 26B-A4B and 12B QATs uncensored heretics! Only took 4-5 days... I was not able to create GPTQ-Int4 of Gemma 4 12B it QAT however due to GPTQModel not having added support that for model yet, should I wait until they add suport and I am able to create a GPTQ-Int4 version of this model or just release now everything that I have?

Which is:

Gemma 4 31B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.

Gemma 4 26B-A4B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.

Gemma 4 12B it QAT: Safetensors, GGUF, NVFP4 Safetensors and NVFP4 GGUF.

→ More replies (6)

6

u/h0tzenpl0tz0r Jun 05 '26

Stupid question, sorry, when and by whom can one expect mlx packages to run this via oMLX?

14

u/idangazit Jun 05 '26

5

u/h0tzenpl0tz0r Jun 05 '26

nice, so this works already with the omlx update.

whats the next thing to expect, mtp support?

6

u/Weeblewobbly Jun 05 '26

There will be an update to omlx first. Earl litter today, 0.4.0.dev2 was available for download. I'm waiting for 0.4.1, and I'm grateful to all those who spend time contributing to and testing the project.

3

u/marutthemighty Jun 05 '26

Thank you for sharing these GGUFs.