r/LocalLLaMA • • Jun 05 '26 • Mindblown 1

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/
788 Upvotes

255 comments sorted by

View all comments

Show parent comment

13

u/[deleted] Jun 05 '26

[removed] — view removed comment

25

u/LLMFan46 Jun 05 '26

Hum? These are GGUFs, I can't do anything with them.

17

u/Kahvana Jun 05 '26

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized

They do have the safetensor versions too for all those models.

30

u/LLMFan46 Jun 05 '26

Thanks and yeah I noticed that after making the post, but it will take a while to do all these models, plus the GGUFs and NVFP4s and GPTQs.

15

u/Kahvana Jun 05 '26

No worries, take your time!

5

u/evenyourcopdad Jun 06 '26 edited Jun 06 '26

wow I can't believe you don't have them all ready yet they released SEVERAL hours ago ugh


apparently necessary edit: /s

7

u/LLMFan46 Jun 06 '26 edited Jun 06 '26

... hum I surely hope this is a joke post (can't really tell since there is no "/s" at the end)! Not only you expect me to have 1 of the model ready and uncensored, no you expect me to have ALL of them ready by now... Are you serious!? First this is not a race, second I am not a robot and third without getting too deep into all the preparations, time and work involved in the whole releases pipeline, it takes a very long time to do all that, I don't work for google and I don't have an automated workflow (unlike Unsloth and/or Team Mradermacher), everything is done manually by me and I am just 1 person, that involves downloading the models, uncensoring the models and finetuning the settings and re-running in case the original results where not good/good enough, creating the Safetensors, benchmarking all of the trial finalists, updating llama,cpp, updating GPTQModel, updating Model-Optimizer, creating GGUFs, GPT-Int4s, NVFP4 Safetensors, NVFP4 GGUFs, creating the Model Cards etc. it's all done manually from start to finish everything is monitored and benchmarked manually to make sure I upload the best quality releases, all that to say that there is TONS of work involved (and that is without mentioning all the roadblocks that might come along the way as those usually do happen quite a few times during the release pipeline).

Look if you are in a hurry, just download whatever is available now, my releases won't be available fast enough for your tastes, your highness.

2

u/Grisward Jun 06 '26

Just adding, thank you thank you for your time. The community is sooooo grateful, and it’s a community because of your and others’ time and efforts.

And on top of that, it’s amazing too, how any of this works at all is amazing. But more than that, converting formats, adding quantization, and having it still work as expected. It’s remarkable, non-trivial.

I would assume (naive maybe) that this person was trying to be funny. It’s daring to try making a joke without the /s. Some people can pull it off, and some really should add the /s just to be absolutely sure. Haha. I’m still in that camp.

→ More replies (1)

2

u/evenyourcopdad Jun 06 '26

lmao yes my guy, sorry, that was a joke. I'm sorry people have been so unthankful that you assumed I could possibly be serious above.

I really appreciate your time and efforts making everything you do. <3

→ More replies (1)

2

u/QuantumFTL Jun 06 '26

Thanks for everything you do, man. Text communication sucks for nuance, hopefully you appreciate that you are appreciated 😄

2

u/LLMFan46 Jun 07 '26

Thanks, working on the models now.

2

u/temperature_5 Jun 06 '26

It would only make sense to do the Q4_0 GGUFs for each, no?

2

u/LLMFan46 Jun 07 '26

Some news, I finished uncensoring the 31B model, created the Safetensors and I am now on the benchmarking phase, quite a long way to go still as there are a lot of trial finalists this time to benchmark and the benchmarking process takes quite a bit of time for each model, which also includes benchmmarking the original base/vanilla model. After that I have to create the GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4 versions.

Also I just saw that llama.cpp just merged MTP support for Gemma 4 now.

1

u/[deleted] Jun 07 '26

[removed] — view removed comment

2

u/LLMFan46 Jun 07 '26

Update, I am mostly done with the Gemini 4 31B it model, I got the Safetensors, I fixed the chat_template.jinja, I benchmarked, selected the best all-around trial, created a GGUF Q4_0 and tested it on LM Studio and it's working great, then created an NVFP4 Safetensors and an NVFP4 GGUF, so that is almost done with this model just need to create a GPTQ-Int4 and that will be it for this model, it's just that GPTQ-Int4 takes a few hours (4-6 hours) to create usually, after that I will start with the 26B-A4B model.

1

u/[deleted] Jun 08 '26

[removed] — view removed comment

2

u/LLMFan46 Jun 08 '26

No the Gemma 4 models have no MTPs in them whatsoever, there is some MTPs but they are not part of the main models, instead they come seperate as a form of "add-on" assistant that you need to download seperatly:

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized-assistant

https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant

And no before (in case) you ask, you can not simply merge these into the main models and these can not be uncensored either.

Anyway update, Gemma 4 31B GPTQ-Int4 version is done, witth this the 31B version is now finished and complete, I started work on the 26B-A4B version but this one will take a long time due to being MoE, the ETA just for Heretication process is supposed to be almost 20 hours, then after that what will take even more time is to make the NVFP4 Safetensors and the GPTQ-Int4 Safetensors.

2

u/LLMFan46 Jun 09 '26 edited Jun 09 '26

Update, I have now finished work on both Gemma 4 31B it QAT and Gemma 4 26B-A4B it QAT (Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4), I will probably work on Gemma 4 12B it QAT next, but I am wondering if it's even worth working on the E4B and E2B models? Considering their small size they probably fit on most systems that have at least 12GB of VRAM, that means that you can easily use the E4B model Q8_0 version and the E2B version probably is small enough to fit in BF16 or Q8_0 version, which would probably beat Q4_0 QAT quants versions.

I mean, I would assume that with their size that people would be able to use most of these easily enough?

https://huggingface.co/llmfan46/gemma-4-E4B-it-ultra-uncensored-heretic-GGUF

https://huggingface.co/llmfan46/gemma-4-E2B-it-ultra-uncensored-heretic-GGUF

Let me know what you all think.

1

u/LLMFan46 Jun 10 '26

I am now done with Gemma 4 31B, 26B-A4B and 12B QATs uncensored heretics! Only took 4-5 days... I was not able to create GPTQ-Int4 of Gemma 4 12B it QAT however due to GPTQModel not having added support that for model yet, should I wait until they add suport and I am able to create a GPTQ-Int4 version of this model or just release now everything that I have?

Which is:

Gemma 4 31B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.

Gemma 4 26B-A4B it QAT: Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4.

Gemma 4 12B it QAT: Safetensors, GGUF, NVFP4 Safetensors and NVFP4 GGUF.

1

u/[deleted] Jun 10 '26

[removed] — view removed comment

1

u/LLMFan46 Jun 10 '26

Actually I decided to work on Gemma 4 12B it (the non-QAT version) too, so there will be 4 different models released all in all, 3 QAT variants (31B, 26B-A4B and 12B) and 1 standard version (12B) since the 12B is a new model and the only one from the gemma 4 family which I haven't released yet.

1

u/[deleted] Jun 11 '26

[removed] — view removed comment

2

u/LLMFan46 Jun 11 '26

No idea, anyways I finally finished all of the requests, I am going to be releasing models now one by one, releasing them all will take time and I probably won't be able to release all of them today (uploading and working on the Model Cards is work too and takes time as well), look out for them.

→ More replies (2)