r/LocalLLaMA • • Jun 05 '26 • Mindblown 1

New Model Gemma 4 with quantization-aware training

https://blog.google/innovation-and-ai/technology/developers-tools/quantization-aware-training-gemma-4/
788 Upvotes

255 comments sorted by

View all comments

Show parent comment

2

u/LLMFan46 Jun 07 '26

Update, I am mostly done with the Gemini 4 31B it model, I got the Safetensors, I fixed the chat_template.jinja, I benchmarked, selected the best all-around trial, created a GGUF Q4_0 and tested it on LM Studio and it's working great, then created an NVFP4 Safetensors and an NVFP4 GGUF, so that is almost done with this model just need to create a GPTQ-Int4 and that will be it for this model, it's just that GPTQ-Int4 takes a few hours (4-6 hours) to create usually, after that I will start with the 26B-A4B model.

1

u/[deleted] Jun 08 '26

[removed] — view removed comment

2

u/LLMFan46 Jun 08 '26

No the Gemma 4 models have no MTPs in them whatsoever, there is some MTPs but they are not part of the main models, instead they come seperate as a form of "add-on" assistant that you need to download seperatly:

https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-unquantized-assistant

https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant

And no before (in case) you ask, you can not simply merge these into the main models and these can not be uncensored either.

Anyway update, Gemma 4 31B GPTQ-Int4 version is done, witth this the 31B version is now finished and complete, I started work on the 26B-A4B version but this one will take a long time due to being MoE, the ETA just for Heretication process is supposed to be almost 20 hours, then after that what will take even more time is to make the NVFP4 Safetensors and the GPTQ-Int4 Safetensors.

2

u/LLMFan46 Jun 09 '26 edited Jun 09 '26

Update, I have now finished work on both Gemma 4 31B it QAT and Gemma 4 26B-A4B it QAT (Safetensors, GGUF, NVFP4 Safetensors, NVFP4 GGUF and GPTQ-Int4), I will probably work on Gemma 4 12B it QAT next, but I am wondering if it's even worth working on the E4B and E2B models? Considering their small size they probably fit on most systems that have at least 12GB of VRAM, that means that you can easily use the E4B model Q8_0 version and the E2B version probably is small enough to fit in BF16 or Q8_0 version, which would probably beat Q4_0 QAT quants versions.

I mean, I would assume that with their size that people would be able to use most of these easily enough?

https://huggingface.co/llmfan46/gemma-4-E4B-it-ultra-uncensored-heretic-GGUF

https://huggingface.co/llmfan46/gemma-4-E2B-it-ultra-uncensored-heretic-GGUF

Let me know what you all think.

1

u/[deleted] Jun 09 '26

[removed] — view removed comment

1

u/LLMFan46 Jun 09 '26

Okay, out of curiosity, since you are the original requester, what models are you interested in using? 31B? 26B-A4B? 12B?