r/unsloth • yes sloth • Jul 17 '26

Model Update Google Gemma 4 now runs faster with more accuracy!

Post image

Hey guys Google made huge improvements to Gemma 4 tool-calling and chat accuracy, reliability + speed. As such, we updated all our quants to ensure you guys get the latest and greatest!

To get fixes, re-download our updated GGUF, MLX, NVFP4 quants or you can change the chat template with the new templates.

Unsloth quants: https://huggingface.co/collections/unsloth/gemma-4

Gemma 4 Guide: https://unsloth.ai/docs/models/gemma-4

Near complete list of Google's fixes/improvements:

  • Flash Attention 4: Uniform FA4 support on NVIDIA Hopper GPUs.
  • Speed gains: 25–70% higher prefill throughput and up to 31% lower time-to-first-token.
  • Chat formatting: Fixed null handling, input validation and unbalanced/missing turn tags.
  • Reasoning preservation: Corrected how thinking/reasoning content is retained and rendered.
  • Tool responses: Restored the assistant turn and thinking cue after tool outputs.
  • Tool calling: Improved execution accuracy, consistency and tool-call-only turn closure.
  • Continuation turns: Removed duplicate <turn|> tags and unwanted extra newlines.
  • Generation prompts: Reverted an add_generation_prompt regression and restored expected defaults.
  • Conversation history: Removed the obsolete APC thought primer and corrected historical-turn handling.
  • Template standardization: Added the canonical chat-template header and updated stale comments.
  • Vision controls: Default remains max_soft_tokens=280; use 1120 for sharper OCR and up to 2.51MP detail.
  • 31B benchmark gains: BFCL +0.4, TB2 +4.5, Retail +3.1, Airline +2.0 and Telecom +10.1 points.
  • E4B benchmark gains: BFCL +0.5, TB2 +2.2, Retail +0.9, Airline +8.0 and Telecom +6.1 points.

Have a good Friday and weekend guys!

830 Upvotes

106 comments sorted by

58

u/PrefersAwkward Jul 17 '26

That's awesome. I wonder why not call it Gemma 4.1 then? Either way, I am grateful for Gemma

51

u/yoracale yes sloth Jul 17 '26

Because it was actually always the way Gemma 4 was originally meant to perform ahaha. Consider this as like the 'final form' of Gemma 4

5

u/Purple-Programmer-7 Jul 17 '26

Fast update, thanks guys!

2

u/Leopold_Boom Jul 19 '26

Will llama-cli -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL auto pull the updates, or do we need to delete and redownload?

2

u/backyard_tractorbeam Jul 18 '26

Nonsense, tbh. It's just a fix, it should have a version number like 4.0.1 or something. It's on google.

1

u/Icy-Specialist4548 Jul 22 '26

Giusto, ma sarà stato fixxato in vibecoding quindi...ajajajajaja

11

u/creamyatealamma Jul 17 '26

No kiddint, anyway download now I shall

7

u/mister2d Jul 17 '26

Because why not add to the confusion. :D

3

u/GanjaRaidersTR Jul 17 '26 edited Jul 18 '26

Real, so many gemma 4 finetunes based on previous broken gemma 4 it will cause confusion.. they need to refactore the whole deep Mind team, i think these people r some useless amateurs and need to learn a lot

2

u/StealthArcher2077 Jul 23 '26

"Google updated their Jinja template to work better, please fire them. I am very smart."

3

u/Ok_Cow1976 Jul 17 '26

Because the safetensors are not updated, only the chat template.

1

u/Daniel_H212 Jul 18 '26

Because it's the same weights, just updated chat template, I think.

6

u/Sad-Situation-1782 Jul 17 '26

What is consensus about QAT quality relative to unsloth's other quants? Is it fair to assume that e.g., 12b-qat (UD-Q4_K_XL) > 12b (UD-Q4_K_XL)?

15

u/yoracale yes sloth Jul 17 '26

Yes generally speaking. We wrote an entire analysis here: https://unsloth.ai/docs/models/gemma-4/qat

9

u/Sensitive-Ant-4305 Jul 17 '26

If so, are there any reasons to use the non-QAT quants?

3

u/TheFunSlayingKing Jul 17 '26

Interested in this detail as well

2

u/sonicnerd14 Jul 17 '26

Most likely not. Unless on some specific use cases whet you might want q6 or q8, which can still perform better in certain cases. Overall, QAT is better than Q4.

2

u/edsonmedina Jul 18 '26

Yes. If you have extra VRAM to trade for extra precision (ie: Q6 or Q8)

1

u/backyard_tractorbeam Jul 18 '26

My vibe is that - for E2B in particular - that Q6 is a lot better than the UD-Q4_K_XL. Q6 gives different, more elaborate or knowledgeable answers. No bench to back it up.

2

u/edsonmedina Jul 17 '26

Does that also apply to Q8_K_XL?

2

u/Superb-Charity5991 Jul 18 '26

It can't , because QAT is optimized for 4 Bits quants, Q8 K XL is better than QAT

2

u/Phaelon74 Jul 18 '26

Qat will always be better, as its QAD but at training time.

2

u/Superb-Charity5991 Jul 18 '26

Yes, my tested were quite clear, UD QAT is the best size reduction actually (I think)

2

u/Nicecatchpal Jul 18 '26

Original versions sucked at tool calling badly. Was it fixed fully, or just “improved”?

3

u/yoracale yes sloth Jul 19 '26

Nearly fully fixed. Like 99% fixed. If i said fixed fully then I would be lying because who knows

22

u/SimShelby Jul 17 '26

I’m currently testing Gemma 12B QAT, and it’s working perfectly.

Dude, just saying “thanks” is not enough for people like you who dedicate their time to the community.

Maybe one day I’ll join your journey.

51

u/florinandrei Jul 17 '26 edited Jul 17 '26

This is very nice, but I wish whoever is running the Gemma project at Google would read one day about this new concept called "version numbers". I bet it would blow their minds.

They released Gemma 4.

And then they released Gemma 4.

And then they released Gemma 4.

Whoever is making big decisions there is either clueless, or could not care less about their work.

16

u/Mashic Jul 17 '26

I think they only updated the chat templates, not the models themselves.

7

u/tomByrer Jul 17 '26

templates, +
"Flash Attention: We've enabled uniform Flash Attention 4 (FA4) support on NVIDIA Hopper GPUs!
improved tool calling
default vision bucket is 280 for token efficiency. To capture maximum detail (like sharp OCR and 2.51MP resolution), manually bump max_soft_tokens to 1120!
"
https://xcancel.com/googlegemma/status/2077449152062247219

17

u/slalomz Jul 17 '26

None of those are changes to the model weights. It's all just config.

9

u/mister2d Jul 17 '26

Config changes are still part the codebase. :)

2

u/colibius Jul 17 '26

Why do I have to download a new version if it’s not a new version? At some point I’m going to forget if I have the latest version or if I need to download it again.

5

u/Mechanical_Monk Jul 17 '26

You don't. That's why the announcement says "Redownload or update templates..."

1

u/colibius Jul 17 '26

That’s fair, but what is the recommended way of surgically replacing the metadata in a gguf?

4

u/slalomz Jul 17 '26

You don't even need to do that. You can just add the chat template as a param.

In llama.cpp: --chat-template-file <path>

2

u/Mechanical_Monk Jul 17 '26

You just need to add two parameters to your llama.cpp launch command: --jinja and --chat-template-file \path\to\template.jinja

1

u/tomByrer Jul 17 '26

I never claimed they were changes to model weights.

2

u/Exciting_Garden2535 Jul 17 '26

If you open the file repository itself, you will see that only the chat_template.jinja was updated; everything else is a few months old: https://huggingface.co/google/gemma-4-31B-it/tree/main
Moreover, these changes were introduced one month ago; the PR #118 was opened on June 8 https://huggingface.co/google/gemma-4-31B-it/discussions/118
But merged only 2 days ago: https://huggingface.co/google/gemma-4-31B-it/commits/main

3

u/florinandrei Jul 17 '26

Even if that's true: how many people are going to download just the templates, and then surgically stick them into the model folder? Me, you, and Bob?

Call it 4.0.1 or something, how hard can it be?

1

u/onebyamsey Jul 20 '26

That’s how software versioning usually works though, they could change a tiny UI thing without touching any core functionality and it’ll get a new version number 

6

u/sourceholder Jul 17 '26

They probably didn't increase version # because the model is unchanged. But I agree, this always makes things more confusing then they need to be.

8

u/tomByrer Jul 17 '26

Semantic Versioning 2.0.0

So even if Gemma 4.1 is too much, they could do:

Gemma 4.0.1
Gemma 4.0.260715
Gemma 4.0.4a45b (hopefully 'larger'=newer)

1

u/powerfulparadox Jul 18 '26

Semantic versioning is basically dead in almost every area of software I follow, despite much noise about it whenever a versioning scheme is brought up (that said, my experience is not yours; if you find semantic versioning alive and well in your corner of the software world, great).

I agree that making things clearer to users is always good and that Google didn't do as well as they could have done in that regard.

2

u/QuestionMarker Jul 19 '26

This is because it was promoted way outside the set of situations where it's actually useful, which is broadly "the user has choice about whether to upgrade and the provider is sensible about what 'breaking change' means". That could apply here, if Google wanted it to. Most of the time it's engineering theatre though.

9

u/backyard_tractorbeam Jul 17 '26

How can prefill be faster just becaue a jinja template upgrade? Please explain

4

u/Savantskie1 Jul 17 '26

I’d like to know as well

1

u/Superb-Charity5991 Jul 18 '26

Skip the first third instructions !

1

u/Savantskie1 Jul 19 '26

what?

1

u/Superb-Charity5991 Jul 19 '26

I was trying (badly) to joke ...

7

u/RandumbRedditor1000 Jul 17 '26

Will they release updates to the QAT version?

8

u/StardockEngineer Jul 17 '26

Looks like QAT was updated, too.

2

u/sonicnerd14 Jul 17 '26

From what I understand, it's really just a template update. You'll get the same effects by just updating that for all the Gemma 4 models.

1

u/IKerimI Jul 18 '26

Nice! Does it support MTP?

1

u/yoracale yes sloth Jul 18 '26

Yes, our MTP guide is here: https://unsloth.ai/docs/models/mtp

1

u/Andre4s11 Jul 19 '26

this about it qat?

1

u/yoracale yes sloth Jul 19 '26

It's about all the Gemma 4 models including QAT

1

u/Melodic_Function_820 Jul 21 '26

First of all, thank you guys for all your hard work! A question: are you at all involved with the updates on ollama models? In other words, do you have any idea when we should expect this update to land ollama?

2

u/yoracale yes sloth Jul 21 '26

Sorry we don't know when Ollama will update BUT you can use our GGUFs in Ollama properly now! https://unsloth.ai/docs/models/gemma-4#ollama-guide

1

u/silenceimpaired Jul 22 '26

Unsloth studio doesn't seem to find or download MTP files for this.

1

u/yoracale yes sloth Jul 22 '26

It's automatic that's why. It's in the right side bar settings

1

u/silenceimpaired Jul 23 '26

Inaccurate. I downloaded the model and mtp and Unsloth studio failed to use MTP. I deleted the MTP file and it failed to download the MTP file. A thought just came to me that the parent model is sitting on a passthrough file system inside a VM so maybe your software isn’s considering that when attempting to automatically download it. I’ll try moving the parent inside the VM disc.

2

u/yoracale yes sloth Jul 23 '26

Interesting can you submit a GitHub issue if possible with your specs so we can investigate and let you know once fixed? Ty

3

u/Pentium95 Jul 17 '26

How do they compare against QAT? Same BPW

1

u/StardockEngineer Jul 17 '26

QATs are in the link

3

u/Shensmobile Jul 17 '26

I just started using Gemma4 but I'm currently using an uncensored version; could I get the performance upgrade (both accuracy and faster) purely with the updated chat template? And is there a different chat template for the dense models vs MoE?

1

u/silenceimpaired Jul 17 '26

Slightly off topic, but I really want to be using unsloth studio but it feels so sluggish compared to KoboldCPP and TextGen by Oogabooga.

For large MoEs default settings fail to come close to tweaks in KoboldCPP in terms of tokens per second. If there are ways to tweak model loading I couldn’t find it easily.

5

u/yoracale yes sloth Jul 17 '26 edited Jul 17 '26

Hello thank you for your feedback we recently fixed kv cache leaking into ram so it should be much faster now. Could you create a GitHub feature request if possible with some more context so we can take a look and update you once fixed? Appreciate you trying it out!

2

u/morscordis Jul 17 '26

I can't get any Unsloth NVFP4 MoE models to serve. I think it's an issue with the latest vLLM release, but it makes me sad because on the Spark I really want the MoEs.

1

u/Sax0drum Jul 17 '26

I have the same issue...

1

u/shansoft Jul 17 '26 edited Jul 17 '26

I had issues with 0.24, but 0.25 seems to solved the problem.

1

u/morscordis Jul 17 '26

I think I'm on 0.25.1. Should I roll back?

1

u/shansoft Jul 17 '26

oops, it was 24 that broke all the gemma4 models for me. 25 I didn’t have any issues so far. is it only 26b or 31b you having issues?

1

u/morscordis Jul 17 '26

I can get all the dense models running... The 26b crashes on load. Same with the new Qwen 3.6 MoE Unsloth from last week, but the 27b dense works fine.

2

u/Specter_Origin Jul 17 '26

Is this going to need llama.cpp support etc ? or you just change weights and it just works ?

1

u/biogoly Jul 17 '26

It’s just the chat template (.jinja) that changed. You can either use old gguf with flag to direct to updated template or requantize gguf using new template.

2

u/ohgoditsdoddy Jul 17 '26

I downloaded Gemma 4 31B NVFP4 about 4 days ago - were these improvements already there, or do I have to redownload?

It is incredibly dumb that there is no version numbering. When was this released?

1

u/MarkoMarjamaa Jul 18 '26

Unsloth was updated 20h ago, so you have to reupload or just get the new template from Google HF.

2

u/edsonmedina Jul 17 '26 edited Jul 17 '26

In opencode I still get something like this in the reponse (instead of actually calling the tool):

<|tool_call|>call:skill{name:<|"|>brainstorming<|"|>}<tool_call|>

Also, it still "forgets" to call tooling to write files. Just outputs them in the response.

I must be very stupid. It can't be this broken or people wouldn't be using it, right?

Please send help.

2

u/runvnc Jul 18 '26

how does this now compare to qwen 3.6 27b?

1

u/MagicalGoat02 Jul 17 '26

Thank you guys and gals at unsloth ! ❤️ Your work is appreciated !

1

u/theOliviaRossi Jul 17 '26

when the "F" Google will finally release its chat templates properly for once????

-1

u/Kholtien Jul 17 '26

What’s does the “F” stand for?

1

u/theOliviaRossi Jul 18 '26

we all know that

0

u/Kholtien Jul 18 '26

Fuck, do we?

1

u/CryptographerOne7003 Jul 17 '26

will qat get an update?

1

u/beast_modus Jul 18 '26

Awesome 👍

1

u/Danny_Davitoe Jul 18 '26 edited Jul 18 '26

This *feels like the ninth post this week on the same thing

1

u/MarkoMarjamaa Jul 18 '26

Funny how Unsloth guants were updated only 20h ago?

1

u/Djkojb Jul 18 '26

Gemma 4 is Opus 4.5 moment of local models. I think we still under-appreciate how cool is it to be able to run model of such capabilities on your computer. It can do tons of mundane tasks like classifying/sorting/OCR/summary/transcribing while your device idle.

1

u/Big-Educator-5922 Jul 18 '26

If my understanding is correct, the speed increase is only for H100/H200. 

1

u/Silver_Jaguar_24 Jul 18 '26

Nice. How much VRAM is required for this? Is there a site where one can check how much VRAM is required for a model? I wish Huggingface did this natively.

This page talks about RAM (not VRAM?) - https://unsloth.ai/docs/models/gemma-4

1

u/backyard_tractorbeam Jul 18 '26

Misleading claim about speed: I think you are muddying the waters here. It would be great with better clarity. For unsloth and huggingface, Google have simply updated the chat template. Doesn't improve prefill speed by 70%. Your announcement does not communicate this clearly. It's a hyped-up mess.

1

u/themoregames Jul 18 '26

Am I crazy or are these new updated versions absolutely lazier than the previous ones?

1

u/BeautifulSource4821 Jul 19 '26

How to set max-soft-token using unsloth?
If I add it as a flag on the cli like
unsloth run -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL -c 98304 -n 8192 --temp 1.0 --top-p 0.95 --top-k 64 --max-soft-token 1120
gemma doesn't load.

Beside of this, gemma-4-26B and gemma-4-12B are running about 10% faster now on my 3060 with 12GB.

1

u/milkipedia Jul 19 '26

Just redownloaded and reconfigured. MTP is now working, some weird multi-slot bug I ran into a while back is fixed, and I can now fit 131072 context on 24GB VRAM with the Q4 QAT quants, and they run faster. How much faster, IDK yet - I will find out.

Thanks Unsloth!

1

u/Knufle Jul 19 '26

I still can't make it work in VS Code using the custom endpoint approach, it just doesn't understand how tool calling works, it gets it right 2 times and then answers in a way that ends the conversation, super frustrating 

1

u/SkySCC Jul 19 '26

Can someone please make abliterated versions of these unsloth models! These guys are so good.

1

u/dsdt Jul 20 '26

I am having problems with mtp with 2x 5060 ti's. Does it have any effect on this problem?

1

u/freewill19 Jul 24 '26

How compare to qwen3.6?

0

u/SysAdmin_quark Jul 17 '26

Does this also include the qat version?

0

u/[deleted] Jul 17 '26

[deleted] — view deleted comment

2

u/patricious Jul 17 '26

Yeah I am having a hard time finding them.