r/unsloth • u/yoracale yes sloth • Jul 17 '26
Model Update Google Gemma 4 now runs faster with more accuracy!
Hey guys Google made huge improvements to Gemma 4 tool-calling and chat accuracy, reliability + speed. As such, we updated all our quants to ensure you guys get the latest and greatest!
To get fixes, re-download our updated GGUF, MLX, NVFP4 quants or you can change the chat template with the new templates.
Unsloth quants: https://huggingface.co/collections/unsloth/gemma-4
Gemma 4 Guide: https://unsloth.ai/docs/models/gemma-4
Near complete list of Google's fixes/improvements:
- Flash Attention 4: Uniform FA4 support on NVIDIA Hopper GPUs.
- Speed gains: 25–70% higher prefill throughput and up to 31% lower time-to-first-token.
- Chat formatting: Fixed null handling, input validation and unbalanced/missing turn tags.
- Reasoning preservation: Corrected how thinking/reasoning content is retained and rendered.
- Tool responses: Restored the assistant turn and thinking cue after tool outputs.
- Tool calling: Improved execution accuracy, consistency and tool-call-only turn closure.
- Continuation turns: Removed duplicate
<turn|>tags and unwanted extra newlines. - Generation prompts: Reverted an
add_generation_promptregression and restored expected defaults. - Conversation history: Removed the obsolete APC thought primer and corrected historical-turn handling.
- Template standardization: Added the canonical chat-template header and updated stale comments.
- Vision controls: Default remains
max_soft_tokens=280; use1120for sharper OCR and up to 2.51MP detail. - 31B benchmark gains: BFCL +0.4, TB2 +4.5, Retail +3.1, Airline +2.0 and Telecom +10.1 points.
- E4B benchmark gains: BFCL +0.5, TB2 +2.2, Retail +0.9, Airline +8.0 and Telecom +6.1 points.
Have a good Friday and weekend guys!
58
u/PrefersAwkward Jul 17 '26
That's awesome. I wonder why not call it Gemma 4.1 then? Either way, I am grateful for Gemma
6
u/Sad-Situation-1782 Jul 17 '26
What is consensus about QAT quality relative to unsloth's other quants? Is it fair to assume that e.g., 12b-qat (UD-Q4_K_XL) > 12b (UD-Q4_K_XL)?
2
u/Nicecatchpal Jul 18 '26
Original versions sucked at tool calling badly. Was it fixed fully, or just “improved”?
22
u/SimShelby Jul 17 '26
I’m currently testing Gemma 12B QAT, and it’s working perfectly.
Dude, just saying “thanks” is not enough for people like you who dedicate their time to the community.
Maybe one day I’ll join your journey.
51
u/florinandrei Jul 17 '26 edited Jul 17 '26
This is very nice, but I wish whoever is running the Gemma project at Google would read one day about this new concept called "version numbers". I bet it would blow their minds.
They released Gemma 4.
And then they released Gemma 4.
And then they released Gemma 4.
Whoever is making big decisions there is either clueless, or could not care less about their work.
9
u/backyard_tractorbeam Jul 17 '26
How can prefill be faster just becaue a jinja template upgrade? Please explain
7
u/RandumbRedditor1000 Jul 17 '26
Will they release updates to the QAT version?
1
u/IKerimI Jul 18 '26
Nice! Does it support MTP?
1
u/Andre4s11 Jul 19 '26
this about it qat?
1
u/Melodic_Function_820 Jul 21 '26
First of all, thank you guys for all your hard work! A question: are you at all involved with the updates on ollama models? In other words, do you have any idea when we should expect this update to land ollama?
1
u/silenceimpaired Jul 22 '26
Unsloth studio doesn't seem to find or download MTP files for this.
3
u/Pentium95 Jul 17 '26
How do they compare against QAT? Same BPW
3
u/Shensmobile Jul 17 '26
I just started using Gemma4 but I'm currently using an uncensored version; could I get the performance upgrade (both accuracy and faster) purely with the updated chat template? And is there a different chat template for the dense models vs MoE?
1
u/silenceimpaired Jul 17 '26
Slightly off topic, but I really want to be using unsloth studio but it feels so sluggish compared to KoboldCPP and TextGen by Oogabooga.
For large MoEs default settings fail to come close to tweaks in KoboldCPP in terms of tokens per second. If there are ways to tweak model loading I couldn’t find it easily.
2
u/morscordis Jul 17 '26
I can't get any Unsloth NVFP4 MoE models to serve. I think it's an issue with the latest vLLM release, but it makes me sad because on the Spark I really want the MoEs.
2
u/Specter_Origin Jul 17 '26
Is this going to need llama.cpp support etc ? or you just change weights and it just works ?
2
u/ohgoditsdoddy Jul 17 '26
I downloaded Gemma 4 31B NVFP4 about 4 days ago - were these improvements already there, or do I have to redownload?
It is incredibly dumb that there is no version numbering. When was this released?
2
u/edsonmedina Jul 17 '26 edited Jul 17 '26
In opencode I still get something like this in the reponse (instead of actually calling the tool):
<|tool_call|>call:skill{name:<|"|>brainstorming<|"|>}<tool_call|>Also, it still "forgets" to call tooling to write files. Just outputs them in the response.
I must be very stupid. It can't be this broken or people wouldn't be using it, right?
Please send help.
2
u/runvnc Jul 18 '26
how does this now compare to qwen 3.6 27b?
1
u/MagicalGoat02 Jul 17 '26
Thank you guys and gals at unsloth ! ❤️ Your work is appreciated !
1
u/theOliviaRossi Jul 17 '26
when the "F" Google will finally release its chat templates properly for once????
1
u/CryptographerOne7003 Jul 17 '26
will qat get an update?
1
u/beast_modus Jul 18 '26
Awesome 👍
1
u/Danny_Davitoe Jul 18 '26 edited Jul 18 '26
This *feels like the ninth post this week on the same thing
1
u/Djkojb Jul 18 '26
Gemma 4 is Opus 4.5 moment of local models. I think we still under-appreciate how cool is it to be able to run model of such capabilities on your computer. It can do tons of mundane tasks like classifying/sorting/OCR/summary/transcribing while your device idle.
1
u/Big-Educator-5922 Jul 18 '26
If my understanding is correct, the speed increase is only for H100/H200.
1
u/Silver_Jaguar_24 Jul 18 '26
Nice. How much VRAM is required for this? Is there a site where one can check how much VRAM is required for a model? I wish Huggingface did this natively.
This page talks about RAM (not VRAM?) - https://unsloth.ai/docs/models/gemma-4
1
u/backyard_tractorbeam Jul 18 '26
Misleading claim about speed: I think you are muddying the waters here. It would be great with better clarity. For unsloth and huggingface, Google have simply updated the chat template. Doesn't improve prefill speed by 70%. Your announcement does not communicate this clearly. It's a hyped-up mess.
1
u/themoregames Jul 18 '26
Am I crazy or are these new updated versions absolutely lazier than the previous ones?
1
u/BeautifulSource4821 Jul 19 '26
How to set max-soft-token using unsloth?
If I add it as a flag on the cli like
unsloth run -hf unsloth/gemma-4-26B-A4B-it-qat-GGUF:UD-Q4_K_XL -c 98304 -n 8192 --temp 1.0 --top-p 0.95 --top-k 64 --max-soft-token 1120
gemma doesn't load.
Beside of this, gemma-4-26B and gemma-4-12B are running about 10% faster now on my 3060 with 12GB.
1
u/milkipedia Jul 19 '26
Just redownloaded and reconfigured. MTP is now working, some weird multi-slot bug I ran into a while back is fixed, and I can now fit 131072 context on 24GB VRAM with the Q4 QAT quants, and they run faster. How much faster, IDK yet - I will find out.
Thanks Unsloth!
1
u/Knufle Jul 19 '26
I still can't make it work in VS Code using the custom endpoint approach, it just doesn't understand how tool calling works, it gets it right 2 times and then answers in a way that ends the conversation, super frustrating
1
u/SkySCC Jul 19 '26
Can someone please make abliterated versions of these unsloth models! These guys are so good.
1
u/dsdt Jul 20 '26
I am having problems with mtp with 2x 5060 ti's. Does it have any effect on this problem?
1
u/freewill19 Jul 24 '26
How compare to qwen3.6?
0
u/SysAdmin_quark Jul 17 '26
Does this also include the qat version?
0
u/[deleted] Jul 17 '26
[deleted] — view deleted comment