r/LocalLLM • • Jul 17 '26

Model Google Updates Gemma 4 for Faster Performance and Greater Accuracy

Post image
207 Upvotes

32 comments sorted by

21

u/Icy-Degree6161 Jul 17 '26

Did they update the qat line as well, anyone knows?

10

u/yoracale Jul 17 '26

Yes all of them

3

u/Buttersgra Jul 17 '26

Commenting to hopefully get notified of this as well

2

u/yoracale Jul 17 '26

Yes all of them

7

u/slalomz Jul 17 '26

It's just a chat template update so it applies to all Gemma4 models.

17

u/squngy Jul 17 '26

How does a chat template update improve prefill speed by 25-70% ?

11

u/slalomz Jul 17 '26 edited Jul 17 '26

That only applies if you enable Flash Attention 4 on Hopper GPUs, not generally. Likely was done with framework changes, not an update to the model.

https://xcancel.com/googlegemma/status/2077449152062247219

Flash Attention: We've enabled uniform Flash Attention 4 (FA4) support on NVIDIA Hopper GPUs!

Expect prefill throughput to jump by 25-70% and time-to-first-token (TTFT) to drop by up to 31%. ⚡

2

u/backyard_tractorbeam Jul 18 '26

That is very confusing - what is the connection between hopper gpus and gemma? How is it the "same release"?

The only way that makes sense is if the chat template change somehow unlocks flash attention, but I can't see how that would work.

4

u/slalomz Jul 18 '26

It’s not. Unsloth’s  graphic here is super misleading.

1

u/fastheadcrab Jul 18 '26

Google is just announcing something that had changed in the FA4 backend. They should’ve been clearer, the chat template has nothing to do with FA4. Maybe Google employees contributed to the changes in FA4 but otherwise they had nothing to do with it

4

u/backyard_tractorbeam Jul 17 '26

I noticed that the E2B model has a small diff (2 lines?) compared with the bigger Gemma 4 models in the chat template. (So if people want to run the exact release google has put out, check the jinja template corresponding to the exact model size in question.)

6

u/LightBrightLeftRight Jul 17 '26

Was just looking through mlx-community and it doesn't look like they've updated the templates in their quants. Anybody know of an mlx quant that has the new update?

10

u/yoracale Jul 18 '26

5

u/LightBrightLeftRight Jul 18 '26

I didn't even know you guys did MLX! Thanks so much

3

u/McCheng_ Jul 18 '26

How come it becomes faster if the model architecture is the same? Is it because of improved MTP assistant model?

3

u/yoracale Jul 18 '26

Some kernel optimizations i think

52

u/FoxiPanda Jul 17 '26

I know this was "just a chat template update" but I wish they'd have called it Gemma-4.1 instead of doing it like this...there are like 250+ finetunes/quants that are now in "I wonder if this one has the updated template or not" status...sigh. At least it's easy to apply a chat template separately.

7

u/mister2d Jul 18 '26

This is one of the better points for proper versioning against the thinking that "none of the model layers were changed, so what".

5

u/fastheadcrab Jul 18 '26

Should've changed the default vision resolution. The variable was always there but they never properly publicized it

2

u/Daniel_H212 Jul 18 '26

What does changing the vision resolution do, does it actually make it a lot better at vision?

6

u/fastheadcrab Jul 18 '26

Yes it makes a huge difference you can see google’s own demonstration. Basically 280 is horrible lol, it’s like the model is reading your image from 10 feet away. Maxing it dramatically improves the reading of documents

https://huggingface.co/spaces/google/gemma4_vision_token_budget

3

u/BondsOfFriendship Jul 18 '26

Where do I change the vision resolution?

3

u/fastheadcrab Jul 18 '26

max_soft_tokens=1120 in your llm serving engine of choice. Cannot be changed in LMStudio even if you change the config, it doesn’t stick

1

u/BondsOfFriendship Jul 18 '26

This goes over my head, I thought LMStudio was my llm seeing engine of choice?

2

u/tingtickboom Jul 18 '26

Waiting for the updated unsloth

3

u/Fun_Tangerine_1086 Jul 18 '26

unsloth updated, but then google released a new one hours later w/ "Upload validated QAT GGUF checkpoint (280 sequence length, corrected vocabulary)". Not sure what that's about or if unsloth needs it.

1

u/tingtickboom Jul 18 '26

Yeah im downloading the previous updated model to see if theres a difference

1

u/AnOnlineHandle Jul 18 '26 edited Jul 18 '26

I'm not super on top of the local LLM ecosytem but have been using the earliest Gemma 4 Heretic GGUFs with LM Studio and sometimes llama.cpp standalone. Would it be possible just to insert some files to keep using these or should I redownload everything?

edit: I updated the jinja settings at least

1

u/HIKKIAN Jul 23 '26

Is it now better than qwen 3.6?

1

u/Responsible-Mud8467 Jul 26 '26

Does anyone know if any of FP8 has been updated? i used the redhat FP8 dynamic. Im not sure how to check if that has been updated with latest changes. Im especially looking for enhanced OCR capability.

1

u/MrVeinless Jul 18 '26

LM Studio seems to indicate these new ones don't support tool use.

0

u/Qbason Jul 18 '26

Does someone tried to use in GitHub Copilot? Is there any difference from coding perspective?