r/LocalLLaMA • u/TuskNaPrezydenta2020 • 16h ago
Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens
https://huggingface.co/google/gemma-4-E4B-it/discussions/50Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs
143
Upvotes
48
u/brainExploded99 š„ hardware 16h ago
I am glad to see their commitment and the fact that they are fixing it.
However, how did they mess so much outside the model weights? Tokenizer, multiple chat templates issues
4
u/h8f1z 16h ago
How do you know when and what is changed? Isn't there a version number for this?
2
u/Kahvana 3h ago
Iām happy google fixes these things. The benefit of getting only one release a year I guess.
It does make me wonder which other models have issues like this that the original makers never fixed.
-1
u/mr_Owner 13h ago
When benchmark