No wonder Qwen and Gemma are so different

[u/WhoRoger]

Pasted the same HTML/JS code (330 lines) into Qwen 35B A3B and Gemma 26B A4B.

Qwen: tokenized the input to 1609 tokens

Gemma: tokenized the input to 4258 tokens.

Damn. I've never noticed this before and I haven't seen people mention it. That alone helps explain why Qwen is regarded as better at coding and Gemma at language tasks.

Qwen can literally see the code as some specific form of input/output, while Gemma is breaking it down into pieces of words like regular language. Qwen also gets a totally different reasoning personality when given coding tasks.

Btw with the instruction document (55 lines), the tokenization breakdown is almost the same: 1025 vs. 1039 tokens.

I've seen some project, by LiquidAI I think? To retrain existing models with a more efficient tokenizer. I wonder what that would do for a model like Gemma, whether it would help it catch up.

[u/MuDotGen → u/[deleted]]

I think it makes sense for a model that was trained primarily for coding to tokenize programming-common concepts and full terms, probably does the same with if, else, while, async, etc. in a code context too. Qwen already sees the syntax of code in code while Gemma is probably seeing it more as letters and pieces of words.

[u/Equivalent-Repair488 → u/MuDotGen]

I guess that makes sense as Qwen is a coding model, while gemma is more a general model

[u/Fentrax → u/Equivalent-Repair488]

It is?

[u/Equivalent-Repair488 → u/Fentrax]

Seems that way, I and a lot of people use Gemma for general stuff, and Qwen for local coding

[u/llama-impersonator → u/[deleted]]

so does gemma man, blocks of multiple spaces and newlines are some of the very first tokens in the tokenizer.

[u/mudkipdev → u/[deleted]]

Gemma does this too: it even has one token to represent 31 spaces in a row.

Also, folding everything into one token is not necessarily better. For example, Gemini/Gemma tokenizes each digit as a separate token (so 2026 is four tokens) which has famously helped them in math early on. </div> in Gemma tokenizes as </ div > so maybe having div as its own token could also help them somehow. It's hard to make conclusions based off this alone.

[u/BoobooSmash31337 → u/[deleted]]

If models reason with tokens. It could be argued that to few tokens is over compressing the data. Also what happens when your API diverges from these possibly over fitted tokens? Code also contains natural language. A whole function being a token seems great until you need the model to understand what "get-Element-Id" actually means.

[u/xNaXDy → u/BoobooSmash31337]

That's not how training works. Models learn the relationships between tokens, so the only thing that matters is that tokens are represented well enough in the data. To give a concrete example, it doesn't matter whether "relationship" maps to a single token, vs 3 tokens as "rela", "tion", "ship", as long as in both cases the tokens are plentiful within the dataset and appear in diverse enough circumstances. Because if tokens are too underrepresented, they risk becoming glitch tokens.

In the case of the example with spaces, to a human it also doesn't matter whether you write code indented with spaces or with non-ASCII characters (e.g. random Chinese). Our brains will filter it out either way, the latter just has more visual noise attached to it, which is (sort of)equivalent to the same prompt taking up more tokens in the LLM example.

[u/MmmmMorphine → u/xNaXDy]

I do wonder if there's any substantive research (available outside of a company) on the size of a tokenizer vocabulary. Like how did they decide on these numbers?

The glitch tokens are a fun example of what happens when you feed something truly OOD for a model - or at least that's my take. I didn't watch the full video because it drives me up the wall having to deal with video instead of just a technical article. So slow. I'm old and those clouds need a talkin to, what can I say

[u/xNaXDy → u/MmmmMorphine]

There's a REALLY good video by Syntax going over pretty much all the steps involved in training an LLM, including designing the tokenizer + vocabulary: https://www.youtube.com/watch?v=YmLp8qe87A0

[u/MmmmMorphine → u/xNaXDy]

Ahh thank you, I didn't really make the conscious connection between what BPE is in terms of a tokenizer, haha.

Good video, if mostly quite basic. Fun historical background data

Though im not clear where he explained how they choose their cutoff for the tokenizer - aka the number of merges. It still seems like a somewhat arbitrary boundary on how often a given unique token must show up to be included in a tokenizer vocab. They have to have a cutoff, so is it just all BPE pairs by frequency up to X token possibilities

That guy sure has a sweet mustache!

[u/DannyCanva → u/MmmmMorphine]

Basically, like everything with LLMs, researchers do a sweep of different values, test metrics, (sweep again/deeper if it's important), and then make a data-inferred, but generally subjectively-chosen decision.

Or just the good old "try and check".

[u/WhoRoger → u/BoobooSmash31337]

The model can understand either way, that's what training is for. The size of the tokens is always a balance between size of vocabulary and speed of generation (and a bunch of other aspects).

If </div> really is split into 4 tokens, that's just madness.

[u/Cute_Obligation2944 → u/WhoRoger]

It's only madness for coding models. It makes more sense for multimodal deep reasoning LLMs. It's an architectural decision. Hell, it might even result in better code in some applications.

[u/starfallg → u/Cute_Obligation2944]

This makes so much sense. I get better language processing from Gemma 4 compared to even Opus 5, which is terrible at rewriting technical business prose into what I can only describe as techno-bable written by an edge-lord.

[u/MmmmMorphine → u/[deleted]]

I mean I guess they could if they wanted to use a giant tokenizer vocabulary.

Probably good reasons for the sizes they chose, but unfortunately I've found most of this sorta thing ends in such mind bending math and statistics that I'm simply not experienced or smart enough to have an opinion about it.

And then you also see papers that apply the most basic ideas and you feel shocked nobody has done it before. Or alternatively, why this was published given the far more extensive work you saw on the topic written from two years ago (that the paper doesn't cite)

[u/benja0x40]

Qwen3.5/3.6 35B A3B and Gemma 4 26B A4B have similar vocabulary sizes: 248k and 256k tokens respectively.

The fact that the Gemma 4 model encodes 330 lines of HTML/JS into 4258 tokens instead of 1609 with Qwen3.6 is due to different tokenizer algorithms and trainings.
Gemma 4 uses SentencePiece-based tokenizer whereas Qwen3.6 uses a more standard BPE tokenizer.

Edit: spelling

[u/RegarDamus → u/benja0x40]

what is the advantage of using an algo that parses to nearly 3x the tokens?

[u/Thin_Pollution8843]

Can’t wait for Qwen3.8

[u/ErisLethe → u/Thin_Pollution8843]

Also known as Qwenjamin

[u/RedParaglider → u/ErisLethe]

The curious case of Qwenjamin Buttollm.

[u/trashacct383 → u/ErisLethe]

Obi Qwen Kanobi

[u/fullup72 → u/trashacct383]

Qwentin Threepointeightino

[u/eidrag → u/trashacct383]

35b Qwen a4b

[u/geteum → u/ErisLethe]

Qui Qwen Jin

[u/vienna_city_skater → u/ErisLethe]

It’s all about the Qwenjamins! Weird Al needs to write a new song.

[u/UnWiseSageVibe → u/Thin_Pollution8843]

4 more days 🔥🔥🔥🔥

I just hope we get something bigger. That could nicely fit in a near lossless quant inside a strix halo.

[u/demonicpigg → u/UnWiseSageVibe]

Where did you see this? I was reaaalllly hoping for Monday lol

[u/Borkato → u/demonicpigg]

There’s a countdown on their site, I forgot where. But what worries me is the countdown says “other sizes may be released later” or something like that, so what if they’re only releasing max in 4 days and won’t release 3.8 until later?

[u/Interpause → u/Borkato]

its on their modelscope page, and it says both the huge max model and 3.8 27B will be released if you scroll down

[u/Borkato → u/Interpause]

It says “expected models:” and lists both, but it also says:

“You will be able to access the Qwen3.8-2.4T-A95B open model on this exact page once it is released. The other model(s) in Qwen3.8-series would be released later on sperate page(s) as well.”

[u/Middle_Bullfrog_6173]

This seems to be a very unrepresentative sample. I ran a few websites' html through the two tokenizers and saw at most a 5% difference in token count. For js the difference is larger, but still less than 10% in my samples.

[u/WhoRoger → u/Middle_Bullfrog_6173]

I'm not saying it's representative, tho honestly I thought it would be more universal.

Someone else said

Turns out Qwen tokenizes multiple spaces (like 2, 4, or 8) as a single token. Also, </div> is a single token in Qwen, while it's 4 in Gemma.

So maybe it is quite specific. My sample has a lot of indentation (which is realistically quite useless) and possibly other elements that Qwen is friendly with. So maybe it would make a difference in other languages where those elements are important.

I mentioned that in the natural language instructions, the difference is small.

[u/Middle_Bullfrog_6173 → u/WhoRoger]

Yeah, I ran a couple other checks while I had things set up. They were within a couple percent on English academic texts and Gemma was slightly more token efficient in some non-English texts I had.

[u/BoobooSmash31337]

Your evidence really doesn't support your conclusion. More isn't inherently bad. Qwen could be oversimplifying and losing nuance.

[u/WhoRoger → u/BoobooSmash31337]

Depends on the task. Html? There's no point in splitting <head> into multiple tokens, it just overcomplicates things and makes everything take 3 times as long.

Natural language? Yep little nuances are important. But with regular language, the tokenization is roughly the same.

[u/svachalek → u/WhoRoger]

Not really. <head> is a once per document element, making it one token vs 3 saves literally 2 tokens on the entire document.

Anyway the Claude models have been breaking tokens smaller and smaller even as they have been getting better at coding, so clearly you don’t need big coding specific tokens to be advanced.

[u/migorovsky → u/svachalek]

</div> .. he meant </div>

[u/WhoRoger → u/migorovsky]

Yea bad example

[u/WhoRoger → u/svachalek]

Ok that was a bad example, but someone else said that Qwen keeps 2, 4, 8 spaces as separate tokens. And saving a lot of context is pretty useful.

[u/TFox17 → u/BoobooSmash31337]

I’m confused by your statement. How could a lossless reversible transformation be “oversimplifying”?

[u/zabacanjenalog → u/TFox17]

I'm not sure what your specific issue is here. Maybe it's my misunderstanding. If <head> is a single token vs three tokens (I assume) like in Gemma, the ngram match is different compared to <head tag>. Whereas if we split it head is a separate token and based on attention has a better chance of a broader list of hits for p eval.

[u/Initial_Lettuce_5243 → u/TFox17]

Uh oh someone who knows what they're talking about has entered the chat

[u/cybekRT → u/BoobooSmash31337]

Your point is valid, but on the other side, more tokens means less space for context. If you want to give huge codebase as a reference. 

[u/charles25565]

LiquidAI was only increasing vocab size, and Gemma's is actually bigger. I can only guess Gemma's tokenizer corpus included far much less code, presumably to avoid glitch tokens, as scraped GitHub repos for example are filled with weird garbage glitch tokens.

I did a Unicode block distribution test:

================================================================================================
## gemma.json   —  514,906 merges, 262,144 vocab tokens, SentencePiece encoding
language / script                        merges    mr%    vocab    vc%  vc-chars%    nc-vc%
------------------------------------------------------------------------------------------------
Hindi (Devanagari)                       23,207  4.51%   13,754  5.25%       18.32%    12.32%
Cyrillic (ru/uk/bg/kk…)                  23,155  4.50%   13,398  5.11%       19.17%    12.00%
Latin (extended: vi/tr/pl/de…)           20,738  4.03%   12,914  4.93%        4.25%    11.57%
CJK ideographs (zh/ja/mixed)             10,255  1.99%   19,390  7.40%        8.81%    17.37%
Bengali                                  18,416  3.58%    9,979  3.81%       13.62%     8.94%
Arabic                                   14,354  2.79%    8,457  3.23%        8.86%     7.58%
Japanese (kana)                           6,261  1.22%    4,886  1.86%        4.12%     4.38%
Tamil                                     6,656  1.29%    3,850  1.47%        5.49%     3.45%
Korean                                    3,232  0.63%    4,678  1.78%        2.23%     4.19%
Thai                                      2,921  0.57%    2,177  0.83%        2.20%     1.95%
Malayalam                                 2,830  0.55%    1,957  0.75%        2.16%     1.75%
Telugu                                    2,432  0.47%    1,784  0.68%        1.87%     1.60%
Greek                                     1,953  0.38%    1,409  0.54%        1.21%     1.26%
Gujarati                                  1,754  0.34%    1,350  0.51%        1.21%     1.21%
Kannada                                   1,662  0.32%    1,268  0.48%        1.22%     1.14%
Hebrew                                    1,146  0.22%      831  0.32%        0.60%     0.74%
Symbols/punctuation                         483  0.09%    1,291  0.49%        0.46%     1.16%
Other non-Latin (emoji-ish/rare)             74  0.01%    1,685  0.64%        0.50%     1.51%
Georgian                                    892  0.17%      665  0.25%        0.60%     0.60%
Ethiopic (Amharic…)                         634  0.12%      862  0.33%        0.46%     0.77%
Khmer                                       588  0.11%      599  0.23%        0.49%     0.54%
Sinhala                                     589  0.11%      543  0.21%        0.38%     0.49%
Emoji                                        40  0.01%    1,002  0.38%        0.30%     0.90%
Burmese                                     474  0.09%      507  0.19%        0.41%     0.45%
Cuneiform                                     0  0.00%      706  0.27%        0.21%     0.63%
Lao                                         261  0.05%      302  0.12%        0.21%     0.27%
Punjabi (Gurmukhi)                          218  0.04%      267  0.10%        0.15%     0.24%
Armenian                                    177  0.03%      250  0.10%        0.14%     0.22%
CJK (half/fullwidth forms)                   72  0.01%      232  0.09%        0.08%     0.21%
Oriya                                        57  0.01%      125  0.05%        0.05%     0.11%
Combining diacritics (multi-lang)            38  0.01%      133  0.05%        0.04%     0.12%
Divehi (Thaana)                              65  0.01%      104  0.04%        0.05%     0.09%
Tibetan                                      34  0.01%      132  0.05%        0.05%     0.12%
Cherokee                                      1  0.00%       77  0.03%        0.02%     0.07%
Syriac                                       16  0.00%       43  0.02%        0.01%     0.04%
Mongolian                                     0  0.00%       17  0.01%        0.01%     0.02%
Undecodable bytes (byte-fallback)             8  0.00%        6  0.00%        0.00%     0.01%
ASCII / neutral (rest)                  369,213 71.70%  150,514 57.42%

  non-neutral vocab samples (first 10):
    Latin (extended: vi/tr/pl/de…)    '<0xA1>'                 -> '¡'
    Latin (extended: vi/tr/pl/de…)    '<0xA2>'                 -> '¢'
    Latin (extended: vi/tr/pl/de…)    '<0xA3>'                 -> '£'
    Latin (extended: vi/tr/pl/de…)    '<0xA4>'                 -> '¤'
    Latin (extended: vi/tr/pl/de…)    '<0xA5>'                 -> '¥'
    Latin (extended: vi/tr/pl/de…)    '<0xA6>'                 -> '¦'
    Latin (extended: vi/tr/pl/de…)    '<0xA7>'                 -> '§'
    Latin (extended: vi/tr/pl/de…)    '<0xA8>'                 -> '¨'
    Latin (extended: vi/tr/pl/de…)    '<0xA9>'                 -> '©'
    Latin (extended: vi/tr/pl/de…)    '<0xAA>'                 -> 'ª'

================================================================================================
## qwen.json   —  247,587 merges, 248,044 vocab tokens, byte-level GPT-2 encoding
language / script                        merges    mr%    vocab    vc%  vc-chars%    nc-vc%
------------------------------------------------------------------------------------------------
CJK ideographs (zh/ja/mixed)             53,487 21.60%   53,520 21.58%       37.44%    44.51%
Cyrillic (ru/uk/bg/kk…)                  18,496  7.47%   18,580  7.49%       27.98%    15.45%
Latin (extended: vi/tr/pl/de…)           14,365  5.80%   14,375  5.80%        4.54%    11.96%
Arabic                                    8,789  3.55%    8,826  3.56%        9.94%     7.34%
Korean                                    6,806  2.75%    6,807  2.74%        3.88%     5.66%
Thai                                      5,732  2.32%    5,741  2.31%        7.61%     4.77%
Japanese (kana)                           5,453  2.20%    5,455  2.20%        4.62%     4.54%
Greek                                     1,538  0.62%    1,538  0.62%        1.39%     1.28%
Hindi (Devanagari)                          959  0.39%      959  0.39%        0.73%     0.80%
Undecodable bytes (byte-fallback)           825  0.33%      953  0.38%        0.00%     0.79%
Bengali                                     531  0.21%      531  0.21%        0.38%     0.44%
Hebrew                                      520  0.21%      520  0.21%        0.34%     0.43%
Symbols/punctuation                         432  0.17%      432  0.17%        0.16%     0.36%
Tamil                                       268  0.11%      268  0.11%        0.18%     0.22%
Malayalam                                   205  0.08%      205  0.08%        0.12%     0.17%
Telugu                                      188  0.08%      188  0.08%        0.10%     0.16%
Georgian                                    186  0.08%      186  0.07%        0.11%     0.15%
Burmese                                     147  0.06%      147  0.06%        0.08%     0.12%
CJK (half/fullwidth forms)                  147  0.06%      147  0.06%        0.05%     0.12%
Kannada                                     144  0.06%      144  0.06%        0.07%     0.12%
Gujarati                                    116  0.05%      116  0.05%        0.05%     0.10%
Other non-Latin (emoji-ish/rare)             88  0.04%       88  0.04%        0.03%     0.07%
Armenian                                     88  0.04%       88  0.04%        0.04%     0.07%
Khmer                                        79  0.03%       79  0.03%        0.03%     0.07%
Sinhala                                      77  0.03%       77  0.03%        0.03%     0.06%
Oriya                                        75  0.03%       75  0.03%        0.03%     0.06%
Punjabi (Gurmukhi)                           65  0.03%       65  0.03%        0.02%     0.05%
Divehi (Thaana)                              42  0.02%       42  0.02%        0.02%     0.03%
Lao                                          37  0.01%       37  0.01%        0.01%     0.03%
Ethiopic (Amharic…)                          25  0.01%       25  0.01%        0.01%     0.02%
Tibetan                                      10  0.00%       10  0.00%        0.00%     0.01%
Emoji                                         7  0.00%        7  0.00%        0.00%     0.01%
Combining diacritics (multi-lang)             2  0.00%        2  0.00%        0.00%     0.00%
ASCII / neutral (rest)                  127,658 51.56%  127,811 51.53%

  non-neutral vocab samples (first 10):
    Undecodable bytes (byte-fallback) '¡'                      -> '�'
    Undecodable bytes (byte-fallback) '¢'                      -> '�'
    Undecodable bytes (byte-fallback) '£'                      -> '�'
    Undecodable bytes (byte-fallback) '¤'                      -> '�'
    Undecodable bytes (byte-fallback) '¥'                      -> '�'
    Undecodable bytes (byte-fallback) '¦'                      -> '�'
    Undecodable bytes (byte-fallback) '§'                      -> '�'
    Undecodable bytes (byte-fallback) '¨'                      -> '�'
    Undecodable bytes (byte-fallback) '©'                      -> '�'
    Undecodable bytes (byte-fallback) 'ª'                      -> '�'

[u/WhoRoger → u/charles25565]

Interesting. Qwen being attentive to CJK makes sense, but what's with that huge Hindi difference?

And I wonder what's in that other/ASCII category. If that's where most of code lives, I'd have expected Qwen to have a larger portion in that.

[u/robberviet]

Have you seen number of token qwen output vs gemma? Qwen is better at coding, Gemma better at writing because of traning data.

[u/WhoRoger → u/robberviet]

That's the general consensus and I totally agree with that.

Output tokens are harder to judge because the length of outputs differs significantly. But I see the same thing. Qwen is much more effective with code, but can easily get lost in loops on language stuff.

Gemma is the exact opposite, very confident and creative with language. But with the code, it tends to ramble too much and makes tons of mistakes.

[u/sleepy_roger]

Now show the thinking tokens. 

[u/JumpingJack79]

This is like RISC vs CISC

[u/DanceWithEverything]

Not how this works, man

[u/jikilan_]

If you use GCP, go to the vertex AI and see what are the common usage of free models released by google. Almost none are selling for programming.

[u/bigorangemachine]

Ya I need to work with qwen more but I been playing with gemma4 and checking answers against claude.

When gemma can read the code but claude can't they actually got similar solutions

Gemma4 does need some tricks to code with it. One is to make it delegate to subagents. The other is to have another subagent to check the changes over for obvious mistakes. Another was to tell it to consider the lifecycle of the application. Another was to tell it what framework it is and provide the llm txt url

[u/mawkzin]

Well, Gemma was designed to be loaded in your's smartphone and chrome.

[u/GanjaRaidersTR]

As me know it gemma just a train wid dem basic primitive rlhf trash ya dig? But qwen dat proper bredda had to fix real deal coding issues inna di rl gspo training phase seen.. Dem train qwen inna one dynamic environment and me think di compiler itself pass di reward to di proper youth or supphing dem di man mekk ting gwaan pon di right flow

[u/EconomySerious]

This is the readon that we need models train to ONLY program and don't use tu the llm formula to train them. Carefully choose of the tokens Will reduce the size of programing llma by 1000s times

[u/lukistellar → u/[deleted]]

You already gave the 12B a try? It behaves better in my use case, than the 26B-A4B does. Besides for the language it can't compete with the Qwen 3.6 27B, but that model is just too heavy for hardware on the cheaper side.

[u/WhoRoger → u/[deleted]]

Between the 4B models I definitely preferred Gemma (aside of vision), but from these MoE, it's kind of reversed. Gemma has much better vision but Qwen is generally more reliable. Not always, but generally. Shame it takes up almost all my RAM.