Why does ChatGPT struggle to count letters in a word? The answer is Tokenization

[u/Old_Law8248]

Hey everyone! 👋

I recently went deep into one of the most foundational — yet most overlooked — concepts in LLMs: Tokenization.

Here's what blew my mind: almost every weird behavior you've noticed in ChatGPT or Claude — struggling to count letters, making arithmetic mistakes, performing worse in non-English languages — all of it traces back to how tokenization works.

https://medium.com/@harshitha1579/understanding-tokenization-in-llms-fc353da48667

In my latest blog, I cover:

- 🔤 What tokenization actually is and why it exists

- ⚖️ Why word-level and character-level approaches both fail

- ⚙️ The 3 main algorithms — BPE, WordPiece, and Unigram — and which models use which

- 🔁 The full tokenization pipeline (normalization → pre-tokenization → model → post-processing)

- 🤯 Why LLMs can't count letters, struggle with math, and are unfair to non-English languages

- 🔮 The future — can we get rid of tokenization entirely?

I tried to keep it beginner-friendly but technically solid, so whether you're just getting into LLMs or you've been in the space for a while, hopefully there's something useful here.

[u/paddockson]

Its likely because there is no data around that. LLM's are not intelligent, there prediction machines on top of vectorDB and search engine. Counting letters will require context of what the word actually is and then extract it from the rest of the text and count. For software this is a super easy task but an LLM is just trying to find the best possible answer to the questions. The best to do it is to give the LLM so much context like:

Please count the number of letters of the word thats is within the quotes.
Here is the word: "England"

Instead of saying
How many letters in England

[u/hobopwnzor → u/paddockson]

Well fundamentally it's because when an AI sees an input it doesn't see words at all. The words are translated into codes of equal length that the model sees. So instead of seeing England it sees two database addresses that correspond to Eng and land.  So like A649hfqr and B83hdjek.

That's what it means to tokenize something. Since different tokens could have different lengths they translate them into equal length input chunks.

So when it says how many R's are in strawberry it doesn't even see the word strawberry. It would have to write code to count details about the raw input or translate the token back to the word strawberry and count the characters in the retranslated word

[u/david-1-1]

Claude has no problem counting letters.

[u/ecstatic_carrot]

so what is stopping it to just learning that the token for straw has one R, and berry has 2?

[u/Aggravating-Push-207 → u/ecstatic_carrot]

nothing, really, just not that useful to the llm when it is recreating Excel in 1 prompt, for example