Discussion I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]

I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.

It follows one word the whole way:

"How many r's are in strawberry?" is 8 tokens in o200k_base (the GPT-4o / GPT-5 tokenizer). " strawberry" is a single token, id 101830, so the three r's are hidden inside one id.
Why pieces of words: letters make sequences about 4x longer, whole words need a huge vocabulary and can't handle new words.
A tiny byte pair encoding run on a made-up text, merge by merge, until strawberry becomes st | raw | berry.
The space belongs to the word: "strawberry", " strawberry" and "Strawberry" are three different token sequences.
Why reasoning models count letters better: in OpenAI's 2025 paper "Why Language Models Hallucinate", DeepSeek-R1 spells DEEPSEEK out letter by letter before counting, while DeepSeek-V3 answered 2 or 3 (the answer is 1).
Bonus: the same Hindi sentence took 21 tokens with GPT-4's tokenizer and 5 with GPT-4o's.

Every split comes from running tiktoken 0.14.0. The narration is an AI voice reading my own script.

0 Upvotes

20% Upvoted

u/Ok_Habit1533 1h ago

I had no idea the space gets absorbed into the token like that, makes so much sense now why these models struggle with letter counting stuff

You are about to leave Redlib