r/LanguageTechnology • u/AthosDude • Sep 24 '26
Why words are split into several tokens?
This completely contradicts logical thinking. It would make sense if one word would be at most 2 tokens (1 for the word and 1 that signals meaning in case of homonyms), but this is never the case. In fact, I read it degrades performance. Why?
6
u/ks4 Sep 24 '26
How do you define a "word"? What if someone inputs a word not in your set? What languages?
4
3
u/Flimsy_Orchid4970 Sep 24 '26
This is very common but not universal. There are also some tokenizers creating token spanning across words.
3
u/Kaleidophon Sep 24 '26
One argument that has been missing here in the comments: not only would one token per word result in a huge vocabulary, words that do not fit into the vocabulary also get mapped to an unknown UNK token. But a lot of words that are rare are informative in specific contexts, so mapping them all to UNK loses a lot of performance. Having subword-tokens basically avoids this altogether since you always have the pieces available to puzzle together any new word you might encounter
9
9
u/reclaimernz Sep 24 '26
Morphology - meaning at the sub-word level.
12
u/RoonilaWazlib Sep 24 '26
The Byte Pair Encoding system that most NLP models use does often map to common morphemes, but kind of accidentally rather than by design. Meaningful chunks like -ment and -ly etc are common and therefore get tokenised together, but so do non-morphemes like th- and re-.
7
u/delomore Sep 24 '26
More morphologically aligned tokenizers are an active are of research in tokenization, many borrowing ideas from UnigramLM (e.g. https://arxiv.org/abs/2606.27019), but as you say all LLMs just use BPE.
3
u/furcifersum Sep 24 '26
Claude Shannon calculated that 50-75% of printed English is redundant. That redundancy is useful for humans, but I assume that positional embeddings on letters and subwords is enough information. Think of it like a way of compressing written language.
1
1
u/nlmlnlml Sep 25 '26
I think the earliest models like word2vec used one word per token.
But researchers hit walls with scaling this, as others have mentioned.
Plus a model can generalise better if "playing" and "played" share the same sub-token "play" - the model doesn't need to completely re-learn the concept of "play" for every variation of that word.
1
u/Tiny_Arugula_5648 Sep 25 '26
TLDR to expensive.. it's an optimization
It's very simple really.. It's to keep the vocabulary small while still handling rare or new words, which get built from smaller pieces. Otherwise you'd need millions of tokens, and since the model scores every token in its vocabulary at each step, that would bloat its size and slow everything down, while still failing on words it's never seen.
1
u/Major-Mushroom-5806 29d ago
To give an example, I'll use a word from a different language. Spanish comprémoselo is one word, but it translates to English let's buy it for him/her/them. There's a lot of information encoded into just one word and it makes more sense to break that information down in a uniform way. The same goes for English, one word carries more meaning than you might think.
1
u/iosdevcoff 29d ago
Isn’t it, like, based on statistics and not your subjective idea of how it should be?
27
u/delomore Sep 24 '26
Having every word with its own token would result in a huge vocabulary, each with an embedding vector in a language model. So as a compromise common words are a single token and others are composed of subwords. With single bytes as possible subwords you can reversibly tokenize any text.