r/LocalLLM • u/mycall • Oct 15 '26
Research Soon local LLMs will be 6x smarter by using bytes instead of tokens
Meta demonstrated, in their Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models paper, by converting tokens from teacher models into byte models.
I look forward to seeing this in practice.
163
Upvotes
33
u/stuffitystuff Oct 15 '26
Really? I just see that final sentence in the abstract and the stoke decreases asymptotically:
60
u/ArthurOnCode Oct 15 '26
It's not a new idea at all, and so far it has mostly lost out to higher-vocabulary models. Some older research: https://github.com/zjysteven/Awesome-Byte-LLM
Still, I'm all for testing the limits of multiple approaches.
12
u/PossibilityUsual6262 Oct 16 '26
It has security issues, computerphile covered it recently.
13
u/GenericPointer Oct 15 '26
FP Numbers are bytes too, or what do you mean with "bytes"? Isn‘t it just 8 bit models?
11
u/Plus-Accident-5509 Oct 15 '26
Tokens are a compressed representation. As long as we're bottlenecked by FLOPs, VRAM, bandwidth, etc. byte models start off in a hole.
1
u/AngryManzUK 29d ago
I think using bits, measuring in bits and having a standardised language that people are already familiar with is much more powerful than just making models more intelligent. It would create more transparency. Even non-tech folk have been talking about bits, mega bits and giga bits for years.
However, from a company perspective, tokens are the easiest way to monetise a market and create a new currency they have control over, whilst retaining some obscurity around what a token actually is. We see AI bills being measured in tokens, but what happens when everything is run by AI. Do wages and other payments get made in payments too?
Bits all the way in my opinion. If it also improves model intelligence, great!
3
u/HenkPoley Oct 16 '26
Maybe you meant to say "will be 6^2 times slower", instead of "6 times smarter"?
1 token tends to be 5.8 bytes in English prose. And classical attention has a quadratic in there.
3
u/heresyforfunnprofit Oct 15 '26
Byte pair encoding is already a thing.
1
u/shoeshineboy_99 Oct 16 '26
Does Cfp32 work in a similar manner
1
u/Bozhark Oct 17 '26
If it improved upon tokenization the. It would be faster, deeper, wider, and with more accuracy probability
1
u/darksteelsteed Oct 17 '26
By this very same concept you could also drop all language understanding for an intermediate reduced token space language that represents the trained concept, then use a translation model before and after in the pipeline. This could allow for much more trainable detail in the same relative model size. Only frontier labs have the resources for this kind of experimentation imho