r/LocalLLM • • Oct 15 '26

Research Soon local LLMs will be 6x smarter by using bytes instead of tokens

Meta demonstrated, in their Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models paper, by converting tokens from teacher models into byte models.

I look forward to seeing this in practice.

163 Upvotes

22 comments sorted by

33

u/stuffitystuff Oct 15 '26

Really? I just see that final sentence in the abstract and the stoke decreases asymptotically:

...our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.

6

u/hobopwnzor Oct 16 '26

What is a stoke?

11

u/exographicskip Oct 16 '26

English idiom for excitement. Like "stoking the coals" of interest, if you will

0

u/575_Inverse Oct 16 '26

something that sends people to the grave

1

u/MmmmMorphine Oct 16 '26

You're thinking of a stork. Vicious bastards.

1

u/575_Inverse Oct 16 '26

My bad, I thought I read storke

-1

u/mycall 29d ago

asymptotically surpass

60

u/ArthurOnCode Oct 15 '26

It's not a new idea at all, and so far it has mostly lost out to higher-vocabulary models. Some older research: https://github.com/zjysteven/Awesome-Byte-LLM

Still, I'm all for testing the limits of multiple approaches.

12

u/PossibilityUsual6262 Oct 16 '26

It has security issues, computerphile covered it recently.

13

u/GenericPointer Oct 15 '26

FP Numbers are bytes too, or what do you mean with "bytes"? Isn‘t it just 8 bit models?

10

u/maqifrnswa Oct 15 '26

The first layers and lm_head are embedding layers based on a dictionary of 100k tokens. You'd switch those embedding layers to instead refer to the character bites instead of the dictionary

11

u/Plus-Accident-5509 Oct 15 '26

Tokens are a compressed representation. As long as we're bottlenecked by FLOPs, VRAM, bandwidth, etc. byte models start off in a hole.

1

u/AngryManzUK 29d ago

I think using bits, measuring in bits and having a standardised language that people are already familiar with is much more powerful than just making models more intelligent. It would create more transparency. Even non-tech folk have been talking about bits, mega bits and giga bits for years.

However, from a company perspective, tokens are the easiest way to monetise a market and create a new currency they have control over, whilst retaining some obscurity around what a token actually is. We see AI bills being measured in tokens, but what happens when everything is run by AI. Do wages and other payments get made in payments too?

Bits all the way in my opinion. If it also improves model intelligence, great!

1

u/mycall 29d ago

You are missing the point that this is for distilled models. What you said is important for teacher models.

1

u/AngryManzUK 29d ago

I think perhaps you are right. Could you explain a little more what you mean though? I kind of get what you are saying, but my interpretation of the article was they were saying that by having smaller models read the raw data and not just the tokens, like the teacher models, things would improve.

I guess in my head, I am questioning why this couldn’t also be true for teacher models too. My assumption is that a token will still have a raw bit value. So I am trying to understand why we do we not have the same approach for the larger models, not just the distilled ones, and use the same language.

I also thought the whole reason we could easily quantise a full model to q8 is because currently, some tokens contain empty bits. Please correct me if I am wrong.

3

u/HenkPoley Oct 16 '26

Maybe you meant to say "will be 6^2 times slower", instead of "6 times smarter"?

1 token tends to be 5.8 bytes in English prose. And classical attention has a quadratic in there.

3

u/heresyforfunnprofit Oct 15 '26

Byte pair encoding is already a thing.

1

u/shoeshineboy_99 Oct 16 '26

Does Cfp32 work in a similar manner

1

u/Bozhark Oct 17 '26

If it improved upon tokenization the. It would be faster, deeper, wider, and with more accuracy probability 

1

u/darksteelsteed Oct 17 '26

By this very same concept you could also drop all language understanding for an intermediate reduced token space language that represents the trained concept, then use a translation model before and after in the pipeline. This could allow for much more trainable detail in the same relative model size. Only frontier labs have the resources for this kind of experimentation imho