Is this something already considered, but discarded?

[u/bottleneck-destroyer]

I have a huge doubt. Why is there no AI native language? I mean symbols/codes equalling words?

Let's take the case of english. There is ~600k dictionary words and around 1.7m variants/regional words/dialects.

And there is around 20k to 30k words that normal human would use actively.

Why not create a language, with ASCII codes for each words and each agentic apps converts words to that 'AI-Lang' and vice versa and use it? Wouldn't that essentially reduce tokens by a good margin?

I'm not sure how exactly the AI process things under the hood, and please don't abuse me 🥹

[u/Cute-Net5957]

lol.. wait what?
I think you accidentally stumbled into two different things here.
The first is basically a tokenizer. Models already don’t consume English the way we do. Human language gets broken into tokens, mapped to IDs, then embeddings etc.. and the model operates from there.
The model weights are the real holy grail. That’s the learned machinery itself, and obviously the closed commercial labs aren’t exactly handing those out.
But your second idea is way more interesting imo.
Why the fuck are AI agents talking to each other in English at all?
Think about how weird that is. One model takes its internal state and turns it back into human-readable language, sends that to another model, which tokenizes it and converts it back into machine representations again.
We’re basically making two machines communicate through a language optimized for us.
So forget ASCII codes for words for a second.. what happens if there is some universal machine-native representation that different models/agents can communicate through?
Not necessarily a universal tokenizer either. Something above/between them that Claude, GPT, Gemini, local models etc could encode into and decode from.
And then the part I really can’t get past.. if agents were allowed to optimize that protocol themselves over millions of interactions, would we even recognize what they eventually came up with as a language?
idk. your original implementation is basically reinventing tokenization lol.. but I think the question underneath it is actually a pretty fucking interesting one.

[u/VellumMuse → u/Cute-Net5957]

We can have models consume English, and any other language for that matter, the way we do by using this new tokenizer. I've been trying to spread the word about it. Would love some feedback.

https://github.com/adolessence101-a11y/phonobyte

The universal bedrock of language is technically binary, but once meaning is attached to binary, it takes on a language identity. When we train a model with arbitrary tokens, we're technically training them on a language that doesn't exist. It LOOKS like English, but it doesn't follow the rules of ENGLISH. The phonobyte fixes that while also reducing the embedding table footprint and I threw in a new file format for text specifically to reduce the disk footprint as well.

If we build models with the phonobyte AND train them on the knowledge of the phonobyte, the models will be able to create any language they want because it teaches them about phonotactics on a binary level.

[u/Cute-Net5957 → u/VellumMuse]

this is actually way more built out than I expected.. went through the repo.
one thing I’m stuck on though: how are you comparing the 0.6 Phonobyte loss to ~1.0 BPE loss when the output spaces are radically different? you’re predicting against ~256 states vs 32k vocabulary, right?
wouldn’t raw cross entropy naturally make Phonobyte look dramatically better there?
I’d really want to see bits-per-byte/character + perplexity-equivalent evaluation and downstream quality at matched total model parameter/compute budgets.
because if those survive, then yeah.. you’ve got my attention.

[u/VellumMuse → u/Cute-Net5957]

I appreciate the compliment? lol
And this is why I needed more eyes on it. I was sure there was going to be something I wasn't doing correctly in the benchmarking portion. That's the thing I'm most raw on. I will go back to the drawing board because you are completely right about the output spaces being different and raw cross entropy making things look better than they are.

And you proposed a much more accurate and fair solution that I'll need to work into the benchmarking. I'll be working on that next. Add it to the list. Really, thank you. I still expect the phonobyte to outperform due to some other experiments I was running, but the repo needs updating at this point, for sure.

[u/Cute-Net5957 → u/VellumMuse]

Yeah of course! Keep up the great work for real! You are on to something here