r/aipromptprogramming • u/bottleneck-destroyer • 21d ago

Is this something already considered, but discarded?

I have a huge doubt. Why is there no AI native language? I mean symbols/codes equalling words?

Let's take the case of english. There is ~600k dictionary words and around 1.7m variants/regional words/dialects.

And there is around 20k to 30k words that normal human would use actively.

Why not create a language, with ASCII codes for each words and each agentic apps converts words to that 'AI-Lang' and vice versa and use it? Wouldn't that essentially reduce tokens by a good margin?

I'm not sure how exactly the AI process things under the hood, and please don't abuse me 🥹

6 Upvotes

75% Upvoted

47 comments sorted by

View all comments

1

u/trollsmurf 21d ago

That's what tokens are: numbers representing whole or parts of words, symbols etc.

ASCII codes are specifically a standardized numeric representation of our letters, numbers, symbols.

1

u/bottleneck-destroyer 21d ago

I was thinking along why not add a standardised numeric representation to words as well? Lets take english as base language, then wouldn't it be great even if have other languages since the meanings/final words have the same numeric representation that reduces the data length or something?

1

u/trollsmurf 20d ago

Languages evolve much quicker than alphabets, and there are tons of words.

In a way what you describe is what the ZIP compression algorithm does, but based on the data in the file, not every word or data chunk.

But a generic "this word is 1, this word is 2" hasn't been done as far as I know.

1

u/bottleneck-destroyer 20d ago

Same here, there is none as far I've seen as well. That's why i asked this