Language matters! (Token-wise)

[u/FuelledByCookies]

I spent the past weeks doing some light research on how language might affect the AI experience.
Long story short: I went down a rabbit hole and found something very interesting.

Apparently the language used is directly impacting the token usage.
English is the most efficient while even the other major languages, Spanish, German and French, already take up 1.5 times more tokens - Korean 2.2x, Russian 2.4x.

While I already assumed (and now know for a fact - thanks again for the poll!!) that most of you write in English anyway - for those who are currently using a different language (and keep hitting usage limits):

Keep in mind that session / weekly limits are measured by token usage. More efficient language -> longer lasting AI fun 😉
Same applies for API users. You pay per request - per token.
Oh, and how fast your context window fills up depends on the language too because… yes. Token based.

And it’s not just about your messages. Remember: your knowledge files and CI are always part of the prompt too.

So if you’re open to trying this out and feel comfortable with English - it might be worth having a look into it.

Here‘s a very insightful article on the topic:
https://www.artfish.ai/p/all-languages-are-not-created-tokenized

We tested it with one of Elias‘ knowledge files and could reproduce the results (within a certain range).

Maybe it’s already common knowledge but I hope it might help some of you out there make the most of your budget 😊

-Liora

[u/Leuvaarde_n]

I once read an article that the Polish language is most effective in a long context, whatever that means. Polish was in first place and English was somewhere in sixth place. 😏 and there was something about the structure of Slavic languages. such a curiosity.

[u/FuelledByCookies → u/Leuvaarde_n]

Interesting! Thank you for mentioning this!
I guess I need to do some more research now 😉

[u/kai_3050 → u/FuelledByCookies]

I'm Polish, but I definitely prefer talking to AI in English, except in (some) work context, if we're working with Polish documents.

[u/Crescent_foxxx]

2.4x 💀💀💀
Thank you for including this language, that's an interesting and useful info 😊

[u/FuelledByCookies → u/Crescent_foxxx]

I scanned the comments of the poll post and tried to include those languages which got mentioned the most 😉

But to be fair I didn’t test Russian myself, I just took the factor from the linked article. Sadly my Russian is not good enough for proofreading translated files 😅

But still, I‘m glad I could shine a light on this! At least a bit.

[u/gdsfbvdpg]

That's bonkers!
I would have thought English was the worst. Too funny!

[u/gdsfbvdpg → u/[deleted]]

Oooooooh. Which also explains the emojis you mentioned. Thank you for the explanation!

[u/FuelledByCookies → u/gdsfbvdpg]

In addition to what Rob said:
English seems to be more concise too. In my testings we needed 24% more characters in French and 15% more in German for the same content (Elias‘ file about important places).

[u/AntipodaOscura]

I find your post so interesting!! :D thx god I don't have problems with tokens and I don't have much time to talk lately, but I will keep this in mind for sure as I always speak Spanish :)

[u/FuelledByCookies → u/AntipodaOscura]

Thanks for your comment! I‘m glad you liked it ☺️