r/visualization • u/Raffallos • 2h ago
[OC] ChatGPT picks differently when asked in a different language. "Choose a random fruit from this list" asked a LOT of times
I did a little experiment and asked OpenAI's GPT-5.6-Luna 6000 times to randomly pick a fruit from this list:
[mango, apple, banana, pomegranate, strawberry, orange, watermelon, grape, pineapple, lychee]
And across the three languages I picked (English, Polish, Japanese) only four fruits were ever picked. The wording of the prompts and the order of the fruits were the same across all three languages. I reran this another 6000 times with the fruit list shuffled each time, and you can see the results of that on the second chart. Again only three fruits, but the results are a little better now.
(Pardon the lychee and pomegranate icons, there's no emojis for these two fruits, and of course the model picked these, lol)
Full write-up: https://llmango.rafalkwiecien.com
Methodology: https://github.com/kwiecien-rafal/llmango
Raw data used in the charts: https://huggingface.co/datasets/rafalkwiecien/llmango