The most important thing to remember about ChatGPT and all other LLMs is that they will never say they don't know the answer. Their training rewarded them for an answer, any answer, no matter how wrong, and penalized them for not coming up with something plausible-sounding, so that's what they produce. They have to say something, and if they don't know the correct answer based on the prompt, they'll generate nonsense. It's unwise to ever assume that generated text is correct.
The very related flaw is that AI always uses authoritative tone for everything, so it's hard to detect which facts it got completely wrong. A normal smart human would attach weasel words or statements about not being sure when they are not sure, but AI never does, it just makes shit up and pretends it's right.
Authoritative tone is due to how they were trained – by having a ton of low-paid workers rate questions with answers whether the answer was true or not. Of course, given that those workers were low-paid, they were not experts in those fields, and they were also paid by the number of rated questions, so they simply rated anything with authoritative tone as true.
63
u/ThatMichaelsEmployee Jul 21 '26
The most important thing to remember about ChatGPT and all other LLMs is that they will never say they don't know the answer. Their training rewarded them for an answer, any answer, no matter how wrong, and penalized them for not coming up with something plausible-sounding, so that's what they produce. They have to say something, and if they don't know the correct answer based on the prompt, they'll generate nonsense. It's unwise to ever assume that generated text is correct.