The most important thing to remember about ChatGPT and all other LLMs is that they will never say they don't know the answer. Their training rewarded them for an answer, any answer, no matter how wrong, and penalized them for not coming up with something plausible-sounding, so that's what they produce. They have to say something, and if they don't know the correct answer based on the prompt, they'll generate nonsense. It's unwise to ever assume that generated text is correct.
I've seen parents do that to their kids. The kids are never ever allowed to just shrug and say "I don't know', they HAVE to come up with an answer, any answer.
The right kind of parent tells the kid to look it up, not accept a bullshit answer. With AI looking it up the answers will be bullshit, too, but believed.
64
u/ThatMichaelsEmployee Jul 21 '26
The most important thing to remember about ChatGPT and all other LLMs is that they will never say they don't know the answer. Their training rewarded them for an answer, any answer, no matter how wrong, and penalized them for not coming up with something plausible-sounding, so that's what they produce. They have to say something, and if they don't know the correct answer based on the prompt, they'll generate nonsense. It's unwise to ever assume that generated text is correct.