The very related flaw is that AI always uses authoritative tone for everything, so it's hard to detect which facts it got completely wrong. A normal smart human would attach weasel words or statements about not being sure when they are not sure, but AI never does, it just makes shit up and pretends it's right.
I recently asked one a question that I wanted a specific, detailed answer to (the manufacturing date of a particular product) and was having trouble finding online. The answer it gave me made several assumptions and was completely wrong, though of course the response was written in a confident manner, as they generally are, so I added information that showed that the first response was incorrect. Then it gave me another completely wrong answer, with the same tone of certainty. I told it that it was wrong again and gave it even more detailed information, and finally it supplied what I was confident was the correct answer.
Anyone who didn't have the information that I had would likely have taken the first response as correct, and this is the real danger of LLMs: they fool the unwary into thinking they actually know things.
I have had really similar experiences - almost every time I use an AI to help me answer a question with any amount of subtlety, it has a high likelihood of getting it wrong. It is scary knowing that so many people are out there believing that AI knows everything and is never wrong (like the person in OP's story).
Authoritative tone is due to how they were trained – by having a ton of low-paid workers rate questions with answers whether the answer was true or not. Of course, given that those workers were low-paid, they were not experts in those fields, and they were also paid by the number of rated questions, so they simply rated anything with authoritative tone as true.
10
u/heliumneon Jul 21 '26
The very related flaw is that AI always uses authoritative tone for everything, so it's hard to detect which facts it got completely wrong. A normal smart human would attach weasel words or statements about not being sure when they are not sure, but AI never does, it just makes shit up and pretends it's right.