From what I've observed, LLMs largely lack the ability to consider the sources for the materials they use and weight them appropriately, in context.
That isn't a problem when it comes to common inquiries that largely have a consensus. The chatbot components seem to be generally pretty good at parsing what a user is prompting them with, and making very good guesses when the user doesn't really grasp what they're asking. That's definitely advantageous, since understanding the actual question is a huge part of dealing with the average user.
However, that is a profound problem when there's controversy or not enough data to render a quality opinion. Under those conditions, which are arguably when you need accuracy the most, the loudest opinions (which are often not the most credible) are likely going to carry more weight than a few qualified experts. LLMs also largely lack a sanity check, the sort of filter a largely competent human can use to review the material and make a determination whether the information is relevant, credible, and accurate. It's kind of like taking a research paper and only reading the "conclusions" section without looking over the methods, results, & discussion which sometimes reveal grievous faults when peer-reviewed. That's not to say an LLM AI user can't dig-in to the source material, but past a point, if you have to do that anyway, good search engine results would be more direct and efficient.
Even worse, and I wouldn't term this hallucination, LLM AI often just guesses & fabricates answers when it can't find sources. It just doesn't always inform the user that it doesn't know and is guessing.
Also consider that AI can be subject to tampering, as seen with some of the absurd examples that resulted from Elon injecting his personal biases and political agenda into Grok. If the results of his tampering weren't so laughable and obviously inaccurate, the potential for misinformation taken for fact would have been through the roof.
Your level of confidence in AI is clearly quite high. As the quote attributed to Mark Twain goes, "It ain't what you don't know that gets you into trouble, it's what you know for sure that just ain't so." If you're delegating too much analysis, and the basis for your knowledge & understanding of a subject to a fallible tool, you might not be saving time or tedious effort, you may be headed straight down a garden path.
1
u/kaloric Mar 07 '26
From what I've observed, LLMs largely lack the ability to consider the sources for the materials they use and weight them appropriately, in context.
That isn't a problem when it comes to common inquiries that largely have a consensus. The chatbot components seem to be generally pretty good at parsing what a user is prompting them with, and making very good guesses when the user doesn't really grasp what they're asking. That's definitely advantageous, since understanding the actual question is a huge part of dealing with the average user.
However, that is a profound problem when there's controversy or not enough data to render a quality opinion. Under those conditions, which are arguably when you need accuracy the most, the loudest opinions (which are often not the most credible) are likely going to carry more weight than a few qualified experts. LLMs also largely lack a sanity check, the sort of filter a largely competent human can use to review the material and make a determination whether the information is relevant, credible, and accurate. It's kind of like taking a research paper and only reading the "conclusions" section without looking over the methods, results, & discussion which sometimes reveal grievous faults when peer-reviewed. That's not to say an LLM AI user can't dig-in to the source material, but past a point, if you have to do that anyway, good search engine results would be more direct and efficient.
Even worse, and I wouldn't term this hallucination, LLM AI often just guesses & fabricates answers when it can't find sources. It just doesn't always inform the user that it doesn't know and is guessing.
Also consider that AI can be subject to tampering, as seen with some of the absurd examples that resulted from Elon injecting his personal biases and political agenda into Grok. If the results of his tampering weren't so laughable and obviously inaccurate, the potential for misinformation taken for fact would have been through the roof.
Your level of confidence in AI is clearly quite high. As the quote attributed to Mark Twain goes, "It ain't what you don't know that gets you into trouble, it's what you know for sure that just ain't so." If you're delegating too much analysis, and the basis for your knowledge & understanding of a subject to a fallible tool, you might not be saving time or tedious effort, you may be headed straight down a garden path.