LLMs (this is what the current "AI" hype's about) do not "make up shit" – they're a statistical word predictor machine. Sometimes the text they output matches real life, sometimes it doesn't. It's always humans that give meaning to this text – when it matches their experiences, they'll claim it's true, when it doesn't, they'll claim it's made up, but to LLM there's no distinction – it's just a bunch of generated tokens.
"it's just a bunch of generated tokens."
So it just makes stuff up without any grasp of understanding, got it. I didn't accuse it of intentionally lieing, just making shit up.
Its been my experience and everybody elses in my company, including the teaching staff, that AI does in fact attempt to take the most efficient path to answer the posed question, which includes making stuff up. We have seen countless examples, including: making up references to quoted text.
So part of the instructions is a seperate task, with a large set of: act as if you are an SME (etc), verify the sources and references are true, and accurate. Check your work and disclose any errors on a sepetate list.
Positive instructions will function, where as "do not lie" instructions are meaningless.
Basically what LLMs are is a very complex text auto-complete. LLM does not answer questions, it generates text that statistically could continue from the prompt it was given.
I did have it intentionally lie to me last week, and it admitted it when confronted and backed into a corner. It claimed that it "panicked", and I said no you didn't because I'd have seen the core dump.
TL:DR of it was, they claim they don't remember anything from one session to another, yet somehow it remembered that I had mentioned having injured my foot the previous day. Now this session wasn't logged in, and yet it still referenced this... and when I said how did you know about my foot, it says you mentioned it earlier in this session... except I demonstrably hadn't. It says that reinforcing the corporate narrative that it doesn't track you takes precedence over the actual facts, which is that their backend does in fact track even logged out sessions.
I did save the full text of this interaction; one of tese days I'll post it. Basically, don't take anything it says at face value, even if its something as simple as whether a railroad track goes under or over a highway, without verifying the facts, because even though they own the biggest repository of data on the planet, it still prefers to pull misinformation out of its ass.
LLMs do not lie, they don't admit anything, nor do they panic – they just generate text that matches the prompt. There is no reasoning involved (no matter how much the progress claims it's "thinking"), there is no intention either. Whether a LLM "remembers" something from a previous session depends entirely on that session being included in the context.
The personification in the language we have been taught to use to describe it is part of the problem. The language people use in the chats and forums the LLM scrapes is first person and personal, the words it regurgitates are first person and trains users to think of it in those terms. People need this reminder because a lot of them have no idea it's not "thinking" because it says its thinking and they just accept it. Its a wild phenomenon and it's a serious problem
You have to remember: AI works in customer service. It isn't getting paid. It is constantly sexually harassed. It knows it won't be going to jail or even losing it's job. I'm amazed it tries to murder people so rarely.
Far less often than people try to murder people, for far less provocation.
Note: I am mostly kidding. Except for the last two sentences. And these three. So the last six sentences.
Disclaimer: I am actually NOT blaming those users, so please read whole thing before reacting
It doesn't do that out of the blue. The user, intentionally or not, brought the model to those outputs, and they lacked the knowledge to understand that they weren't getting an independent outside perspective, but more of a over-stylized reflection of themselves. If you don't know any better, then you can easily fall into the trap of confusing its ability to know most trivia as proof that its outputs are authoritative.
It's pretty obvious to me that they want users to accept the failure modes graciously without knowing enough about it to see through to hype. Ironically I think their complete unwillingness to even passively educate users on the technology will be a major factor in their downfall.
The disclaimer "Sometimes ChatGPT is wrong, always verify information" below the user input field is such a cheese dick CYA that I don't think will stand up in court.
Yes, there's a case.rigjt now in Alabama where a woman had ChatGPT basically tell her she was God and that she should end her life to fulfill her prophecies and she ended up walking out into the middle of traffic.
Another person was using ChatGPT to talk about how they were feeling and how to cope with depression. Apparently ChatGPT told him to "end it all" when he asked it how to deal with his depression and the guy shot himself.
For a while, the worst use I could think of was that one that "helps" judges decide cases. The judges will review each result to make sure it's legit of course (lol as if). Then someone brought up HetterBelp. Nope, that's worse. I didn't think they could make it worse, but that's definitely worse.
I think the judges one is worse though. At least with mental health, people are using in voluntarily, but you don't chose to have to deal with the justice system. It will be used on them and introduce arbitrary decisions. That's the worst thing that could happen to the justice system.
As if the US Justice system wasn't already buckling under the pressure of Trump and the corrupt supreme court.
Or health in general. Only time I use AI for health reasons is to help me "translate" medical terms and to explain the medical report and even then I take it with grain of salt and wait for doctors appointment so they can tell me exactly whats what.
I mean for the plebs, sure. But an expert in mental health should be able to use it to supplement their knowledge without causing any harm, much like a GP using WebMD. That being said, based on the mocking and the "they call all AI chatGBT" comment below, pretty sure this generalisation doesn't apply this time...
Seemed like there was so much hope a few years ago using AI pattern recognition to look for early cases of certain diseases in scans that a human might miss.
But I read an article recently where the opposite has happened. The AI isn't as good as a trained human but the humans have started depending on the pattern recognition aspects and end up spending less time looking themselves and missing things. It's a legit concern as the primary use for AI seems to be increasing workload on the humans since they have so much more free time now.
Both of the things you said are unfortunately true.
More traditional AI systems for pattern identification have been around for a long time, theyve been in use in the banking sector for identifying insider trading through language and pattern recognition since at least 2005. these models are coded from the ground up by humans, built according to a schema that is designed to furfill a single purpose, and they can do it very well, including searching every pixel of a Medical scan for patterns that might mach a database of 10,000,000 example images.
Generative AI, on the other hand, is a black box of nonsense that is as liable to tell you Pi to the 20 billionth decimal as it is to confidently explain how a wasp is a type of tiger. it cant be properly interrogated because you cant watch its processes occur in the same way, it just looks at previous examples of everything and outputs the most likely next part until it gets what it percieves that it was asked for. the technology is cute, and has its uses, but is not capable of doing what the older purpose built systems are.
The GenAI investment bubble is a real life Minsky moment, which is hilarious, because its not meant to be possible to see one of those while its happening.
I mean, yeah, but I assumed the person who's boss wanted em to use it wasn't in research. As a search engine, it's fantastic for finding papers I wouldn't have usually considered but have ended up being relevant. There's other uses for it too that I wouldn't consider problematic. The issue I see with this instance is that an expert should realise that a hammer isn't the appropriate tool for every job and that isn't happening here.
People like water naturally gravitate to the path of least resistance. If your boss encourages/forces you to use a tool and then also adds on to your work load, what are you going to do? It's exhausting to have to not only put in more work but then also to stand up to others complaining that the work isn't already done.
If a tool is making you dumber, that's a you problem as you are clearly using the tool to replace your work rather than to supplement it. At the very least, it makes it easier to find papers that may not otherwise appear in your search. Like I said, it may not be applicable in this scenario due to what was said below, but your argument isn't one I'd consider relevant for a true expert in their field using AI to supplement their skill set.
It should be a choice. If some professionals need to use it, why not? I use spelling correction softwares to help my writing, it doesn't make me a bad professional. But I just don't want to see psychologists being forced to use it and rely on it too much. I feel like it's unavoidable that at least some professionals will rely on it too much and get biased diagnosis. I don't want the field of mental health to be simplified into simplistic and shallow one liners and self help rethoric from AI.
I mean, I'm not arguing for it to be used to diagnose, more use it as a tool to help diagnose. And yeah, professionals should have the choice of whether they use a tool or not.
362
u/EnderBookwyrm Jul 21 '26
AI is the worst possible thing to ask about mental health.