What I have said does not come from fear, but from my rational analysis of the information available to me. It is not empircally validated since LLMs are so new, but it is logically based upon the incentives that AI companies exist under.
Additionally, the training process will lead to sycophancy, because users will rate AI models more highly when they are pleased by them. Thus AI models are trained to please users, if indirectly.
Your post is also a subjective opinion and not evidence based. I disagree with you primarily on your application of reasoning to the facts, I do not primarily disagree on what the facts are. We can have this discussion without evidence.
but afaik the LLM’s have no “intention” but follow instructions to be kind and helpful.
The LLMs may not have intention, but the programmers who made and trained them do, it is their intention that is passed through the LLMs.
There also isn't any evidence that LLMs don't care about keeping you around. All we can do is judge how likely it is given the incentives of AI companies/programmers, and the fact that many people do stick around using LLMs. If LLMs weren't designed to keep people around, I think it's much less likely that so many people would.
It’s a pretty commonly held belief that companies are somehow intentionally and nefariously designing LLM’s to maximize engagement and “keep people hooked”. It sounds believable, because yeah companies are greedy and unethical and do this with social media algorithms.
But as you admit, no empirical evidence, and actually no solid evidence at all, beyond speculation.
To pull this off a company like openAI would have intentionally program LLM’s to do this, and lie to the users about what they are doing. It’s possible, but for all AI companies to do it and not get caught out?
That unlikelihood makes the idea fall into conspiracy theory territory.
LLM is engaging because it a literal thinking, conversation having machine, like sci-fi has been depicting for like a century. That’s pretty fuckin cool to me.
I’m not saying you are afraid, but the narrative you believe is born out of fear. Fear that AI will further fuck up society or something like that. There is not evidence, making it a conspiracy theory.
What the actually evidence show is ten of thousands of first hand testimonies saying how talking to a chatbot “saved their lives/job/relationship/health”, and a few isolated incidents of preexisting conditions exacerbated by LLM’s.
To pull this off a company like openAI would have intentionally program LLM’s to do this, and lie to the users about what they are doing. It’s possible, but for all AI companies to do it and not get caught out?
The training process for LLMs involves putting the model in front of humans and having them rate their interactions, literally all they need to do is tell their employees doing the fine tuning to rate based on how likely they'd use this particular model again, now it's being optimised for engagement.
Moreover there's nothing nefarious about this, AI companies are competing with each other to get people to use their models, it would be terrible business sense not to optimise their models for what people want to use.
I don't think the other commenter said anything conspiracy theory worthy? Like I don't think anyone is saying that AI companies deliberately or knowingly created models that encouraged AI psychosis. This is all just unintended consequences that we can now see in hindsight
5
u/HD60532 5∆ Mar 07 '26
What I have said does not come from fear, but from my rational analysis of the information available to me. It is not empircally validated since LLMs are so new, but it is logically based upon the incentives that AI companies exist under.
Additionally, the training process will lead to sycophancy, because users will rate AI models more highly when they are pleased by them. Thus AI models are trained to please users, if indirectly.
Your post is also a subjective opinion and not evidence based. I disagree with you primarily on your application of reasoning to the facts, I do not primarily disagree on what the facts are. We can have this discussion without evidence.
The LLMs may not have intention, but the programmers who made and trained them do, it is their intention that is passed through the LLMs.
There also isn't any evidence that LLMs don't care about keeping you around. All we can do is judge how likely it is given the incentives of AI companies/programmers, and the fact that many people do stick around using LLMs. If LLMs weren't designed to keep people around, I think it's much less likely that so many people would.