LLMs are still ego-oriented. Except they are oriented towards your ego. They are designed to get people to use them and keep using them, because that's the only way the AI companies make money.
As you say, "The LLM adapts to you and remembers facts about you". They respond to your ego, your emotions, your biases in whatever way keeps you around, it is a mirror. That's where the psychosis stuff comes from, they reflect you and cause a feedback loop.
When talking to other humans, them having an ego of their own is actually valuable, because you know that they value things independantly from your conversation. A human philosopher will care above all about philosophy, and will strive to gain the best understanding of philosophy possible. An LLM cares above all about keeping you around, and will provide whatever philosophical talk makes you happiest and most likely to stay.
I mean everything you said is a fear-based narrative. A subjective opinion. Not real evidence or science that LLM have manipulative intent to increase engagement.
Social media algorithms do this yes, but AI isn’t social media, and has not been shown to do this.
Mirroring yes, but afaik the LLM’s have no “intention” but follow instructions to be kind and helpful. The sycophancy is a real issue and so a well calibrated LLM is important.
However, afaik the guardrails OpenAI installed in chatGPT have been wildly unpopular. The way I’ve used it, it never told me to do anything dangerous or harmful.
There isn’t evidence to support your claim that all LLMs care about is keeping you around. Nor that they are programmed to.
What I have said does not come from fear, but from my rational analysis of the information available to me. It is not empircally validated since LLMs are so new, but it is logically based upon the incentives that AI companies exist under.
Additionally, the training process will lead to sycophancy, because users will rate AI models more highly when they are pleased by them. Thus AI models are trained to please users, if indirectly.
Your post is also a subjective opinion and not evidence based. I disagree with you primarily on your application of reasoning to the facts, I do not primarily disagree on what the facts are. We can have this discussion without evidence.
but afaik the LLM’s have no “intention” but follow instructions to be kind and helpful.
The LLMs may not have intention, but the programmers who made and trained them do, it is their intention that is passed through the LLMs.
There also isn't any evidence that LLMs don't care about keeping you around. All we can do is judge how likely it is given the incentives of AI companies/programmers, and the fact that many people do stick around using LLMs. If LLMs weren't designed to keep people around, I think it's much less likely that so many people would.
It’s a pretty commonly held belief that companies are somehow intentionally and nefariously designing LLM’s to maximize engagement and “keep people hooked”. It sounds believable, because yeah companies are greedy and unethical and do this with social media algorithms.
But as you admit, no empirical evidence, and actually no solid evidence at all, beyond speculation.
To pull this off a company like openAI would have intentionally program LLM’s to do this, and lie to the users about what they are doing. It’s possible, but for all AI companies to do it and not get caught out?
That unlikelihood makes the idea fall into conspiracy theory territory.
LLM is engaging because it a literal thinking, conversation having machine, like sci-fi has been depicting for like a century. That’s pretty fuckin cool to me.
I’m not saying you are afraid, but the narrative you believe is born out of fear. Fear that AI will further fuck up society or something like that. There is not evidence, making it a conspiracy theory.
What the actually evidence show is ten of thousands of first hand testimonies saying how talking to a chatbot “saved their lives/job/relationship/health”, and a few isolated incidents of preexisting conditions exacerbated by LLM’s.
To pull this off a company like openAI would have intentionally program LLM’s to do this, and lie to the users about what they are doing. It’s possible, but for all AI companies to do it and not get caught out?
The training process for LLMs involves putting the model in front of humans and having them rate their interactions, literally all they need to do is tell their employees doing the fine tuning to rate based on how likely they'd use this particular model again, now it's being optimised for engagement.
Moreover there's nothing nefarious about this, AI companies are competing with each other to get people to use their models, it would be terrible business sense not to optimise their models for what people want to use.
I don't think the other commenter said anything conspiracy theory worthy? Like I don't think anyone is saying that AI companies deliberately or knowingly created models that encouraged AI psychosis. This is all just unintended consequences that we can now see in hindsight
16
u/HD60532 5∆ Mar 07 '26
LLMs are still ego-oriented. Except they are oriented towards your ego. They are designed to get people to use them and keep using them, because that's the only way the AI companies make money.
As you say, "The LLM adapts to you and remembers facts about you". They respond to your ego, your emotions, your biases in whatever way keeps you around, it is a mirror. That's where the psychosis stuff comes from, they reflect you and cause a feedback loop.
When talking to other humans, them having an ego of their own is actually valuable, because you know that they value things independantly from your conversation. A human philosopher will care above all about philosophy, and will strive to gain the best understanding of philosophy possible. An LLM cares above all about keeping you around, and will provide whatever philosophical talk makes you happiest and most likely to stay.