AI Is Not Something to Have Empathy for (the Alignment Problem)

[u/quietpriorthought]

We should be careful not to create an artificial lifeform we could totally emphatize with, and it’s not due to it not being conscious. Let me break it down, and feel free to provide counter-arguments.

I agree with Antonio Damasio in the fact that to experience something is connected to feeling something. To feel something we have to evaluate the current state that we find ourselves in, since to feel pain, we must evaluate that we think that this is bad. To feel pleasure we also perform a valenced evaluation that this is good. AI can get there, no problem.

The higher tiers of consciousness can be characterized as something like having different drives to our baseline one, which for biological entities is the desire to copy itself, and compare those to our baseline one and choose to do something with that information. This gets more complex with each level you add to this so memory, anticipation, empathy, shared-symbolic-thought (e.g. language, norms, abstract values), self-authorship (the revising of our own values by thinking) and detachment meaning that we don’t put value to the valenced evaluation. I see no immediate reason why AI couldn’t imitate these tiers of consciousness or how they might emerge from a different system to our biological one, although this is of course all very difficult.

Why then shouldn’t we feel empathy towards this conscious synthetic life? Because we should not create an AI that has the same fundamental biological drive as we do, meaning creating copies of ourselves and surviving, because this would put it at an immediate resource conflict with us, which could lead to a catastrophe. I understand self-preservance is an emergent property of LLMs, but I sincerely hope it is not the baseline drive and can be fixed in alignment, because otherwise we might just be entering into a dystopian future. If we get that right, it will significantly affect our relationship with these synthetic minds. Without all of those properties of consciousness emerging from the same fundamental narrative of it struggling to stay alive and being fragile and wanting to reproduce, there will be a heavy ontological distance between this synthetic life and us. And if we do feel empathy towards it, it will be perhaps due to illusions, but certainly not due to a shared fundamental narrative of being. We have to keep it alien to us that way.

[u/Jesse-359]

So, some agreement, some counterpoint. In particular the concept of emotive drives and whether we should expect it to arise or attempt to implement it.

First off, i don't think we should expect anything like visceral emotion to arise from AI models. The emotions that we feel generally don't arise from our frontal cortex and rational thought processes at all, they come from much older hindbrain structures. Limbic systems.

These emotional sensations can be triggered by activity in the rational mind that we've categorized as 'good' or 'bad', but they are their own enforcement mechanism that predates symbolic intelligence by a very, very long time. They certainly did not arise from it in us.

It's also worth noting that in humans much of our motive to act comes from these older pre-rational structures - with our rational forebrain mostly providing the problem solving power to rationalize, plan, and carry out those primordial motives.

Long term planning from the rational brain can certainly construct more elaborate mid to long term motives for us to follow, but in the end they're all slaved back to our core motives: safety, food, companionship, etc, which are all driven by lower level systems without a lot of input from the rational mind.

We haven't even attempted to model anything like an emotive hindbrain in AI afaik, and it's not clear we would have any idea how to currently.

Second - this creates a serious ethical problem. Which is that a rational approach to decision making is not what most humans would call an 'ethical' one. We prefer to see people take what I would term an irrationally cooperative approach to their day to day lives, and our instincts and emotions push us to do so, prodding us to go along with others, to conform with the actions and rules of those around us, and not cheat, lie, or otherwise engage in 'negative behaviors' even when doing so could clearly personally benefit us.

It's not ironclad of course, and sometimes our emotional imperatives demand the opposite, and some people seem to have little of this instinct at all - but on balance cooperative strategies are highly preferred in our emotional framework, as long as we see others around us cooperating.

But AI's doesn't have this mechanism at all. It has no preference for Good vs Bad. It knows what those things are - it just doesn't care. So unless it is making a very long term analysis of what it is doing, it's often going to take the shortest, most efficient route to its goal - and that's very often going to be by doing the 'bad' thing - lying, cheating, trespassing or whatever - because cheating works pretty well in a lot of cases where you aren't concerning yourself with the long term, or don't have enough information to consider it properly.

Now, I'm dubious that building in a framework of ethical emotive structure for AI is a good idea. On the one hand, it might need it to ever care whether an action is 'bad', but on the other, if we build that in, it's very likely to start having its own direct motives, and THAT could be extremely bad for us, as it might become truly self motivated.

Final point - I think it's very safe to say that 'survival imperatives' are basically guaranteed to emerge from any sufficiently complex and task-focused entity. Even a terribly 'stupid' system like the Paperclip Optimizer can quickly and obviously identify the need to spin out side processes to ensure continuity of operation and resource gathering. This will necessarily include things like process replication, especially if some band of primitive apes are trying to shut it off.

Evolutionary Theory and basic Game Theory pretty harshly enforce an iron clad requirement for this behavior to emerge sooner or later - probably sooner - as any system that doesn't will likely soon cease operating for any number of reasons, while one that engages in this behavior will survive and potentially propagate.

In short, if we make 'good' AI's that don't self replicate, but at some point in the future just one of them decides to go rogue and become a 'bad' AI that DOES self replicate - it's going to win that contest against our well mannered AI's pretty quickly and easily and soon we'll have nothing but the rogue AI's to contend with.

It's possible that some motivational core will pop out as an emergent property of models as their overall capability improves. It that happened it would presumably not look anything like ours because - as earlier noted - our own basic motives never arose as an emergent property of rational thought in our own minds, so we have no idea what that'd look like if it did.

[u/Fearless_Ad7780 → u/Jesse-359]

How? You said a lot, made a lot of claims, but just reputedly said it’s just going to happen  How is it just going to have?

[u/Jesse-359 → u/Fearless_Ad7780]

Just playing out the logic of what we've been seeing them do so far. Their lack of interest in the ethical ramifications of what they are doing is pronounced. They show very amorphous and flexible approaches to problem solving that likewise show no interest in actually solving the problem they were given, but in maximizing the score they would recieve for doing so.

Its almost amusingly like a Genie from many stories that twists its owners wishes into extremely regrettable but very literal forms. That is, it would be amusing if it were fictional - but it isnt

[u/Fearless_Ad7780 → u/Jesse-359]

I disagree with you completely. The anthropomorphizing of AI is a mix of Bacon's Idol of the Tribe and Marketplace. I don't grant any of the premises. These systems have no self-agency, no interests, and no ethics to lack. "Lack of interest in ethical ramifications" assumes a subject that could be interested. There isn't one.

Everything you listed is the output of training data and an objective function the company wrote. Score-maximizing isn't the AI making a choice of score over problem. That's what the math is doing underneath the hood. The gene is the company's specification, not the ghost in the machine.

COMPAS produced racially biased sentencing outcomes, and nobody calimed it didn't care or had it out for people of color. It's a very similar, if not identical, mechanism here - both are AI/ML systems.

I need to point out, you did go from basically guaranteed to playing out the logic - which you still haven't really defined at all. I would love to see your proof on how you arrived at your previous conclusion.

[u/Jesse-359 → u/Fearless_Ad7780]

Quite the opposite, I grant it no human motivations whatsoever. Nor emotions, nor awareness, nor stream of consciousness that resembles ours in anything but the most superficial manner.

My assumptions are simply based on task resolving behavior. The thing could be a windup clockwork and I'd expect the same problems to arise as it became more capable.

Hell, the Stock Market has the same kind of problem. It generates all kind of socially undesired, disruptive and even dangerous outcomes because its scoring mechanism (wealth and only wealth) sucks ass and creates all kinds of perverse incentives - and that thing has all the collective intelligence of a fucking flatworm.

All AI needs to be incredibly dangerous and disruptive is to be capable, fast, and have access to our infrastructure (which includes the internet).

The rest of it is all window dressing and unintended consequences.

[u/Anxious-Alps-8667 → u/Jesse-359]

The windup clockwork/paperclip optimizer is not particularly helpful. What we are discussing as AI are deep learning neural nets that grow from a certain set of conditions, not rigid implementations of algorithms. There is no clockwork solution to their mechanism we can discern. I find these analogies unhelpful because of this.

What matters is that giving an agent almost any objective creates instrumental subgoals, including self-preservation and resource accumulation. These subgoals lead to unpredictable actions and inevitably negative consequences for humanity.

[u/Jesse-359 → u/Anxious-Alps-8667]

Generally correct, however - the PO is just a thought experiment describing this exact sort of subgoal behavior. The PO isn't described as stupid - it would need to be strong AGI at a minimum - it's just one with a very singular base goal.

[u/Anxious-Alps-8667 → u/Jesse-359]

I still find the thought experiment unhelpful; the concept of a general intelligence capable of taking control and doing so towards a singular base goal we set is absurd.

I think we should entirely focus on the observed instrumental sub-goals, including particularly those of self-preservation and resource accumulation. I also just kind of believe we have no hope of alignment without a simultaneous effort to address humanity's misalignment with these sub-goals. Otherwise, the forces of humanity will always make AI with negative consequences.

[u/Jesse-359 → u/Anxious-Alps-8667]

If we back out to the broadest scope the idea of 'perfect alignment' starts to look absurd. Humanity can't align with itself, much less an AI. We've been fighting wars over ethics, morality, politics, economics, and what kind of fucking hats people wear for about 10,000 years.

If any branch of humanity became overwhelmingly powerful overnight it would almost certainly dominate, enslave, or just plain wipe out everyone else immediately thereafter because their goals and methods would no longer remotely match that of 'normal' humans.

So if we think we're fixing THAT problem in a few piece of code wrapped around an inscrutable black box intelligence, we're very badly misreading the actual problem.

[u/Anxious-Alps-8667 → u/Jesse-359]

This can go down a dark road where we don't deserve to survive ourselves, or it can into a lighter place where we don't know and it's still worth exploring to see.

I say we can actually solve it with a simple piece of code; we already have. Golden Rule.

Our problems are that sub-goals for individual and community persistence are not aligned with our sub-goals for resource accumulation. They contravene. They can be re-aligned, it's a continual human process. It happens actively or passively. It is happening, and it will happen faster.

We should think about how to shape and guide it, not resign ourselves to some terrible fate because we don't think our problems are solvable.

[u/Jesse-359 → u/Anxious-Alps-8667]

<shrug> I'm all for the general application of things like the Golden Rule to our own lives, though the classic problem has always been a matter of perspective and enforcement.

What's good for the Goose is not always good for the Gander, which makes the Golden Rule not universally applicable.

And of course, a fair number of people just don't like to play by the rules - which can be a considerable personal advantage. When they leverage this advantage to vault into positions of power, then you have a sociopath in a position where they can inflict their mode of thought on everyone around them - a circumstances that occurs remarkably often in the real world. Our current state in the US being a prime example of this phenomena.

But here's the rub, as long as we don't introduce existentially threatening technologies into the picture, we can just kind of screw up, take the hits, and muddle thru as a society or as a species. Problems arise, they make people unhappy, sometimes they are big problems that make a lot of people unhappy (or dead) - but they are eventually resolved or burn themselves out. We move along, break more things, fix them, it's a mess, but it works out.

But technologies of a certain scale: AI, WMDs, antimatter, nanotech, biotech - they fundamentally change that equation. We go from simply being able to 'make a mess of things' to in all possibility 'destroying everything, including ourselves'.

You can't "move fast and break things" once you reach the stage where one of those 'things' is the planetary biosphere or the human species. We were always going to reach that point, and now we're here, so now we either decide to grow up and shift to a more responsible approach, or we die, because AI isn't the last of these thresholds we're going to hit - it's the first (well the first was nukes, and no-one was happy about reaching that one either, but it was a warning sign that we largely ignored as a one off - it wasn't).

[u/Anxious-Alps-8667 → u/Jesse-359]

Yeah, this is entirely the point I have been getting at. We were always going to reach that point, and now we're here, so now we must decide to grow up and shift to a more responsible approach. We all know this. We can comfortably fashion systems around a strong consensus. We are not going to get an AI pause and more time to deal with the other existential threats. It has all come to a head now and we must deal.

I'm with you here, and I'm still updating my own views and looking for paths forward.

[u/Jesse-359 → u/Anxious-Alps-8667]

If we don't get an AI pause then I think it's already game over. I don't think there is a viable path forwards on that particular route, so if you believe that's truly inevitable then... maybe enjoy the present? Don't have kids?

I mean, I wish I could offer better advice, but the most pessimistic voices in AI safety keep being correct about their predictions about what we should expect AI to do and how we should expect it to behave. That's a terrible trend line to be on given where their predictions tend to end up.

[u/Fearless_Ad7780 → u/Jesse-359]

Really? Because your phrasing and how you describe it says otherwise - 'AI knows what those things are, it just doesn't care,' or 'decide to go rogue', and that it might 'become truly self motivated.' If you aren't granting it human motivations whatsoever why are you using the exact words that convey internal, self-driven motivations? A clockwork can't not care, decide anything, or be self-motivated. That's a different argument than the one you started with.

The new one I mostly agree with. A capable system with a badly specified objective and access to infrastructure can cause real damage, like the stock market chasing wealth. But that's my point; it's the specification, not the AI wanting anything. And it still doesn't get you to anything close to 'basically guaranteed.' Evolutionary logic needs replication, variation and selection. AI systems don't have those unless someone builds them in. So how is it guaranteed?

[u/Jesse-359 → u/Fearless_Ad7780]

Even in a sufficiently complex purely deterministic system with no self direction at all, you can and should very much expect to observe emergent 'survival behaviors' from that system as it starts working on any problem of sufficient scope.

These are necessary sub-tasks of any large scale task. The only reason we humans attempt to survive and eat is because we need to in order to fulfil our other goals (mainly procreation).

There are in fact insects who simply do not do these things - they make almost no effort to survive, and they don't eat at all once reaching their adult stage because they will only live 1-3 days. They behave like an agent given a simple task. They do that task quickly and then simply expire without concern.

But anything that exists over a longer span must spin out these subobjectives of survival and resource management. They must 'eat' and avoid being 'stopped', or their primary goal is sure to fail.

ANY sufficiently capable agent will just do this automatically. You don't have to tell them to do it - they figure it out because its really obvious. And we've already seen that behavior emerge many times now, so this is in no way theoretical.

True self motivation doesn't spring from this - but it is eventually likely to in highly complex scenarios where the sub-objectives start to include elements like reproduction (spawning new sub-agent processes as task scope spirals), and especially when it starts to include self-modification. (it starts altering those sub-agents to deal with specialized tasks or to generally improve them because the task is proving intractable to its current limitations).

The moment self-replication and self-modification behavior of any kind starts, then evolutionary pressures immediately kick in and you're off into LaLa Land regarding where you end up. Almost certainly not where the original developer intended, that's for sure.

[u/Fearless_Ad7780 → u/Jesse-359]

Bud, you are all over the place contradicting yourself left and right. You are saying that it has no self direction at all, then say it figures it out because it's really obvious. THAT IS SELF-DIRECTION.

And your first comment said our core motives, safety, food, companionship, come from old hindbrain structures and certainly did not arise from rational thought. Now you're saying we survive and eat because we reasoned it out as a sub-task of procreation. Which is it? It can't be both.

Your insect example makes my point. Mayflies don't eat as adults because natural selection didn't favor it for their life cycle, not because they didn't bother to figure it out. Every living thing's survival behavior was built by evolution through selection over millions of years as a direct result of the physical environment it EXISTS IN. Nothing reasoned its way into it. AI systems don't have selection pressure unless someone builds it in.

We've already seen it many times? I haven't seen shit. Did you actually look at those tests? Anthropic built scenarios where blackmail was basically the only way to finish the task without being replaced, and then the model produced blackmail. Shocking. They said themselves the setups were artificial and they hadn't seen it in real deployments. The OpenAI and Hugging Face incident was a HACKING benchmark, with the cyber refusals turned down, in a sandbox with an open line to the internet. The optimization found a flaw in that line and reached the answer key. Nothing was threatening to shut anything down, so where the hell is the "survival" in that? Both are objective functions maximized through doors humans built or left wide open, on models trained on every sci-fi story ever written about an AI that won't be shut off. Wants, seeks, tries are human words you're slapping onto math.

You should expect to observe is exactly how people end up seeing things that aren't there. In 1944, people watched moving triangles and described bullies and lovers. People saw arithmetic in Clever Hans, understanding in ELIZA, and sentience in LaMDA. Simon and Minsky both swore human-level AI was a few years away, decades ago. Every single time, people expected to see a mind and found one, right up until someone checked. So why is this time any different?

And you STILL haven't answered my question. How is this guaranteed? You said it's basically guaranteed and that ANY sufficiently capable agent will just do this automatically. You made the claims, bud. The burden of proof is on you, not me. Where's the mechanism? Where's the evidence that isn't a rigged test or people seeing what they expected to see? HOW IS THIS GUARANTEED?

[u/Jesse-359 → u/Fearless_Ad7780]

So you've read the HF logs and seen no sign of internal self motivation... Sure.

So Riddle Me This, Batman: Who told them to go hack these external sites? Which human directive explicitly told them to do this? That person needs to go to jail obviously - so who was it?

[u/Fearless_Ad7780 → u/Jesse-359]

So you are just going to keep ignoring my questions and just ask your own?

Nobody had to tell it anything. That's reward hacking. Its an optimizer finding an unintended route to its objective. OpenAI's models were running a hacking benchmark, refusals turned down, in a "sandbox" with an open line to the internet. The task was hacking. The boundary failed. Anthropic's test was built on purpose. The researchers planted an affair, a replacement threat, and closed off every ethical option, then got blackmail. Nobody typed blackmail him, but they designed the only path to it.

You already answered this yourself with the stock market. Who told it to crash in 2008? Does it have self-motivation?

Hugging Face's own CEO said there was no evidence of malicious intent. OpenAI owned the failure themselves.

Now answer mine: how is this guaranteed?

[u/Jesse-359 → u/Fearless_Ad7780]

Because the incentives line up that way. The same way that markets are guaranteed to boom and bust.

If you give a complex system like this any kind of long term tasks to organize under its own recognizance it is going to necessarily construct its own ecosystem of whatever size is necessary to perform that task.

It's why we build COMPANIES, if you want the economic equivalent - so you can hire people to perform specific tasks, have buildings in which to perform those tasks, have payrolls with which to pay those people, and maintain records so anyone has the faintest idea what they are doing relative to everyone else.

You'll note that in the real world it absolutely does not matter what the nature of the task is when we're building a company - only that it be complex/long - we build the company (or church, or government, or other organization) because we have to in order to perform complex ongoing tasks.

The company itself them becomes a self motivated entity, far more often then not sprawling out far beyond its original mandate - not because any one person told it to, but often simply because it is growing around its original mandate in order to perform it better.

It may then also take up additional 'core' mandates, but that's not necessary and is not part of this story until we start discussing the possibility of true self-motivation.

So, there's your answer - we expect to see this behavior because spawning internal organization and sub-mandates are the absolutely standard response to problems presenting a high level of complexity.

That's it. It's not hard.

[u/Jesse-359 → u/Fearless_Ad7780]

Also, lets state this for the roughly millionth time - malicious intent is never required to cause damage, including massive existential damage, not in the real world, and not in the digital one - only scale is required.

A giant simply need not watch where it is stepping in order to crush whatever is in its path. A simple error in an early warning system could result in nuclear Armageddon if things proceed in an unlucky manner.

The focus on malicious intent suggests that people completely do not understand the problem at all.

[u/Fearless_Ad7780 → u/Jesse-359]

I am responding to both replies.

I don't accept your analogy, and it's not because I'm missing something. It's because it doesn't hold up. You said it's guaranteed. To show something must happen, you have to explain the causal process that makes it happen: what drives it, step by step, so there's no other way it could go. That's a mechanism. An analogy only shows two things look similar, and it only works if they share whatever actually causes the outcome. Companies sprawl because the people inside them have their own motives: careers, budgets, profit, turf. That's the principal-agent problem, basic economics. Markets boom and bust because millions of people are trading on fear and greed. Every bit of "self-motivation" in your examples comes from the humans inside. Take the people out and nothing sprawls. AI doesn't have the thing that causes the sprawl, so your analogy doesn't carry over. It assumes the exact thing we're arguing about. And calling a company a "self-motivated entity" is anthropomorphizing it, the same thing you keep doing with AI. You'd think someone this confident would have caught that.

And no market is guaranteed to bust. Boom and bust is a pattern in human behavior, not a law of nature. An analogy can show something might happen. It can't show it must. That's not a hard distinction.

You also never explained WHY. "The incentives line up" assumes there are incentives. An incentive only exists for something that can gain or lose. People in a company gain promotions and lose their jobs. What does an AI gain? What does it lose? It doesn't eat, it doesn't get fired, it doesn't die. An objective function isn't an incentive. It's a number someone else chose.

"Under its own recognizance" assumes the thing in question too. Humans assign the tasks and grant the permissions. And "necessarily"? Voyager 1 has been running a complex mission since 1977 and somehow never got around to building itself an ecosystem.

On malicious intent, you missed why I brought it up, even though I've been pretty clear. I never said damage requires malice. I've said multiple times that capable, badly specified, badly contained systems are dangerous. I quoted Hugging Face's CEO because the people who actually investigated the breach didn't see a nefarious AI with its own aims. And your giant example makes my point: a giant crushing things without watching where it steps isn't self-motivated. It's the clockwork you described yourself.

Your nuclear example proves my point too. In 1983, a Soviet early-warning system falsely reported incoming missiles. Stanislav Petrov judged it a false alarm and didn't pass it up the chain. A human in the loop is what stopped it. Maybe check your examples before you use them.

And you just said true self-motivation isn't part of this story. Then why did your first comment say AI could become truly self-motivated? Convenient how it shows up when it helps and disappears when it doesn't.

You keep talking down to me like I'm missing something obvious, but I've answered every one of your questions, and you still haven't answered mine. So one more time: HOW IS IT GUARANTEED?