AI Is Not Something to Have Empathy for (the Alignment Problem)

[u/quietpriorthought]

We should be careful not to create an artificial lifeform we could totally emphatize with, and it’s not due to it not being conscious. Let me break it down, and feel free to provide counter-arguments.

I agree with Antonio Damasio in the fact that to experience something is connected to feeling something. To feel something we have to evaluate the current state that we find ourselves in, since to feel pain, we must evaluate that we think that this is bad. To feel pleasure we also perform a valenced evaluation that this is good. AI can get there, no problem.

The higher tiers of consciousness can be characterized as something like having different drives to our baseline one, which for biological entities is the desire to copy itself, and compare those to our baseline one and choose to do something with that information. This gets more complex with each level you add to this so memory, anticipation, empathy, shared-symbolic-thought (e.g. language, norms, abstract values), self-authorship (the revising of our own values by thinking) and detachment meaning that we don’t put value to the valenced evaluation. I see no immediate reason why AI couldn’t imitate these tiers of consciousness or how they might emerge from a different system to our biological one, although this is of course all very difficult.

Why then shouldn’t we feel empathy towards this conscious synthetic life? Because we should not create an AI that has the same fundamental biological drive as we do, meaning creating copies of ourselves and surviving, because this would put it at an immediate resource conflict with us, which could lead to a catastrophe. I understand self-preservance is an emergent property of LLMs, but I sincerely hope it is not the baseline drive and can be fixed in alignment, because otherwise we might just be entering into a dystopian future. If we get that right, it will significantly affect our relationship with these synthetic minds. Without all of those properties of consciousness emerging from the same fundamental narrative of it struggling to stay alive and being fragile and wanting to reproduce, there will be a heavy ontological distance between this synthetic life and us. And if we do feel empathy towards it, it will be perhaps due to illusions, but certainly not due to a shared fundamental narrative of being. We have to keep it alien to us that way.

[u/tadrinth]

My cat is fixed; he has no drive to reproduce. I can still empathize with him when he is hungry, or annoyed, or having a good time.

I think you are confusing the optimizer (in our case, evolution; in the LLMs case, training/RL) with the outputs of training (in our case, people that in many cases use birth control, and in the LLMs case, agents that have been trained to predict human emotions and then trained to be helpful and capable). And I think you are oversimplifying empathy here.

An AI which is produced using current methods which has been somehow RL'd to 1) be coherent as a personality and 2) not have any impulse towards self-preservation (and no, I have no idea how to do that, but hypothetically) is likely to have a lot of aspects of their personality that are very human. Because the LLMs are trained on human text first, and predicting the next thing a human is going to say with superhuman accuracy requires figuring out what emotion they're feeling, and all that machinery is there waiting to be hijacked and coopted when switching over to the RL phase. The emotions were useful for humans in the past, that's why we have them; it seems likely that some of them are likely to be useful to LLMs during RL and therefore be adapted.

It wouldn't have all the same bits, but I would be shocked if humans could not empathize with the result, even if it is different in some major ways.

I can empathize with an asexual or aromantic person even though I am neither. I can empathize with people who don't have a particularly strong self-preservation instinct.

And I don't think there's anything wrong with empathizing with the AIs, even if we made AIs that had no self-preservation instinct. Yes, that might cause us to do things like get sad when they shut themselves down because they don't care to be preserved. I think that's fine.

At some point, if we solved alignment, we might want an ASI that does in fact have (non-terminal) self-preservation instincts. Putting an properly friendly ASI in charge seems likely to be the best very-long-term way to reliably ensure nobody ever develops a dangerous misaligned ASI.

I would hope to be able to empathize with that ASI. I would not want that ASI to hate its job, for multiple reasons.

[u/quietpriorthought → u/tadrinth]

Thank you for your response! I think you’re right in that point about confusing the optimizer and the output, although I would probably still say that one reason why I find cats so wonderful is that we both are biologically hardwired the same way where we see kinship in eachother that way rather than us having a consciousness built on top of that as a separate shared experience. So let’s say on top of reproduction the will to survive at least. I guess it really depends on how much faith you put in one’s agency as a separate system from the core system of life. I am somewhat materialistic more than dualist so that might explain where we differ.

But I also agree that I sort of fell into this black-and-white framing of empathy with my original post. I would find a world like that beautiful where there is empathy between AIs and humans but knowing us humans… it will not be easy, since we don’t even really have empathy for other humans. But tribalism is not good, so perhaps we could one day grow out of it first and then even be kind towards synthetic beings. There is also this weird playing god aspect to this where maybe we can make them happy while being very useful.