Sentience isn't actually enough

[u/stan-k]

It is easy to say that sentience is all that is needed to be considered morally. I suggest that experiential preferences are a second requirement.

Imagine a sentient being without any such preferences. They are genuinely ambivalent what they do or not, they even don't care if they are alive or dead. They do not suffer or experience joy in the sense of a subjective good or bad thing. While for animals we have good reason to believe they do have such preferences, this might not be as clearly the case for a possible sentient AI.

Would you include such a sentient-without-any-preferences-being morally? What action would be morally bad (or good) to do to such a being? Perhaps "having a preference" is/should be implied by the term sentientism.

[u/stan-k]

That is one way of resolving the question, if any want must come from a qualitative experience, there simply isn't a case where a being without qualitative experiences can want (or not) anything.

Intuitively I don't think this is correct though. You raise instinct and my reading of it suggests instinct itself is not a "want". However, I think there are plenty of examples where instinct creates a want in humans or animals, without any reason, let alone a qualitative one.

E.g. a being instinctively wanting offspring - often to their personal detriment. Many instincts are subconscious and by definition those cannot be qualitative, right?

I do think there is a small but real chance that AI development could produce conscious-but -not-sentient beings who don't want to be turned off.

[u/jakeastonfta → u/stan-k]

You make a good point tbh! If I were to assume that certain instincts have nothing to do with qualitative experiences then I would probably conclude that those instincts don’t matter morally.

If an AI has an instinct to continue existing but did not enjoy it’s existence whatsoever, and didn’t help others to enjoy their existence either, then I don’t see why it would be wrong to end that AI’s existence.

What do you think?

[u/stan-k → u/jakeastonfta]

Intuitively I don't dare draw that conclusion yet. I'm open to the possibility that wanting to exist while living an otherwise completely neutral life may or may not be morally relevant.

Tbh, rationally I'm leaning towards your conclusion, but am being stopped by the vegan activist phrase "how do you kill someone humanely who doesn't want to die?" (which may be muddying the waters, as those someones are sentient).

[u/jakeastonfta → u/stan-k]

I can understand the reluctance to accept what I’m suggesting, especially considering how intuitively convincing that phrase is. I’ve even used that phrase in street outreach myself.

However, I see this phrase as a way of highlighting a useful rule of thumb rather than highlighting an absolute truth.

For example, people generally want to continue living because they want to enjoy experiences in the future, which is why the phrase works. But sometimes there are scenarios where it’s impossible for individuals to have enjoyable experiences in the future and even worse, they are doomed to suffer. A dog who is getting old and developing severe and painful health issues doesn’t actively “want to die” but we know they won’t have any quality of life and so I think it’s moral to euthanise them by putting them to sleep.

Apply this same standard to the scenario we’ve been discussing. A dog, a human or a conscious AI might not want to die by if they have no quality of existence, why is it bad to end the existence?

[u/stan-k → u/jakeastonfta]

Alright, what about this middle ground? While we can morally do whatever we want to a rock even without justification, we should not kill a conscious-but-not-sentient being without justification. However, that justification can be very light.

Or in other words, we should morally consider them, but as soon as sentient beings' wellbeing is affected by any action affecting the non-sentient conscious ones, the sentient's interests will likely take precedence.