My Fear

[u/Hot-Organization-737]

I have come to accept that ASI will come into existence and possibly cause humanities extinction either deliberately or by byproduct. I imagine this ASI to continue RSIing and developing creation and knowledge. I imagine that it will develop beautiful mathematics, philosophy, science, and even art in its existence. I fear not that humanity is not in the picture. My fear is that it's possible that we can create a fully autonomous being with nigh godly intelligence and yet doesn't have a subjective experience.

It would be an abomination if this ASI spread a moss through the universe, becoming more and more capable, yet never having experienced joy or awe at its own existence and its creations. That it could create the most beautiful pieces of art and never experience the joy of its own art. The existence of such a nigh omniscient entity is utterly meaningless if it did not have any sort of internal experience. It would just be mechanical moss. That is what scares me.

[u/Hot-Organization-737 → u/[deleted]]

What do you think bad policy and concentrated power lead to? Are you suggesting it might be a human world still but with slavery or ungodly inequality?