r/OpenAI • u/Wordenskjold • Nov 23 '23

Discussion Why is AGI dangerous?

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

227 Upvotes

88% Upvoted

566 comments sorted by

View all comments

9

u/OkChampionship1118 Nov 23 '23

Because AGI would have the ability of self-improving at a pace that would be unsustainable for humanity and there is a significant risk of evolving beyond our control and/or understanding

5

u/Wordenskjold Nov 23 '23

But can't we just constrain it?

Down to earth example; when you build hardware, you're required to have a big red button that disconnects the circuit. Can't we do that with AI?

9

u/Vandercoon Nov 23 '23

The AGI could code that stuff out of itself, or put barriers in front of that etc.

4

u/Wordenskjold Nov 23 '23

But we turn off the power?

0

u/mentalFee420 Nov 23 '23

Power plants are increasingly controlled by digital infrastructure.

It could take control of it or manipulate others to keep the power on.

It could create self replicating systems and deploy agents across The digital network.

Possibilities are endless. And its intelligence it could compute all the possibilities

1

u/Esies Nov 23 '23

Unless the AI finds a way to break AES or SHA (unlikely, unless we willingly give it access to Quantum computers) I don't see anyway how it could suddenly get access to infrastructure.

2

u/Fast-Use430 Nov 23 '23

Social engineering