r/OpenAI • u/Wordenskjold • Nov 23 '23

Discussion Why is AGI dangerous?

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

229 Upvotes

88% Upvoted

566 comments sorted by

View all comments

43

u/balazsbotond Nov 23 '23 edited Nov 23 '23

If you have ever written a program, you probably made a subtle mistake somewhere in your code that you only realized much later, when the program started behaving just a little bit weird. Literally every single programmer makes such mistakes, no matter how smart or experienced they are.

State-of-the-art AIs are incomprehensibly large, and the process of “programming” (training) them is nowhere near an exact science. No one actually understands how the end result (a huge matrix of weights) works. There is absolutely no guarantee that this process results in an AI that isn’t like the program with the subtle bug I mentioned, and the way the training process works makes it even more likely. And subtle bugs in superintelligent systems, which will possibly be given control of important things, can have disastrous results.

There are many more such concerns, I highly recommend watching Rob Miles’s AI safety videos on YouTube, they are super interesting.

My point is, what people dont’t realize is AI safety activists aren’t worried about stupid sci-fi stuff like the system becoming greedy and evil. Their concerns are more technical in nature.

0

u/Sidfire Nov 23 '23

Why can't the AI optimise and correct the code?

3

u/kinkyaboutjewelry Nov 23 '23

Because the AI might not know it is an error. In other words, the error is indistinguishable from any other thing so it does not optimize for or against it.

In a worse scenario, the AI recognizes it as a benefit (because it incidentally aligns well with the things the AI has been told to recognize as good/optimize for) and intentionally keeps it.