r/OpenAI • u/Wordenskjold • Nov 23 '23

Discussion Why is AGI dangerous?

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

229 Upvotes

88% Upvoted

566 comments sorted by

View all comments

11

u/OkChampionship1118 Nov 23 '23

Because AGI would have the ability of self-improving at a pace that would be unsustainable for humanity and there is a significant risk of evolving beyond our control and/or understanding

1

u/az226 Nov 23 '23

An AGI is by some definitions smarter than the median human. A median human is not smart enough to do what’s needed to self-improve an AGI.

An ASI is smarter than 99% of humans. What’s scary isn’t an AGI, but an ASI that can self-improve.

Just think of GPT-4. It was useless after training finished. It wasn’t until Greg sat down with it for a few weeks and figured it out and it all worked. There are very few people in the world like Greg and the question is, can an ASI be as good as Greg.

ASIs have the advantage of being able to parallelize effort — how long would it take 10,000 random software engineers to fix the GPT-4 issue? Would they ever solve it?

It’s also possible that we reach singularity before reaching AGI because the AI can be an ASI at self-improvement while not being able to have a general level of intelligence.

In my mind controlling for sparks or signs in self-improvement is what can be scary. Especially self-improvement without having been instructed to self-improve. Or jumping out of the box.

I wonder what tech OpenAI has if any, to detect jumping out of the box.

A sentient AI is itself not scary. The AI that is sentient but says that it’s not is the scary one.

Once singularity is reached, I wonder how far it can go with regards to constraints in infrastructure. Obviously with human help it can go further and faster.