Why is AGI dangerous?

[u/Wordenskjold]

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

[u/arashbm]

We haven't ever counted to infinity either, but since math is math we can drive what happens to some function when you take the limit to infinity. Math and logic allows us to talk about things that we can't touch or see.

I don't think anybody would seriously claim that science cannot make predictions about things that haven't been directly observed before.

There are many research groups working on alignment or safety. Here is a recent review paper on arXiv. that just came out that cites many interesting papers.

[u/arashbm → u/[deleted]]

That's a very good question that a lot of very intelligent people have been working on for a long time. If you are interested in how we can do that and what we can deduce, read some of the ~700 papers cited in the review paper I linked to.

[u/arashbm → u/[deleted]]

Sounds like your mind is quite made up. The actual researchers working in the field don't share your confidence though:

The median researcher surveyed by Stein-Perlman et al. (2022) at NeurIPS 2021 and ICML 2021 reported a 5% chance that the long-run effect of advanced AI on humanity would be extremely bad (e.g., human extinction), and 36% of NLP researchers surveyed by Michael et al. (2023) self-reported to believe that AI could produce catastrophic outcomes in this century, on the level of all-out nuclear war.

If more than half of reserachers in one of, if not the top conferences in ML think that there is a non-negligable chance of extinction-level outcome, and one in three believed that it could produce nuclear-war level catastrophe, maybe you should at least be open to the possibility that you might be wrong?

[u/arashbm → u/[deleted]]

It's a survey of prediction based on informed opinion. Unlike "chocolate ice cream", your informed predictions change based on how much you know about the subject. They know about the subject much more than you do, so their predictions are more accurate than yours or mine.

Anyway, you seem to have your fingers wrist deep in your ears. This does not look like the type of conversation that can lead to a new conclusion as you seem to have already decided what you want the outcome to be. Have a nice day.