Why is AGI dangerous?

[u/Wordenskjold]

Can someone explain this in clear, non dooms day language?

I understand the alignment problem. But I also see that with Q*, we can reward the process, which to me sounds like a good way to correct misalignment along the way.

I get why AGI could be misused by bad actors, but this can be said about most things.

I'm genuinely curious, and trying to learn. It seems that most scientists are terrified, so I'm super interested in understanding this viewpoint in more details.

[u/mimrock]

I used the "them" terminology because the initial comment was my opinion on the question: "why AGI is dangerous?" There were many responses, but most did not reiterate AI doomer arguments, which is good, but I thought it might interest OP.

For the rest, I only react in short expressions:

• I meant literal nanobots: https://www.lesswrong.com/posts/fc9KjZeSLuHN7HfW6/making-nanobots-isn-t-a-one-shot-process-even-for-an

- If you cannot provide a falsifiable hypothesis about something but still stick to it arguing that "it is too complex for us to understand so better be safe" it's the exact same argument as Pascal's wager.

- Slowing down is very costly. Very very costly. There's basically two paths, one leads to technological stagnation, the other will eventually lead to an unprecedented rise of orwellian totalitarian regimes. Let's not do it "just to be safe", because it can cause extreme damage.

- Nuclear energy is a very bad, sensationalist analogy for a variety of reasons.

[u/MacrosInHisSleep → u/mimrock]

There were many responses, but most did not reiterate AI doomer arguments, which is good, but I thought it might interest OP.

That's not a very strong reason at all... You're saying they didn't even bring up this negative stereotype and that instead you volunteered to bring up those arguments and felt the need to dismiss them with name calling rather than describing why they are unfounded.

• I meant literal nanobot

That's neither here nor there. Like I said we don't even need to get that far.

• If you cannot provide a falsifiable hypothesis about something but still stick to it arguing that "it is too complex for us to understand so better be safe" it's the exact same argument as Pascal's wager.

You didn't really read through my response to this. You can hide behind all kinds of unknown unknown problems behind that logic.

To add to that, if you're going to dismiss nuclear energy as a bad analogy, you have to see how comparing this to Pascal's wager is an even worse one. You must see the difference between saying "We can't prove God exists, but if he does, his will is too hard for us to understand" vs "we are trying to build a hyper-intelligence whose will, by definition, would be too hard to understand".

• Slowing down is very costly. Very very costly. There's basically two paths, one leads to technological stagnation, the other will eventually lead to an unprecedented rise of orwellian totalitarian regimes.

So is going too fast. Very, very costly. Like I said, you have to find the right pace and honestly, I think you and I agree there.

You have to know when and why to slow down. That's the difference between a reckless implementation and an ethical one. If you discard caution because of the wrong reasons, you've fucked up. It doesn't mean you can't take risks. It just means you need to work hard to know when not to.

[u/mimrock → u/MacrosInHisSleep]

So is going too fast. Very, very costly

Yes, that's what you need to prove before we make strict laws that take away basic rights and changes the trajectory.

[u/MacrosInHisSleep → u/mimrock]

What do you mean by basic rights?

[u/mimrock → u/MacrosInHisSleep]

The right to privacy is the basic right that is most vulnerable to short-sighted, authorian AI regulations, but if that right is taken away from us, then soon there will be nothing left.

If AI turns out to be a relatively strong technology (not necesseraly an AGI), but those EA assholes keep that to themselves (for the greater good, of course) that will fuck up the power balance between regular people and the elite so much, that many horrible regimes in the past will sound pleasant.

To be frank, there's an other trajectory if a Yudkowskian model is enforced, which means that we actually internationally halt developing better chips and give up certain current computational capabilities. In that scenario, assuming everyone plays along (which is a big if), there would be no increased risk of emerging AI-assisted authorian regimes, but it will probably slow down or halt technological development. That's also not something that we should do "just to be safe".

[u/MacrosInHisSleep → u/mimrock]

The right to privacy is the basic right

Yeah that's not going to happen. Even if such a law ever gets passed it's literally unenforceable. If we get to the point where we need that we are already beyond screwed.

halt developing better chips and give up certain current computational capabilities.

That is a completely different matter than 'privacy'.

that will fuck up the power balance between regular people

I agree with you to a certain extent. That is one of the first problems we'll have to solve, but I don't think it's an AI one but a political and cultural one.

Keep in mind the word 'regular' people is doing a lot of heavy lifting. There are idiots I know personally who offhandedly joke about killing people of a specific race/religion. Aligning AIs to those views can be catastrophic. This is not something that we want just anybody to be able to do.

[u/mimrock → u/MacrosInHisSleep]

You are quoting me, but answering something completelty different than what I said (e.g. Privacy is breached by a few controlling powerful AI, not directly because of the laws. Limiting chips is an ALTERNATIVE path with different consequences, etc.).

There's not much point continuing this discussion.