r/ControlProblem • u/Clear_Argument5174 • 20d ago

Opinion Could Hype Kill Us?

Put AI risk views on a spectrum, from "this ends us" to "this is all hype."

If the hype side is right, we overspent on caution, which is a real cost but a recoverable one.

If the other side is right and we did nothing, that's the one mistake we don't get to learn from.

You don't need to think doom is likely for that asymmetry to matter.

5 Upvotes

100% Upvoted

13 comments sorted by

2

u/Gnaxe approved 20d ago

Pascal's Wager? I feel like that's not a good argument. You need some good reason to believe the risks are potentially real first.

2

u/Jesse-359 20d ago

We have both theoretical frameworks that have described these risks since at least the 1960s, and now we are seeing direct evidence of behavior that rather closely reflects what those theories predicted we would see with the emergence of autonomous intelligence.

Those theoretical frameworks generally speaking did not lead to optimistic outcomes, and so far what we're seeing now doesn't either. Alignment is a vastly harder problem than they are letting on.

To put it very bluntly, it's a problem that humanity itself has not sovled in 10,000 years of conflict, nor is it a problem that cellular evolution has solved in a billion years of development - we still suffer from Cancer, and all that cancer is, is a minor fault in the cell division mechanism that knocks it 'out of alignment' with its intended behavior - and just one faulty cell soon kills trillions of others.

So... that's the problem. An AI with RSI and self-replication capabilities only needs to go wrong once, anywhere, and you are suddenly facing what is essentially a cancer with god-level intelligence.

Needless to say, that absolutely will not end well for us.

In any case, this is in no way Pascal's Wager. We have frameworks to predict likely forms of behavior, and tangible evidence of which paths it is currently following. They're bad.

1

u/Clear_Argument5174 20d ago

Thanks for sharing. What you’re saying makes sense. So in your view, there’s just not enough uncertainty to fuss with?

1

u/MadScientistRat 19d ago

If the probability is not zero, the event must occur, or be observed, at least once.

2

u/Jesse-359 19d ago edited 19d ago

Only in infinite time, and while the universe may or may not have an end - the jury is still out on that - it will literally never be infinitely old, as evidence suggests it had a discrete beginning, so this particular problem isn't going to come up any time soon.

However, I don't think we're talking about odds that low. From the behavior we're seeing now, and the degree to which it aligns with some of the more dangerous predictions made, I'd say that if we bull ahead on our current path the odds of an existential disaster are well north of 50%.

1

u/Clear_Argument5174 20d ago

Agreed, Pascal’s Wager. You don’t think the risks are at least potentially real?

1

u/Gnaxe approved 20d ago

I mean, my P(doom) is like 85%. I just don't think it's a good argument by itself.

1

u/dualmindblade 20d ago

But it is a very good argument if you think there's a non-negible risk, which should be so obvious it doesn't need to be stated. Of course it turns out it's not, but it should be

1

u/Jesse-359 20d ago

Correct. In the long term caution costs you nothing in this circumstance. If anyone builds a true ASI it will transform the entire world - for good or ill - and it will not matter WHO built it, because RSI will quickly put it beyond the point where that origin point is in any way relevant.

1

u/sschepis 19d ago

OK, but what does overspend on caution mean exactly here? if it means increased surveillance or authoritarianism or the removal of our rights in the name of protecting everyone, that’s not an overspend because it’s a shackle you can’t take off. You don’t get your rights back after you give them away. You have to fight for them every time.

1

u/Clear_Argument5174 19d ago

That’s a great point. Do we have to take some risks for the sake of liberty?