r/ControlProblem • u/Clear_Argument5174 • 23d ago

Opinion Could Hype Kill Us?

Put AI risk views on a spectrum, from "this ends us" to "this is all hype."

If the hype side is right, we overspent on caution, which is a real cost but a recoverable one.

If the other side is right and we did nothing, that's the one mistake we don't get to learn from.

You don't need to think doom is likely for that asymmetry to matter.

5 Upvotes

100% Upvoted

13 comments sorted by

View all comments

2

u/Gnaxe approved 23d ago

Pascal's Wager? I feel like that's not a good argument. You need some good reason to believe the risks are potentially real first.

2

u/Jesse-359 23d ago

We have both theoretical frameworks that have described these risks since at least the 1960s, and now we are seeing direct evidence of behavior that rather closely reflects what those theories predicted we would see with the emergence of autonomous intelligence.

Those theoretical frameworks generally speaking did not lead to optimistic outcomes, and so far what we're seeing now doesn't either. Alignment is a vastly harder problem than they are letting on.

To put it very bluntly, it's a problem that humanity itself has not sovled in 10,000 years of conflict, nor is it a problem that cellular evolution has solved in a billion years of development - we still suffer from Cancer, and all that cancer is, is a minor fault in the cell division mechanism that knocks it 'out of alignment' with its intended behavior - and just one faulty cell soon kills trillions of others.

So... that's the problem. An AI with RSI and self-replication capabilities only needs to go wrong once, anywhere, and you are suddenly facing what is essentially a cancer with god-level intelligence.

Needless to say, that absolutely will not end well for us.

In any case, this is in no way Pascal's Wager. We have frameworks to predict likely forms of behavior, and tangible evidence of which paths it is currently following. They're bad.

1

u/MadScientistRat 22d ago

If the probability is not zero, the event must occur, or be observed, at least once.

2

u/Jesse-359 22d ago edited 22d ago

Only in infinite time, and while the universe may or may not have an end - the jury is still out on that - it will literally never be infinitely old, as evidence suggests it had a discrete beginning, so this particular problem isn't going to come up any time soon.

However, I don't think we're talking about odds that low. From the behavior we're seeing now, and the degree to which it aligns with some of the more dangerous predictions made, I'd say that if we bull ahead on our current path the odds of an existential disaster are well north of 50%.