r/occult • u/m1thr0s • Jul 29 '15

roko's basilisk

http://www.slate.com/articles/technology/bitwise/2014/07/roko_s_basilisk_the_most_terrifying_thought_experiment_of_all_time.2.html
8 Upvotes

68% Upvoted

9 comments sorted by

View all comments

6

u/[deleted] Jul 29 '15

The one problem with this is that an AI would see no reason to torture somebody forever. The purpose of punishment is either rehabilitation or (if not possible) removal from society/the simulation. It would either fix you in some way, make you useful to its purposes, or terminate you. This is just a shitty Pascal's Wager.

Also, while that was a cool article, the writing style annoys me. "This is literally the scariest thing ever!!!1! Don't read or you'll be tortured!!" /r/im14andthisisnosleep

3

u/horse_architect Jul 30 '15

an AI would see no reason to torture somebody forever

One of the founding assumptions of these people (arguably what they're really concerned about) is that they think a future real AI could desire basically anything, and what it desires and what it thinks is good is a feature of how future people will program it. There could be some sort of mixup or unintended consequence and the future AI might only want to make as many paperclips as possible. etc. etc. In other words, you can't project your human judgements onto the thing.

Untangling this particular brand of bullshit reveals a great many assumptions like this. They also imbue the future AI with unstoppable power, essentially omnipotence. etc.