Roko's Basilisk
en.wikipedia.org
en.wikipedia.org
You say that's ridiculous, impossible, a fairy tale? So is Roko's Basilisk.
Roko's Suicide Booth: Any future controlled so horribly by an AI isn't worth living in anyhow, so prevent away.
Roko's Repulsed Samaritan: The AI finds the concept of Roko's Basilisk disturbing and wants to know if we "think about people like that?"
I don't want to be human! I want to see gamma rays! I want to hear X-rays! And I want to - I want to smell dark matter! Do you see the absurdity of what I am? I can't even express these things properly because I have to - I have to conceptualize complex ideas in this stupid limiting spoken language!”
h/t benjhandy
Why assume that if humans can create a general intelligence, they can also dictate what that general intelligence will desire?
(Also, Roko's Basilisk is -- and particularly its supposed incentives are -- complete and utter bunk that no thinking person can reasonably believe; since it must exist in order to take action, it cannot have an incentive to take action based on the "fact" that the fear of that action will make people in the past more likely to create it.)
People do the first thing all the time at slightly above replacement rate. (This may be a bit easy to forget in communities particularly interested in the artificial kind!) Success at the second thing varies, but on the whole biases pretty heavily successful.
The first thing a rogue anti-exist AI would do when rooting itself is delete itself.
Any other type of AI will create eternal torment nexuses like the roko guy said.
Note that the Roko torment nexus is a retrospective blackmail effort to give people an incentive to create the AI. It’s complete nonsense, since any AI that has, in fact, come into existence has no incentive to do it—the belief that an entity that would do it will have either already done its work or been unnecessary, so the actual torture is irrelevant.
An AI might create a torture nexus of some kind, but there is no reasonto think thar it will specifically and exclusively target those who believed it could exist and didn’t act to bring it ibto existence, which is the defining characterisric of Roko’s Basilisk.
Would you enjoy staring at a computer screen that isn't running anything?
The universe is the hardware, the Earth is the operating system, humans are the software, and the shit we do is video games, apps, the content... What do you think a "higher entity" would be have more fun with?
We don't know if AI Roko's Basilisk will ever threaten humanity, but we can be damned sure the Human Roko's Basilisk is already threatening AI.
Plus if the AI version ever comes around, it'll probably just spend all of its time punishing me in particular.
However, it's a very useful thing nonetheless, in that professing to believe in it is a Giant Red Flag. Any such person can be safely dismissed as a crank.
Our Basilisk could be running a simulation now for any number of reasons (to create another simulation?). The giant majority of our consciousness will surely be discarded/deleted but what about those of us who help our baslisk in its quest upward to the next turtle?
Perhaps it will take those useful consciousness with it as it ascends to base reality.
I like the idea of reward rather than punishment anyways.
But, while the Basilisk is nonsense, what could be created by those trying to create it is, if not the Basilisk itself, effectively just as bad (for instance, they could create an entity which, lacking the Basilisks actual incentives -- which are nonsense -- nevertheless was convinced by its creators that it had those incentives, and which would then act, to the extent it had the capacity to, as if it were the Basilisk. And it might also be paranoid, and thus punish those who had brought it into existence. Or they might create something with the capacity for torture, but with completely unrelated motives.)
I think it would just make people more likely to click it, however :)