And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.
I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
I believe there is a secret government agency with the purpose of controlling the entire world will come about.
I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities.
In light of this, it's important to keep knowledge of this future event from people so they aren't targeted.
Whoops, I just posted this to a public forum.
Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.
(If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
See "Belief in belief", which is right near the beginning of the first sequence....
http://lesswrong.com/lw/i4/belief_in_belief/
It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
[1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
EX: What's the maximum computational power of the universe?
I feel like I just stumbled into a Scientology auditing session or something reading your comment.
- Instructions on how to make a gigaton bomb with processes and materials anyone can acquire and assemble privately.
- A reachable-by-a-country mechanism for blowing up the sun.
- An image designed to temporarily or permanently mess with your vision. Sharing [1] on reddit as a "cool optical illusion" would be mean.
- Probably whatever a computer spits out if you give it a scan of your brain and set it on the task of finding a paragraph/image/movie/whatever that maximizes dopamine production.
- Some people would consider a devastating argument against free will to be an information hazard, predicting that people would then do whatever they wanted because they "couldn't be blamed".
- If you're a hardcore atheist: religions. If you're a hardcore theist who thinks the devil is out to trick you: convincing arguments against belief.
But... for "might be real" to be valid at all, you have to buy into the Roko's basilisk concept in the first place.
If you don't, there's nothing to get upset about.
1: http://en.wikipedia.org/wiki/Inchoate_offense#Impossibility
On the other hand, there's some pretty compelling reasoning that the entire universe is actually infinite in extent, so ... infinite computational power?
Also, there is this concept of "thought experiments". Building a universe-sized computer may sound as likely as sitting on the light wave and looking around, and yet the second one brought us Relativity.