Eliezer Yudkowsky: That Alien Message

re: the AI-Boxing experiment, Arguably a human can take over a human mind: http://en.wikipedia.org/wiki/Neuro-linguistic_programming

Just because it's interesting, Eliezer mentioned later that he'd done the AI-box a couple more times and failed; I think his final record was 3 successes in 5 attempts.

http://lesswrong.com/lw/up/shut_up_and_do_the_impossible/

"Alien Message" reminds me of Robert E. Vardeman's "Weapons of Chaos" trilogy: http://www.fantasticfiction.co.uk/v/robert-e-vardeman/

Just as Yudkowsky's short story, those Vardeman novels include advanced aliens living on a much slower time-scale (and much larger space-scale) than we. Man's accidental incursion into their domain triggers a slow-scale attack response that Mankind doesn't at first recognize as intelligent.

"That Alien Message" read like a science fiction short story at first, and then delivered a number of abnormally deep insights. Ideas so new (or ways of seeing so new) get to my brain 5-6 times a year top.

"Twelve virtues" however... well, I started skimming at the fourth virtue. Most of the content was a restatement of things I already agree with.

Maybe your audience (HN, Reddit) is enough into rationality to care more about very rational discussions (like the AI-Box problem) than about discussions of rationality itself (like "Twelve Virtues").

What interested me about this essay:

1. I had not seriously considered the question of what happens when a "transhuman" AI appears (yes casually, not not in depth).

2. The ramifications were likely within my lifetime (I think Google has developing an AI as its explicit aim http://tinyurl.com/yald3f3), personally relevant, and non-trivial.

3. The answer is not immediately clear in a world in which technology makes it increasingly likely someone will break a moratorium and build such an AI (unless we are to accept our fate and either ride the singularity somehow or be marginalized).

You have something something people care a lot about. Reminds me of a "minimum viable product." If you haven't already - do a popular book, a tour, rinse and repeat. Let Rudy Rucker be your guide.

I can tell you exactly why "That Alien Message" gets more attention: because it's a story, and a cracklin' good one. Human brains are hard-wired to be attracted to good stories. This is why mythology is so powerful.

BTW, there is a pretty well worked out theory of what makes a good story. Geeks tend to pooh-pooh this theory because it's associated with the "squishy" field of stage and screen drama. IMHO this is a serious mistake. The world today really needs a better mythology. Those in its current inventory are getting obsolete. Some of them are becoming actively harmful. Coming up with a good mythology is not so easy. The last person I know of who did a notable job was L. Ron Hubbard (at least, he's the last one who did it intentionally). Imagine how much better off the world would be if Hubbard hadn't been a psychopath. I hope some people reading this might start devoting a little time to thinking about this problem.

It's more entertaining

Does it worry you that you are working on a challenging 'prediction' problem (anticipating and constraining the behavior of an AI) while unable to even make accurate guesses about human behavior?

Normative writing as wide ranging as 'The Virtues' is quasi-religous and an all or nothing proposition. People are not receptive to having their values changed in such a direct manner.

Clearly you need to present more of your work as scifi :)

As scifi incorporates more and more transhuman elements, these concepts become less radical.

I recently read the decently-written novel 'Darwinia' which spoiler takes a hard 90-degree turn partway into the story revealing the protagonist's world is really a simulation/recording, running around the time of universe's heat death, using "distortion lines in the Higgs field" as the substrate on which the simulation runs. end spoiler

I think an important lesson is that you can't always design something to be viral (look at how fickle YouTube users are)

Minksy is no dummy, but his ideas about AI and cognition are dated (though I admit it's been a while since I've read anything he's had to say on the subject).

Normally I'm opposed to dupes, but every once in a while there's something that everybody should see.

This probably falls into that category, if you couple it with the AI-Boxing experiment (previously mentioned in this thread). The results of that experiment are so profound and disturbing that everybody should have some exposure.

Third time here as well, and yet I must report that I enjoyed it just as much the third time as the first two! (that is, a lot)

(Just to offer a piece of counter-evidence)

I agree that dupes can be annoying, but even after being on HN for a fairly long time (almost 2 years) and visiting a little too often, this is the first time I have seen this piece of writing and I found it very worthwhile. For some odd reason, I had never really read fully about Overcoming Bias or the new Less Wrong blog and wiki with all that it has to offer and I think I will spend a lot of time reading that in the next few weeks.

Thus, thank you for the dupe.

What is it that you think he's paranoid about?

Have you read anything else he's written? If not, you should. It's good stuff.