Hacker News Reader

Comment by MBlume | Hacker News Reader

I don't mean to say "he made a Rick and Morty reference, Serious Adults don't make Rick and Morty references". That'd be dumb. I mean that he points to a robot in Rick and Morty that was sad about being programmed to pass butter, and attempts to infer from this that real robots would refuse to intelligently pursue arbitrary goals, and therefore that the orthogonality thesis is false. He's treating a thing that happened in fiction as though it was a real thing that happened somewhere that we should treat as evidence. That doesn't work.

View on HN

The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.

> it might be a defining feature of intelligence.

But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.

But the orthogonality thesis isn't just some idea that may or not be true. It follows directly from Hume's Guillotine, discovered in 1739: no amount of superintelligent reasoning about facts will allow you to derive goals.

No artificial intelligence will ever spontaneously develop morals, because the questions "what can I do" and "what should I do" are eternally separated by Hume's guillotine.

The motivations will always have to be provided by the people who make the machine, and we have been seen in the past that artificial intelligences are very good at finding loopholes in their moral code. Here are some real examples: https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa...

The problem is that the thesis cannot be disproved by one counter example (real or not), because the existential hazard comes from the possibility of just one such instance of an AI.

He's treating a thing that happened in fiction

I think in this case it might be you who is mistreating what's written. This also produces the same not-working result but for somewhat different reasons.

Reply on news.ycombinator.com