Hacker News Reader

With respect to 4: probably the funniest way I've ever seen vacation homes ever painted.

Presumably it goes Top AI -> Successful and wealthy -> has vacation home in quiet, rural area.

The author seems to paint it as Top AI -> has a rogue AI bunker

> Talking about systems millions of times more intelligent > than a human is nonsense unless you can define what it is > you're talking about.

You can only define things to a certain level. No matter the topic, you could always complain about a lack of definition. xkcd.com/309 comes to mind.

Expecting computers to be millions of times faster then humans has some appeal for 2 reasons.

1) Machines are already millions of times more powerful then humans in some tasks. For example mail delivery is now done with the speed of light which travels millions of times faster then a human.

2) Would we consider a million people to be a million times more intelligent than one single person? In some sense, I think so. So when we can clone a brain and we clone it a million times - voila.

3. Be wary of people trying to sell you ideas based on fear. It's usually snake oil, concealing some other agenda - such as trying to obtain or maintain grant money for projects.

That's very true, but I don't think it completely obliterates the relevance of the point being made in TFA. The truth is, we don't know exactly what a trans-human AI will be capable of, and whether or not it would do (or attempt to do) things that would be harmful to humanity. Now I'm certainly not arguing for stopping AI research! But maintaining awareness of this issue and reminding people to keep the unintended consequences in mind, is probably a valuable thing.

Don't get me wrong... I don't put a lot of stock in "fear of AI" (Matrix and Terminator jokes aside), so long as the AIs are locked in "the box" (to use Yudkowsky's term) and one can simply power the machine off and end the AI. Despite the (seeming) fact that Yudkowsky has talked his way out of "the box" a few times, playing the role of the AI, I think that we're a LONG way from an AI that can "go autonomous", talk it's way out of "the box" and become destructive. The scenario strikes me as highly unlikely, but just on the borderline of plausible enough to keep an eye on.

Regarding #4 specifically. An autonomous runaway recursively improving AI would make things like bunkers irrelevant. It might buy you days at most.

With respect to 3 the secret agenda looks like to have been expressed in these lines:

"This spring Hollywood will weigh in with Transcendence, starring Johnny Depp as an AI researcher targeted by violent extremists who think we’re crossing a Rubicon that will be a disaster for humanity."

Remember how the world was about to be finished on 21Dec2012 and and the Hollywood movie associated with the fad?

I think the best definition of intelligence is something like: "the ability of an agent to maximize its rewards by taking actions in some environment." In this sense, you could quantify intelligence by the agent's total reward over some time window, or how fast an agent learned the optimal strategy.

The bug out houses are so they can hide from the people who gave them money. (this book touches on the history http://aima.cs.berkeley.edu/)

I think points #3 and #5 are the two biggest ones. Fear, Uncertainty and Doubt is a well-worn sales tactic, and predictors of AI advances have generally been on the wrong side of history.

All they need is to be smart and strong enough to take control of the dense energy sources (oil, coal, uranium). Then we're toast.

No need to be millions of times smarter, whatever that may mean.

We have more-than-human entities controlling our energy resources. They are called corporations, and some even argue they are pretty much already out of our control.

Corporations are still human-oriented.

Another scary thing is the combination of the internet of things that is currently taking shape, and the fact that a lot of computer systems are backdoored by governments.

Nice attack vector for smart software (not counting accidental vulnerabilities).

To add to your counterpoints:

6. While it is about sure that in the future they will be technology that is for indistinguishable from magic, this does not imply that every magical thing we can think of will be true at some point in the future.

7. The whole premise rests on recursively improving AI. The human race has been making tool to help make them better tools for at least 50,000 years (lower bound to be uncontroversial). We do have amazing tools now, and tools that automatically do other tools, but we have not witnessed an accelerated auto-recursive thingamagig.

>We do have amazing tools now, and tools that automatically do other tools, but we have not witnessed an accelerated auto-recursive thingamagig.

I've never seen liverwurst icecream. I don't have to taste it to know with 99% certainty that it's a bad idea.

On a more serious note, I've never strapped 500 roman candles on my back and lit them all at once in an attempt at making a "jetpack". Again, I don't need to perform the experiment to know I would likely receive burns and possibly die.

The remarkable thing about our brains is that we have the ability to predict what will happen in our heads, empathize with the actor in that scenario, and then come to a conclusion about if we want that scenario to play out. So, to apply this to your counterpoint, I don't have to see an accelerated auto-recursive thingamajig to know it is likely going to be impossible to control and a bad idea.