>It simply does not outside of the realm of science fiction.
Why?
Why?
But humans did make artificial flying machines.
I don't mean to be unkind, but your argument doesn't have close to the level of certainly that we should bet our species on.
No one knows where the roadblocks to AGI are, or what the timelines might be; but there is what seems like huge progress happening recently in an area which might eventually lead there. While no one knows the path, many intelligent people have thought about this without finding any theoretical roadblocks. Please don't publicly dismiss the concerns as 'science fiction' without a little more thought.
You're not getting it.
The effective argument of the danger seer is akin to "humans will learn to fly but then our cities will be covered with feathers" -
Humans have incentive to produce flexible, learning, language-understanding machines. Humans don't an incentive to produce a thing that will suddenly go rogue and decide to kill us.
It is entirely reasonable to argue that the tendency of humans themselves to go rogue and decide kill other humans comes from the evolutionary processes that produced humans rather than from the quality of human intelligence. We can see other relatively intelligent animals that live with in more (and in less) harmony than humans. No reason intelligence implies entity X won't fully cooperate with the people who intentionally engineered it.
I would argue that we human imagine that AGIs would have the same "downsides" as ourselves simply because we don't have any conception of intelligent things besides ourselves.
Maybe a carefully human-engineered intelligence would for reasons unknown turn into the things we see on movies. Maybe the next generation of supercolider will create an subatomic chain reaction that destroys the earth. But it's premature to worry about these particular hypothetical, especially given humans with ordinary AIs do pose problems.
So: Yes, we might all get along fine; ideally the more intelligent entity would be even more enlightened that we are; etc - those are all valid possibilities, and no one is saying they are implausible.
All people are saying is that some bad possibilities are also plausible (Eliezer Yudkowsky writes some great stuff about all the accidental ways we could screw up trying to program an AGI to be benevolent), and that, as the downside of the bad possibilities is potentially so great, then, in the absence of any argument why they definitely won't happen, we need to think carefully about them.
Some would say that, because the potential consequences of a bad outcome are so great, we need to be able to prove the bad outcomes won't happen before we build this.
In my opinion, Eliezer Yudkowsky and those who follow his argument are not intellectually credible and should dismissed with zero consideration.
In looking at this issue, I would note that Yudkowsky's (Bostrone's) ultimate argument is exactly homologous to Pascal's Wager. [1]
IE, since there's a hypothetical small probability that some entity [A Future AGI/Christian God/Flying Spaghetti Monster] could exist and the stakes for ignoring the possibility are "infinite" then we must "rationally" labor now to deal with possibility regardless of not understanding the mechanism involved in that entity.
And the thing is that modern science and modern statistics pretty hinge on ignoring, uh, "guff", demanding that extraordinary claims submit their extraordinary evidence and so-forth.
Further, the other fallacious argument pushed by Yudkowsky et al involves treating an AGI as a wish-granting Genie - human make overt logical requests and the AGIs "twist" these in a world-changing/destroying fashion. The problem is the over-request-format is identical to "logical specification" approach of gofai, the generally carded view that AI can be achieved with only the symbolic specification of how the world works (notably Yudkowsky and friend simultaneously acknowledge AI won't be based on symbolic specification/gofai and ground their argument on AGIs operating on explicit orders only, which boil down to symbolic specification. The Bostrone example is you tell the Genie "make me a thousand paper clips" and it turns the planet into paper clips "just to be sure").
[1] https://en.wikipedia.org/wiki/Pascal's_Wager
[2]https://en.wikipedia.org/wiki/Wish
[3]
I myself am on fence, on the one hand it is a bit like worrying about over population on mars, and there really is nothing we can productively do right now to research AI safety, the best thing we have are convolutional long/short term memory nets, and there's not much you can say about it.
But at the same time worrying is a great move if the alternative is to just wait until we can do something.
We haven't unintentionally produced a program that had a whole range of unexpected behaviors. An intelligence having human-like "survival instincts" and etc would be a big variation on a tool.
>In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior.
You mean that it's impossible to make a human level general intelligence ('HGI', to use your term), without understanding those things?
If so, I disagree with your assertion: it is not obvious that those things have to be done before a human level AGI could be made.
It might be a good idea to do so, but its not clear that they are a necessary requirement. Some people would say this is why AGI is dangerous.
Maybe there's a path to bootstrapping an AGI, where we build something from relatively simple bits we can understand, that then becomes very much more complex than we can understand.
This happens all the time: for example, we get very complex classification behaviour from deep nets: [simple pieces] + [lots of data] doing something more complex than we have explicitly trained them to.
If we were convinced that "In our journey to achieving HGI, we must first achieve an understanding of HGI ethics, motivations, and behavior" was a necessary thing to build before we could build a HGI, people would be less worried. But its not; and that's partially why people are worried. In fact, some of the people working on AI risk could be said to be basically racing to understand these things before other folk build an AGI.
It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
That may or may not be ethics as such but if something we make on purpose, we'd need to understand it. And considering it's difficulties, making it on purpose seems necessary, contrary to movies etc.
Instead, andreyf relied on it as a fact. I pointed out we can't assume that.
As such: its reasonable to fear AGI coming before we have that stuff sorted out.
Hence, as such: there is a real risk here, which the safety folks are trying to mitigate; hence they aren't worried about a threat scenario easily dismissable by andreyf's argument.
>It is entirely logical to argue that a high level intelligence could not be constructed without a broad and deep understanding of it's motivations and behaviors.
Ie. If you, or anyone had convincingly argued this, then it would be reasonable to dismiss (or at least greatly reduce) the concerns of the safety folk. But no one has, so its not reasonable to.
My claim is that I have seen no evidence that it is reasonable to fear a fictional self-aware autonomous AI (what I call HGI). The evidence people point to seems to have a very tenuous relationship with the reality of AI research.
Once we have even an inkling of an idea of how one might approach creating an artificial HGI, we can discuss how we can prevent it from being evil and taking over the world. Until there's actually a proposal for how one might construct one, though, any such discussion is worse than useless.
How can you talk about the risk of something when you literally have no idea what it is you are describing or how it's built, embodied with magical properties imagined by science fiction authors? To me, this makes about as much sense as talking about stopping aliens which could very well exist and could very well wipe out our civilization, i.e. great subject for wrapping up a party or going to the movies, but not exactly material for peer reviewed journals.
Feral: Yes - but no one provided any argument for that.
Me: I should rephrase, I sort-of do above but still, negation of "it requires understanding" is "AGI could happen without understanding, at random, in some fashion".
And I'd say that is the position that's no has provided to support for - again, akin to the worries that bring a giant supercollider online could destroy the earth. I mean, given that an AGI is an unknown, maybe bringing the supercollider online, another unknown, could create an AGI, which would then destroy the earth!
Plus, human beings have generally not succeed in producing complex engineering achievements by accidents. The atomic bomb required a massive engineering effort, heavier-than-air flight required considerable effort, alchemists never synthesized CPUs by baking minerals at random, even largest pieces of modern software have not threatened to "achieve consciousness" even as they malfunction repeatedly.
My argument is "consciousness by accident" is the extraordinary claim which requires extraordinary evidence. And AndreyF also adds - the dangers of ill-intentioned humans using AIs is here today, why worry about the hypothetical "getting out of control" problem when you humans who have historically inflicted vast amounts of misery on others of their species - oh, and consider several madmen fighting each other, that couldn't happen, not in Syria or whatever place one might name.
I'd be interested in your take on it.
More specifically, there are an infinite number of reasons for and against doing anything. Once we understand how humans weigh these reasons and choose between them and model them in an HGI system, we will be able to give it the values and morals that its creators choose.
"1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
"2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task."
The first of those is what Bostrom calls "perverse instantiation" and Dietterich and Horvitz call the "Sorcerer's Apprentice" problem (http://cacm.acm.org/magazines/2015/10/192386-rise-of-concern...). The second of these is what Bostrom calls "convergent instrumental goals" and Omohundro calls "basic AI drives."
The first of these seems like a fairly obvious problem, if we think AI systems will ever be trusted with making important decisions. Human goals are complicated, and even a superintelligent system that can easily learn about our goals won't necessarily acquire the goals thereby. So solving the AI problem doesn't get us a solution to the goal specification problem for free.
The second of these also has some intuitive force; https://intelligence.org/?p=12234 shows Omohundro's idea can be stated formally, so it's not purely sci-fi. Averting the "Sorcerer's Apprentice" problem in full generality would mean averting this problem, since we'd then simply be able to give AI systems the right goals and let them go wild. Absent that, if AI systems become much more cognitively capable than humans, we'll probably need to actively work on some approach that violates Omohundro's assumptions (and the assumptions of the formalism above). Bostrom and MIRI both talk about a lot of interesting ideas along these lines.
The first problem is not new. We have a similar problem with some corporations, for example.
"A sufficiently capable intelligent system" is as real as "sufficiently hostile aliens". It's hard to argue and reason about a fictional system with a assortment of properties picked by someone aiming to spreading fear.
I would say that the central concern is with notional systems that can form detailed, accurate models of the world and efficiently search through the space of policies that can be expected to produce a given outcome according to the model. This can be a recommender system that tells other agents what policies to adopt, or it can execute the policies itself.
If the search process through policies is sufficiently counter-intuitive and opaque to operator inspection, the "Sorcerer's Apprentice" problem becomes much more severe than it is in ordinary software. As the system becomes more capable, it can look increasingly safe and useful in its current context and yet remain brittle in the face of changes to itself and its environment. This is also where convergent instrumental goals become more concerning, because systems with imperfectly understood/designed policy selection criteria (introducing an element of randomness, from our perspective) seem likely to converge on adversarial policies due to the general fact of resource limitations.
There's no reason to think this kind of system is inevitable, but it's worth investigating how likely we are to be able to develop superhuman planning/decision agents, on what timescale, and whether there are any actions we could take in advance to make it possible to use such systems safely. At this point not enough research-hours have gone into this topic to justify any strong conclusions about whether we can (or can't) make much progress today.
http://givewell.org/labs/causes/ai-risk gives a good summary of this topic.