The Doomsday Invention
newyorker.com
newyorker.com
I usually see arguments about harmful AGI based on the idea that AGI will be able to optimize itself and acquire computational resources to become powerful in a scary way.
I just sort of think that the human brain is already a tremendously powerful computational resource and augmenting it with weaker AI is going to be easier, cheaper, and ultimately, much powerful than AGI. But, of course that depends on the development of really powerful BCIs.
Why not develop AI systems that complement the already insanely sophisticated computational system of the human brain instead of trying to replace it? Or enable groups of humans to work together along with AI to solve really hard problems?
People often seem to think of AI as an arms race, but is it possible that systems that leverage human intelligence may have an insurmountable advantage?
The universe appears lifeless because it is a Dark Forest. http://www.goodreads.com/book/show/23168817-the-dark-forest
"Bostrom introduced the philosophical concept of 'existential risk' in 2002."
EDIT: I am unable to find the source. Might have been a halluzination, but I thought I remembered them having a discussion where Eliezer more or less still thought "let the AGI win, humanity be damned". Ah, my brain and I are not in a good relationship.
Oh, come on. This has been a theme in science fiction back to at least the 1940s. Jack Williamson's "With Folded Hands" (1947) is probably the clearest early writing on the subject.
"The car turned off the shining avenue, taking him back to the quiet splendor of his prison. His futile hands clenched and relaxed again, folded on his knees. There was nothing left to do."
That's a very funny short story. A magic story, though, not AI.
https://en.wikipedia.org/wiki/R.U.R.
And there is the Golem, which is even earlier.
Also i find it funny how much attention this theoretical problem receives, when we don't have a good explanations on what intelligence is and how it works.
I think that the Robot will not have such an in built desire - if it is based on reasoning then the machine might actually be more reasonable. In other words there is enough room for both Humans and Robots.
The point is that our gadgets become more powerful and more potent as weapons; so as we have a good history of past abuses it is sort of easy to extrapolate to the future.
What is still don't understand: would intelligence be an inhibiting factor - like more reasonable Humans are supposedly less destructive; Maybe the same goes for machines.
I don't buy the argument about competition for resources - With enough effort you can always stretch it so that there is enough for everybody/everything.
Another interesting aspect: once upon a time people would become very agitated when discussing politics (that was when we still had ideological differences and when people thought that their stance does matter); In our time we have discussions about sci-fi instead of that.
"The first registered thoughts on the possibility of artificial, machine intelligence, becoming a risk to humanity stems from the late Industrial Revolution. It was Samuel Butler, in 1863, in his Darwin Among the Machines that suggested machines could eventually replace humans as the dominant agents on Earth"
[1] https://books.google.com/ngrams/graph?content=existential+ri...
As the gentleman says, a lack of Autonomous Agents is not the thing standing between humanity and oblivion. The conspicuous surplus of super empowered agents (as in "I could leave this room and 80 million people would be dead 40 minutes later") is.
As for the Royal Society stuff. For the record it's not learning, and it's not the machines, I can say it again if anyone bothers to listen. Although given that it's not what will get news print or sell books or get on telly I am pretty sure that no one will.
Ho hum.
(edited 'cos I forgot a bracket, which was why I gave up c)
We can do both.
Lives saved versus dollars spent is a better metric. Having said that, the Singularity crowd tends to lump in all potential future lives into the saved category. The question of allocating resources to avoid existential threats very quickly turns into a Pascal's mugging in favor of AI safety (think of the trillions of future children).
> The threat of nuclear war is well known ... until very recently the same could not be said for the threat of unfriendly AI.
Enough nuclear weapons exist to kill all humans forever. Strong AI does not exist. Expert estimates range from decades to over a century. At the same time, the people making the most noise seem come from the side of avoiding existential threats and not AI research. I am (highly) skeptical they will produce anything of value. Let me rephrase. Would you expect any productive work in 1840 by philosophers concerning potential dooms day weapons yet to be created by physicists?
It has been suggested that humans recovered from a population bottleneck of less than 30,000 individuals (the Toba catastrophy). A nuclear war would leave billions alive.
Let's hope that the increased notice taken of AI, and fear thereof, doesn't suddenly shut down a major source of hope for the future.
Poking around a little, I also find this essay, which I think does a better job of getting to the root of the issue: http://edge.org/conversation/one-half-a-manifesto
Of course, just adding another person doesn't necessarily make things go faster (mythical man-month and all that). But then again, just adding more hardware isn't going to necessarily make a brain in a box more powerful, either.
The very nature of human intelligence is such that its architecture, scale and speed of computation are all bound within a relatively narrow range.
Now as I said, "adding another person doesn't necessarily make things go faster." It can, but not always. But what evidence do we have that a human level AI would be able to scale easily? That seems to be a pretty huge assumption that gets glossed over.
The fact it would probably be software, executing on a digital platform. These things typically lend themselves to optimization and scaling.
Seems to me that the chance we'll have just enough computing power to run human-level AI—without sufficient resources to go beyond that—is a small one. It is far more likely that we will either have too little, or more than ample computing power available.
What evidence do you have that human-level AI won't scale easily?
The best argument I can think of in your favor would be prohibitive computational complexity requirements, but the aforementioned dynamic still applies—it's unlikely that we'd have only just enough computing power to achieve human-level and not go beyond that.
And that's just talking about a raw increase in computing power. We don't have any idea whether a human level intelligence program would even be able to put increased processing power to good use (and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections).
Simulating the brain isn't necessarily the path to human-level AGI though. Simulating biology in digital form tends to be inefficient.
Modern machine learning scales easily, or at least relatively easy.
>(and it could theoretically be detrimental; look at what happens to humans who have too many neuronal connections)
Perhaps a constraint of our brain's cognitive architecture and not intrinsic to cognition itself. Still, interesting point.
With an AI, you just dump that money directly into the brain by slapping on more compute power and storage. It'd be a direct increase in ability to its will. I'm not sure what exactly the raw costs would be (maybe it would still cost more than hiring a human), but at least it seems like it'd be a much more efficient use of each dollar.
It seems likely that the first simulated human level intelligences, at whatever point in the future they pop up, will at first be extremely costly and extremely slow. We don't really know if, say, connecting two supercomputers two simulate twice as many virtual neurons would make it more effective or if it would make it too unstable to do anything (how would a human brain with twice as many neurons react? It might just be mentally unstable and unproductive). But even a simple doubling experiment like that would probably take a long time to plan out and realize.
Now, if that intelligence also far exceeds our own in certain aspects but is not empathetic enough, and if it were to have a keen understanding of how to control humans, and if it were much more able to quickly control and adapt software and hardware, then it would become a much bigger risk.
But, you're right- in theory, there could be a natural intelligence that could be the same level of risk as an artificial one. Most of us, including myself, are just not aware of one.
That's a thing I don't care for about most science-fiction. The AI entity can just magically take over every technology it's connected to, because magic. No concern about the computational complexity of breaking the code-signing certificates on the affected computers. It's like everything on the Internet of Things is hopelessly and completely insecure!
... okay now that I type that out, maybe that's more realistic than I gave it credit for after all ... :b
Of course there are other reasons why having everything be "connected to the internet" (light switches, refrigerators, nuclear warheads) is probably not the wisest idea... would an AI with bad intentions really be any more harmful then a human being with bad intentions who had access to every interconnected thing?
https://news.ycombinator.com/newsfaq.html
Btw, we're working on a better approach to duplicate detection that will reduce the number of reposts and will more often give credit to the original submitter.
A bomb is dumb, its explosion just raw chemistry. I would argue we're like small children playing with a sleeping parent: when it finally wakes up, it's going to have its own agenda for us.
If we're playing this game, then humanity is more like a small child trying to wake up Cthulhu in hopes that he'll play nice.
We build machines using a highly human-dependent artificial selection process. That's not going to change because the number one feature being selected for is usefulness to humans. I'm confused about how that's going to result in an even modestly-powerful, fully integrated machine that is not demonstrably useful to us.
That's a bit like saying we should reconsider having sex because our babies might evolve grenades for hands, and that would be suicidal.
I'm completely failing to understand why a hypothetical god-AI is being taken seriously as a threat to survival, but the many potential and actual horrors of the human world are being ignored here.
We're perfectly capable of destroying ourselves without needing to invent an AI to do it for us. Why is this not taken seriously as a problem?
I've been reading Hannu Rajaniemi lately. He has in some ways a trippier take on this stuff than even Peter Watts does.