UC Berkeley Center for Human-Compatible AI
humancompatible.ai
humancompatible.ai
This is interesting, since I was on a panel with Russell a few weeks ago, and he argued that a "hard takeoff" scenario with sentient AI was a risk worth taking seriously. I guess he tones down these comments for the press.
If you have a full AGI, and it has the wrong value function, the scenario isn't Terminator, it's "you've already lost and have zero hope before you know what happened". Science fiction AI problems have absolutely no relation to real AGI problems.
For one thing, a real AGI would have to hit a very narrow target (nearly as hard as the one for success) to do anything understandably evil, warlike, or in any way recognizable to humans as an adversary; that would require understanding human value systems and then acting against them. The far broader target is simply that the AGI has a goal function, you're not part of the goal function, and you're made of matter that isn't optimally organized to meet the goal function.
And the reason why, despite that, AGI is not only worth contemplating but in fact critically important and valuable, is that the success path for AGI solves literally every single problem humans currently face, leaving only challenges like "does the universe have a finite amount of entropy to make use of?".
This assumes said creation is automatically at the top of the universal food chain. Given the prospect of other intelligent life before us, I wouldn't bet on it.
Ironic if humanity were to never create AGI, if only due to covert bounding by an existing intelligence.
That seems like a huge leap. You don't think anyone would notice anything at all? Or is this supposed to be a slippery slop argument?
The far broader target is simply that the AGI has a goal function, you're not part of the goal function, and you're made of matter that isn't optimally organized to meet the goal function.
The Borg? Again, no one would care we're being optimised away? No one would notice? Maybe it would just ignore us. Maybe it'd want to be our friend?
is that the success path for AGI solves literally every single problem humans currently face
This must be the bit that made the whole comment feel off. If AGI solves literally every single problem humans currently face, how can AGI be a problem? How are you resolving the contradiction here?
Very basically, it goes like this:
- the space of possible minds is very vast. We are animals that evolved in one set of circumstances, and so our minds (and thoughts we can think) occupy only a tiny corner of the space.
- an artificial AI would operate on computer-like timescales, not human like ones. Therefore we would have only hours to react before it got control of the physical world (hacking the internet, making nanothings, ordering and synthesizing DNA).
- an AI could (and would) recursively self-improve until it hit some kind of physical limit very far from our current capabilities. From a human standpoint, it would have God-like intelligence
- such an AI could model people and their motivations successfully enough to persuade them to do whatever it wanted
- because the space of minds is so vast, a generic AI would have goals that would be inscrutable and alien to us. To survive as a species, we would have to design them in from the beginning, in a way that survives all the levels of recursive self-redesign.
There are too many assumptions about the nature of intelligence to unpack here, but I believe it fairly describes what a lot of people believe about AI, thanks to people like Yudkowsky who have a flair for writing about it engagingly.
Claims of recursively reaching godlike intelligence seem particularly dubious to me. Why would AGI be superhuman at all? What if the hypothetical paperclip maximizer produces one paperclip and then dies out of exhaustion? Why would it recursively self-improve to a godlike status? Humans can recursively self-improve to an extent too and it doesn't happen to them. Why would it be particularly interested in recursive self improvement (this one seems to be imported straight from a particular human value system)?
I guess this skeptical way of thinking is part of reflexes of an engineer. When presented with the whitepaper of a database with miraculous consistency properties you don't immediately start to fantasize about the possible applications. Instead you ask "Can I play with it? Gimme the demo. Does it pass jepsen?"
What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement.
It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist.
Scio me nihil scire.
I absolutely agree that it's very hard to get to recursive self-improvement. And until you get there, certain classes of failure modes can't really happen.
(Others can, like the grey goo scenario; there are many people who consider carefully built AGI a good idea because the odds of some other existential threat arising approach 1 over time. There's a kind of Moore's law of existential threats: the intelligence and power required to destroy the world drops by some small multiplicative factor every 18 months. What happens when the equipment needed to destroy the world can be built in any well-equipped university lab? Or in a garage?)
The failure modes I'm talking about are that once you do hit recursive self-improvement, it's a lot easier to build something deadly than something safe, let alone something useful.
More like smarter grey-goo. The classic idea of a nanotechnological disaster ("grey goo") is building a tiny self-replicating robot that consumes matter to replicate. It's "smart" enough to disassemble matter to construct more of itself. Imagine something with similar "smarts", enough to work towards a much more complex goal ("build and expand into more compute nodes to compute faster"), but with absolutely no term in its value function related to humans. You're not its enemy, any more than a rock is its enemy; it doesn't have enemies, it just has matter not yet in the shape of more compute nodes. Your existence or non-existence isn't something it notices.
> Again, no one would care we're being optimised away? No one would notice?
You can notice something without having any ability to stop it. You can notice an explosion happening right next to you, right before you're consumed by it. You can notice everything dissolving into component atoms, but you can't do anything about it.
The science fiction scenario is "build an army of killer robots"; that's an understandable threat, and one you can imagine fighting against. The more likely scenario is "dissolve all adjacent matter into component atoms and rearrange it".
> Maybe it'd want to be our friend?
(Aside: my first reaction to this was to think of the whale from Hitchhiker's Guide to the Galaxy: "Ground! Ha! I wonder if it'll be friends with me? Hello, Ground!")
A recognizably human-like mind (let alone a concept of "friend") is a ludicrously tiny possibility, compared to being utterly inscrutable. As much as people like to anthropomorphize computers, an AGI isn't necessarily a "person" in any recognizable sense. The most likely kind of failed AGI wouldn't have a useful concept of "human", or "friend", or "life".
(There's a separate tier of failed AGI possibilities that does have a concept of "human" but an utterly broken value system; those are the ones that lead to the "worse than death" scenarios, rather than just the "destroyed by accident" scenarios.)
> If AGI solves literally every single problem humans currently face, how can AGI be a problem?
An AGI with the correct value system would. And an AGI with the correct value system is the most reliable protection against an AGI with the wrong one; by the nature of such a value system, it's unlikely that there'd ever be a second AGI, rather than just extensions of the first.
Personally, I think that AGI smarter (maybe much smarter) than humans is possible and even probable, but assuming that it will reach physical limits of intelligence and will do it quickly seems to require a leap of faith.
An AGI doesn't have to reach physical limits of intelligence (if there are any) to rapidly become far more intelligent than humans.
Since you asked about the argument for recursive self-improvement goes like this. Say Alice is a human engineer, who designs an AI (Bob) who is her intellectual equal. Then she can run Bob on faster hardware, and the now-smarter-than-her Bob can repeat the trick to design Clara, and so on. It's AIs all the way up.
If you're a certain kind of beardo, you imagine this process could be compressed into a few hours.
My suspicion is that GAI will turn out to be a big hot mess of interdependent systems and drives, just like every other brain. The paperclip maximizer will write songs about paperclips and argue with copies of itself about whether staplers are immoral.
The only differences between grey-goo and cockroaches is that cockroaches have come up against the limitations of the physical world. That is to say the physical world consists of minerals that have a strong tendency to not dissolve in to their constituant atoms, energy that is difficult to collect and use with any efficiency, and literally millions of other creatures eating your lunch.
It seems more plausible that this self replicating grey-goo would simply become cockroach food before it became a problem. Maybe ants would farm it like they do aphids and fungi.
My hunch is that the two assumptions will prove incompatible. Complex entities will have complex motivations, like people do.
A true AI born out of a paperclip maximizer would be more likely to spend its time writing hyper-symphonies about paperclips, and majoring in Clippy studies.
On a related theme, I've been kicking around an ontology of technological mechanisms,[1] one element of which is control systems. It's somewhat underappreciated that what made complex and/or highly-scaled systems possible in many cases were systems for either regulating or mitigating processes or discrepencies between parts of systems.
Swingletrees for draught teams, Watt's governor, for his steam engine (and additional tweaking to damp down oscillation tendencies), infrared temperature sensing and gas chromatography in steelmaking.
We figure out how to make a thing go, then we figure out how to make it go where we want.
Even high-speed rail was a bit more about quality control (rail and roadbed specification) and control (wheel truck hunting, pantograph-commutator interactions) than dumping gobs of energy into a rail-wheel system.
________________________________
Notes:
I agree with your point and think we will discover that intelligent systems are by their nature full of interlocking feedback loops with negative controls, and that this is a necessary prerequisite for a system to even be intelligent.
There are many neurons which are exitatory -- they respond when stimulated by others.
There are a large number which are inhibitory. They supress activity of other neurons.
I'm suspecting inhibition is more critical than excitation for consciousness and intelligence.
Even if this trick works, you have other problems, such as specifying what a human is.
Oh yes it would. How to exploit that fact into something that can reliably be carried out without misinterpretation however, is less clear.
Imagine you're just telling the AI to "do what humanity wants you to". Hmm, what do we want? How individual wills can translate into collective (dis)agreement? How drives in a single individual translates into will? Depending on how the machine answers all those questions, it might perform quite differently. How about perpetuating a cycle of pain an hatred, on the basis that many people have (sometimes mortal) grudges?
And even if you solve that, our current ideals may not be so ideal upon closer inspection (say, when our lives change and we take a second look). Would you want the AI freeze our current morality into an unwavering tradition? It's not clear it wouldn't.
Much work to be done.
Based on how the truly scary smart people I know behave, a GAI would spent much its time panicking about meta-GAI, and how to make it safe for AIs like itself.
My favorite part of this scary tale is the assumption
that a general AI would be highly motivated.
Don't think that an AGI would automatically destroy humans. We're still pretty efficient self replicants that can be useful, even for an AGI boss. Perhaps enhanced humans would even be able to keep up with the AGI. Imagine having an AGI design a human brain.
(Also, it pretty much predicted the state of NSA surveillance a year before Snowden thing happened, so there's that.)
I think AI true believers who invent such scenarios have been burned one too many times by journalists who (rightly) sensationalize their beliefs, and have learned to not talk about them to the press.
She told me that you basically need to lie. If your ideas contradict the narrative, or are complicated in any way, you need to come up with a baby version of those ideas that doesn't contradict any sacred journalistic beliefs. Even then you'll face an uphill battle.
There aren't many "sacred journalistic beliefs" beyond "don't take things at face value" and "coffee is necessary."
Also, "don't take things at face value" looks about as dead in modern journalism as their revenue streams.
Here's an example from a few years back. Journalists got a bee in their bonnet that Costco wasn't into MBAs. That must be the key to their success, relative to Walmart!
http://washingtonmonthly.com/2013/06/09/the-secret-of-costco...
The wonkish answer - fewer SKUs, less labor intensive process involving higher skill workers, more affluent customers - is unfortunately too boring to be picked up.
http://www.thedailybeast.com/articles/2012/11/26/why-can-t-w...
See also discussions of high frequency trading, basic income, algorithmic discrimination, and a variety of others. Algorithms wearing klan robes and voting Trump is a good story, but a stats 101 lecture explaining why linear regression doesn't do that is a very boring story.
Does anyone buy into this at all?
It seems inherently anti-intelligent to limit AI to what we think is smart. Are we going to make AI inherit our very limitations? That seems backward.
I don't see any objective value in artificially constraining progress as a mean to preserve humanity. However, if I did seek to preserve humanity, I would do so by coming up with a strong case for humanity's necessity, which I would make available to AIs.
Thinking of AI in terms of anything other than Aliens is a mistake.
If nothing else, he may be the best songwriter I personally know. ;)