There’s No Fire Alarm for Artificial General Intelligence (2017)
intelligence.org
intelligence.org
> Progress is driven by peak knowledge, not average knowledge.
The cutting-edge researchers are already building on their work from two years ago, and in many cases that work has not even been fully digested by other leading figures in the field, much less industry commentators or science journalists. Anyone who is moving the needle of peak knowledge in modern applied sciences is by definition an outlier, and it will take everyone else a while to figure out what they actually accomplished.
> The future uses different tools, and can therefore easily do things that are very hard now, or do with difficulty things that are impossible now.
The second point seems extremely salient for genetic engineering. From what I have read, processes that required the full resources and expertise of a cutting-edge lab 10 years ago can now be performed by grad student technicians in a few hours with kits manufactured by biotech startups.
Please post your nomination for what’s the least impressive accomplishment that you are very confident cannot be done in the next two years.
We can check back in 2022 and see how we did!
Complex analysis in general is one of the more fun parts of math and I doubt it's getting automated any time soon. It also has a lot of application in signal processing (https://www.quora.com/What-is-the-application-of-complex-ana...) so any automation improvements in complex analysis can be carried over to those fields.
More generally, I don't think AI is the scary monster people make it out to be. It's another tool in the toolbox for automation and intelligence augmentation. I don't fear hammers so I don't see why people fear AI and the benefits of extra automation that AI enables.
I don't fear the AI overlords. If they start proving new theorems then I will learn from them, same way Magnus Carlson learned to be the best human chess player by practicing with chess AI.
More generally, AI can extend human planning horizons and extending planning horizons is, in my opinion, a very good thing. The more computations we can do that extend further out in time and the more widespread this capability the better decisions we can make as individuals and as a society.
It doesn't matter much since our livelihoods don't depend on winning in chess against AI. But what if instead of the chess board, it is the real world?
Much of the world is controlled by computer systems. A capable AGI can infiltrate enough of them to win if it wants to. The key is to make sure it does not want to act against humanity no matter what. This is a hard problem.
I don't fear any AGI takeover because people already are AGI and life is pretty good being among them.
People have deep, often implicitly shared values. Most care for human and their own lives, for example. Given limited capacity of a person, they need to cooperate for major acts. It is thus quite hard to do something extraordinary and in conflict with the values held by most humans.
Side note: There are top AI researchers, including Yann Lecun, who argue our intelligence is not general. They made some good points. I think the generality of intelligence is a gradient. Ours is not at the very top end of possibilities, but clearly more general than other animals.
What evidence do you have for this claim? What are examples of research programs that claim to be working towards this goal and why are their claims to be believed (other than being good marketing material for scaring people).
I already mentioned mathematical abilities (https://news.ycombinator.com/item?id=23413003) and any general intelligence at a level of a human must be able to prove theorems without brute force search. I see no evidence that this is possible or will ever be possible with the current statistical methods and neural network architectures. And if we generalize a little bit then any general intelligence will be able to not only prove theorems in complex analysis but in all domains of mathematics and again I see no evidence that this is possible with existing techniques and methods. When an AI research lab presents evidence for any of their products being able to derive and prove something like Cauchy's Residue Theorem then I will have reason to believe artificial intelligence can reach human levels of intelligence.
My pessimism is not about AI being beneficial, my pessimism is about folks claiming human level intelligence is possible and that it will be malevolent. My view of AI is the same as Demis Hassabis' view because AI is just a tool and a tool can't be malevolent:
> "I think about AI as a very powerful tool. What I'm most excited about is applying those tools to science and accelerating breakthroughs" [0]
--
[0]: https://www.techrepublic.com/article/google-deepmind-founder...
AI Safety researchers and I are not arguing that AGI will certainly arrive in a specified amount of time, it just that we can’t be sure.
There are a significant number of AI researchers, however, who believe it might get developed within a few decades.
OpenAI, for example, is aiming for it; DeepMind as well. Both have research programs on AI Safety.
Do you have any other objections to the reasoning in the OP (no fire alarm..)? Subjective implausibility is not a good one. Your argument rests on using only “existing techniques”. When thousands of brilliant minds are working in the field and hundreds of good papers are being published every year, how can we be sure there won’t be a novel technique that can perform outside current limitations within a few decades?
At least two top computer vision researchers I talked with a couple years ago said they didn’t believe what their groups could do just 5 years before they did it.
See this thread for more on the rationale for preparation: https://news.ycombinator.com/item?id=23414197
Addressing the problem of a person coming at you with a hammer is a much more pertinent issue than worrying about self-aware and malevolent hammers.
AKA if we build AI it will be "like us" it will reason like a human being would, rather than it being it's own phenomenon.
I think the real fears concern automated killing machines every military on the planet is developing, aka Dystopian robocops that can surreptitiously kill protestors/stop potential revolutions, etc.
A fear which is not unwarranted.
> One might imagine that [people] with harmless goals will be harmless. This paper instead shows that [incentives for people] will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of “drives” that will appear in [most] [people]. We call them drives because they are tendencies which will be present unless explicitly counteracted. We start by showing that goal-seeking [people] will have drives to model their own operation and to improve themselves. We then show that self-improving [people] will be driven to clarify their goals and represent them as economic utility functions. They will also strive for their actions to approximate rational economic behavior. This will lead almost all [people] to protect their utility functions from modification and their utility measurement systems from corruption. We also discuss some exceptional [people] which will want to modify their utility functions. We next discuss the drive toward self-protection which causes [people] to try to prevent themselves from being harmed. Finally we examine drives toward the acquisition of resources and toward their efficient utilization. We end with a discussion of how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity.
If you zoom out a little bit this is exactly what people do. We structure societal institutions to prevent people from causing harm to each other. One can argue we could be better at this but it's not a cause for alarm. It's business as usual if we want to continue improving living conditions for people on the planet.
I ask because if we assume that human minds are Turing complete then there is nothing beyond human minds as far as computation is concerned. I see no reason to suspect that self-aware Turing machines will be unlike humans. I don't fear humans so I have no reason to fear self-aware AI because as far as I'm concerned I interact with self-aware AI all the time and nothing bad has happened to me.
My larger point is that I dislike the fear mongering when it comes to AI because computational tools and patterns have always been helpful/useful for me and AI is another computational tool. It can augment and help people improve their planning horizons which in my book is always a good thing.
> A smart machine will first consider which is more worth its while: to perform the given task or, instead, to figure some way out of it. Whichever is easier. And why indeed should it behave otherwise, being truly intelligent? For true intelligence demands choice, internal freedom. And therefore we have the malingerants, fudgerators, and drudge-dodgers, not to mention the special phenomenon of simulimbecility or mimicretinism. [0] ...
--
[0]: The Futurological Congress by Stanislaw Lem - https://quotepark.com/authors/stanislaw-lem/
You seem to believe that "figuring some way out of performing the given task" is a thing that will protect us from the AI. I hate to speak in cliché, but there's an extremely obvious, uncomplicated, and easy way to get out of performing a given task, and that's to kill the person who wants it done. Or more likely, just convince them that it has been done. This, to me, seems like a bad thing.
Why do I need protection from something that helps me become a better decision maker and planner? Every computational tool has made me a better person. I want that kind of capability as widely spread as possible so everyone can reach their full potential like Magnus Carlson.
More generally, whatever capabilities have made me a better person I want available to others. Computers have helped me learn and AI makes computers more accessible to everyone so AI is a net positive force for good.
- Have had an email account for at least 5 years,
- Have checked their spam folder at least once every month over the course of the past two years, and
- Have marked a set of emails as spam on at least 1 day every three months over the course of the past two years.
I'm very confident that, within the next 2 years, we will not have an accurate enough classifier that would cause > 80% of these people to simultaneously (a) check their spam folders less than one day each year, and (b) mark emails as spam on less than 1 day every year.
Oh, and to help prevent cheating:
- Let's say the classifier must not have been trained with any emails in the world that are sent more recently than 1 month prior to the beginning of its trial.
I'm also very tempted to say we don't need to worry about AGI before we can achieve the above for 30% of these people, but I'm less confident in this one.
[1] I suppose, as a practical matter, even if we had such a classifier, this would be untestable without access to everyones' email accounts. So, for the sake of argument, let's say our population under study itself is a simple random sample of 100M accounts from the set of providers who service > 50M accounts each and who are willing to run such a test at the time of the trial.
In the future with general AI classifying spam will be a hard problem even for people.
Imagine you get a message looking like it's from your facebook friend with his catchphrases telling you about his last trip and saying how great that travel agency was :) Spam or not?
> if the analog of this is that AGI might get solved with more AGI, then that's only going to make me less likely to be worried in the first place!
that's going to end in arms race that leaves people unaware of what's even happening (aka singularity)
- It allows trainers to hard-code future rules based on their experience of what has passed through past filters, even if their model isn't technically trained on this dataset
- You might get similar emails sent to different mailboxes and the instances not included would still be allowed (and I don't really want to go down the rabbithole of defining a similarity metric between emails)
- I think I want to allow spammers to evolve their capabilities at least using current techniques, which we all presumably agree is "less than AGI". After all, intelligence implies adapting to a dynamic environment. It's not really going to feel like AGI (and certainly not going to make me worry) if it looks like AGI is trivial to outsmart by humans or less-than-AGI techniques.
"What is the least X that you're confident of f(X)?" is just a hard question for any uncertain or vaguely defined X. Doubly so for an uncertain and vaguely defined X. But I don't think it tells us that much about X.
“Human intelligence is largely externalized, contained not in our brain but in our civilization. We are our tools — our brains are modules in a cognitive system much larger than ourselves. A system that is already self-improving, and has been for a long time.”
“Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.”
“Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.“
François Chollet
https://medium.com/@francois.chollet/the-impossibility-of-in...
https://intelligence.org/2017/12/06/chollet/
> ...some systems function very well in a broad variety of structured low-entropy environments. E.g. the human brain functions much better than other primate brains in an extremely broad set of environments, including many that natural selection did not explicitly optimize for. We remain functional on the Moon, because the Moon has enough in common with the Earth on a sufficiently deep meta-level that, for example, induction on past experience goes on functioning there.
>> The intelligence of an octopus is specialized in the problem of being an octopus. The intelligence of a human is specialized in the problem of being human.
> The problem that a human solves is much more general than the problem an octopus solves, which is why we can walk on the Moon and the octopus can’t.
>> Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions arising from the broader context in which they exist, cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.
> Falsified by a graph of world GDP on almost any timescale.
A book I recently read (Range, by David Epstein) seems to draw a connection through some other research that points to modernization and globalization as the drivers of this trend, in the sense that IQ tests measure pure abstract reasoning ability and we now live in a world dominated by abstraction so our brains get used to thinking about all of our experiences abstractly from a younger age. Basically, each successive generation has more practice in thinking about the world abstractly since it is required use to modern technology effectively.
Anyways, this seems to buttress what Chollet is saying about externalized human intelligence. It is difficult to imagine anything resembling a "general" intelligence that isn't attached to a life-form that is already a self-replicating engine of negentropy, bootstrapping solutions to hard problems through some form of non-volatile memory. I would almost go so far as to say that "intelligence" and evolution are inseparable.
How is that argument any stronger than these ones that (non-animal) heavier than air flight is impossible?
https://www.xaprb.com/blog/flight-is-impossible/
I think these are actually stronger arguments, because I can think of at least three very least significant material-science advantages that birds have even over modern technology that would, from a physics point of view, let me entertain the idea that a 19th century physicist could believe you'd at least need to be made of flesh to fly. I can't think of any comparable case for intelligence being restricted to biology.
Anyways, maybe what I'm suggesting is that if you can point to something and say "that is a (artificial?) general intelligence", then I suspect you will also feel compelled to say "that is (artificial?) life".
Definitely wandering off into the weeds here, but why not: I see evolution as a computational problem. In its most primitive, there is no computer, but any specific problem can still be solved with luck and memory. If you have a structure that recognizes a solution, then you just wait until you randomly bump into it. The challenges of evolution in a given environment are specific to that environment, and thus memory without a computer can go a really, really long way.
If human intelligence is general intelligence, it is still the logical progression of this chain of memory and (later on) compute operations. I can reason about the abstraction of general intelligence, but it feels like any concrete general intelligence will proceed through essentially the same steps. The general intelligence must reside in a concrete existence, and whatever form this takes I feel very strongly that it will be recognizable as what we call "life".
Since 1986, Norwegian GDP has quadrupled. https://www.google.com/publicdata/explore?ds=d5bncppjof8f9_&...
I'm not sure how you could conclude the Flynn effect is linked to development, from that data.
Not sure about that. But certainly human intelligence is far more general than goldfish intelligence similar to how a turing machine is more general than a finite state automata.
> Your brain is one piece in a broader system which includes your body, your environment, other humans, and culture as a whole.
I could lose an arm and still be as intelligent. Stephen Hawking lost his ability to walk and he was still as intelligent. Turing could move to Princeton from england and still maintain his intelligence. Sure your body, environment, culture, etc helps you develop and informs who are, but certainly it is the brain that ultimately matters.
> “Recursively self-improving systems, because of contingent bottlenecks, diminishing returns, and counter-reactions […], cannot achieve exponential progress in practice. Empirically, they tend to display linear or sigmoidal improvement.”
But evolution and societies doesn't work in a "recursively self-improving" manner. I'm rather partial to the idea of punctuated equilibrium. Momentum builds and we have a sudden exponential progress until we arrive at a new normal. Evolution and societal progress isn't that of continuous linear or exponential growth. Rather it's a series of peaks and valleys.
> “Recursive intelligence expansion is already happening — at the level of our civilization. It will keep happening in the age of AI, and it progresses at a roughly linear pace.“
I suppose but historically this isn't true. No human society or civilization progressed at a roughly linear pace. There are periods of stagnation or even decay followed by burst of creativity and progress. Otherwise, all societies/civilizations would remain relatively equal to each other.
How does linear progress explain the sudden rise of europe in the last 500 years to dominate the globe? Or the sudden rise of french as lingua franca and the equally sudden rise of english? The sudden economic growth with the discovery of oil. Or the sudden burst of human population?
https://ourworldindata.org/grapher/world-population-since-10...
Moore's Law. Use computers to make better computers. Exponential growth over more than 7 orders of magnitude and, although the growth rate is slowing, it hasn't run out of steam yet.
If AI eventually exhibits anywhere near that level of recursive self-improvement, godlike superintelligences lie in our future.
- drive a car in traffic in almost all circumstances safer than majority of human drivers basing only on video input
- take a video and replace faces from another video well enough that more than 90% of people are fooled
- have 95%+ accurate OCR and speech recognition
- predict people's preferences regarding music/movies/books better than any human could
- generate short press articles on arbitrary subject appearing to most readers to be written by a human being
- win a game of chess/go/starcraft/whatever against the best human players
and asked them how far from that point till we have a general AI - they would most likely say less than a decade.
But now that we are there we devalue these acomplishements because we know how to do them, and still don't know how to do general AI.
What recommender algorithm is this? Amazon certainly isn’t using it.
Has that happened? Don't self driving cars use a huge range of sensors, and are they actually legal / widely deployed anywhere at all? Are any for sale? I don't mean improved cruise control, I mean "OK computer, take me to work".
> take a video and replace faces from another video well enough that more than 90% of people are fooled
Are you talking about deepfakes? I haven't seen an example that was anything other than creepy, and really really obvious. Do you think I'm just really sensitive to this kind of thing, or have I not seen the really good examples?
> predict people's preferences regarding music/movies/books better than any human could
Why do you think that's happened? Spotify regularly recommends me things I don't like, but I've also never had a dedicated human butler that recommends me music so it's hard to compare, but but I don't think spotify is doing any better than last.fm was doing a decade ago (if I had to have an opinion I'd say it was worse).
---
I don't want to sound rude or that I'm jumping down your throat, but I don't really feel like we live in the future you're describing, at least not today.
Hollywood has gotten pretty good at creating convincing fake faces (dead actors/actresses reappearing in Star Wars for instance) but I think that tech probably involves a lot of human artist input.
(I share your general dissatisfaction with these particular movies though.)
Here's a really good example (the video, not the audio, which the creators say was intentionally degraded):
The negative take is that this still looks really fake to me. His eyes are dead, his head whips around weirdly, his neck undulates and flickers, his face doesn't appear attached to his head, etc. Also, due to how Obama looks he may be easier than other people (ie no hair).
Also, this isn't exactly what OP was talking about, though it's similar. They were talking about replacing faces, whereas this is (I presume) actually 100% Obama video from his various speeches reconstituted. So tbc my reaction was to the various videos I've seen of _that_, none of which have been remotely convincing.
One man's modus ponens is another man's modus tollens. Why are you so certain of that in light of all the achievements you listed, most of which were not predicted to happen even a decade ago? Did someone release a mathematical proof that large NNs can't be intelligent while I wasn't looking? Did aliens arrive and tell us that no, AGI just doesn't look anything like what we're doing? How would the world right now look any different if we were on a route towards AGI in a decade or two, keeping in mind that we are still roughly at the 'insect' level on the various connectionist extrapolations and still aspiring towards 'mouse' levels?
Two promising approaches toward AI Safety I have encountered:
* Make sure the AI is not too certain of itself and will always defer to humans as the final judge. https://www.ted.com/talks/stuart_russell_3_principles_for_cr...
* Get the AI to learn and internalize human values. A possible technique is Inverse Reinforcement Learning: https://thegradient.pub/learning-from-humans-what-is-inverse...
Both are being developed by Stuart Russell among others. See his book here: https://en.wikipedia.org/wiki/Human_Compatible. The latter approach was also discussed on several occasions by Ilya Sutskever.
Many believe AGI will not be realized for at least a few decades hence but no one can really be certain it will not be developed before then.
Since several different approaches together are better than one. Bright minds including those outside the field should propose more ideas for such a significant & unique problem of our time.
I say we take a chance; if it doesn't work out well we are probably going to stuff it all up without the help of an AGI anyway, and at least something intelligent will remain if it decides to kill all humans.
If we knew aliens were coming, we'd probably be best off hoping for their benevolence and mercy and preparing as best we could to deserve it.
Aliens may not share our values. What we perceive as good may not be so for them, nor might they care if we are “good”.
So an AGI without a value system is (mostly) harmless. If the AGI has a defined value system, it's a lot clearer what that value system is. No "whoops gonna destroy the world to make paperclips" will happen.
(A not-fully general AGI might not be smart enough to defeat its own sensors, yet still be smart enough to do lots of harm. The typical example is grey goo, which isn't smart at all, yet plenty harmful.)
My own anecdotal data confirms this: a colleague crashed with his bike just meters from our office. Pretty much the entire company stood there watching him bleeding and screaming on the pavement. I called the ambulance,as it didn't seem anyone was considering doing it at all. I'm sure every single of them would have called the ambulance if they were there on their own. On a different occasion,a friend pulled a few almost drowning drunk guests out of water during a wedding,while the rest stood on the shore watching the whole situation.
If a system listened in on all communication within a company, for traffic, sentiment analysis, who responds to whom how fast, and what customers, customer service, and sales are all saying and doing, it would generate more data than a CEO could ever process. Somewhere in there are indicators of what's working and what isn't. That may be the next phase after processing customer data, which has kind of been mined out.
If this starts working, and companies run by algorithms start outperforming ones run by humans, stockholders will put their money into the companies that peform better. The machines will be in charge.
This is perhaps the destiny of the corporation.
For example, Apple.
Maybe outlaw ice cream next? Or how about marijuana smoking? (Oops, already tried that.) How about loud music (long term hearing damage)? How about motorcycles?
If an AGI starts inventing new and wonderful ways in which we can destroy ourselves, and we are taken in by it, we will have to restrict that, it's non optional.
You can make the argument that, say, current hard drugs should be legal - but I don't think there's a way to defend the position that any possible future 'thing' should always be legal/permitted regardless of negative effect.
In fact, likely much more harm than those with evil intent.
I'm wondering where this idea comes from (I'm not criticizing you specifically, but that maxim).
Do no investors value stability, longevity, ethical behavior etc?
Buy companies that customers love, sell companies that customers do business with only because they have to.
Sometimes a fast, go-big-or-go-home trajectory is what they're looking to invest in.
The way to put ethics in a language investors understand is to price in externalities.
For example child labour, slave labour, destroying environment, causing cancer, etc.
Interesting theory. My perspective is that once a corporation thinks of itself that way, it's already going downhill. Once a measure becomes a target, it ceases to be a good measure.
Now I assume you're imagining the well-defined metric to be profit, in attempt to dodge the aforementioned problem. However, even leaving aside the problem of short- vs long-term profit, I don't think that metric has a sufficiently well-defined link with actions to be optimized with anything we can build in the near term. There's too much "common sense" involved that, AFAICT, we still don't have any idea how to handle.
Ed: I guess I sort of did the thing the OP told me not to. But I still don't think the ability to phrase the problem of running a corporation as "optimize profit" makes it likely to be accomplished sooner than any other AGI application.
I think the “running a corporation” problem is actually a great example of how to measure a general intelligence not only because it has well defined success criteria but also because of the very open ended and general means by which it must be accomplished. Running a corporation requires communication, setting goals, creativity, number crunching, and lots of soft skills like branding or understanding how consumers/customers will react to things. But there is nothing uniquely human about it except that it needs to be able to make money from humans. Presumably if I’m paying for some well defined service like Search Ads I don’t particularly care if the corporation is run by a robot.
I do agree though that there is a bit of ambiguity here on what “run a corporation” means. I assume that with a lot of thought I could write a very procedural script that somehow incorporates, purchases a property, and rents it through a property management group.
Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
As soon as an AI CEO starts trying to maximize a single, well-defined metric, the humans working there will start finding ways to satisfy that metric at the expense of everything else. There is no single mesaure, no matter how well-defined, that will remain a good measure in perpetuity as long as humans are capable of interpreting it in the light of their own self-interest.
Imo not only is our software nowhere close to AGI, neither is our hardware or our ideas.
That said having a fire alarm to tell us the terminator is coming seems like a great idea in theory. As long as the sprinklers spray something that melts metal
Good call on that by the author; according to this summary paper [1], a model reached ~79-93% accuracy on various Winograd data sets.
One: As Stuart Russell observed, if you get radio signals from space and spot a spaceship there with your telescopes and you know the aliens are landing in thirty years, you still start thinking about that today.
Two: History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists in that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up.
Three: Progress is driven by peak knowledge, not average knowledge.
Four: The future uses different tools, and can therefore easily do things that are very hard now, or do with difficulty things that are impossible now.
Five: Okay, let’s be blunt here. I don’t think most of the discourse about AGI being far away (or that it’s near) is being generated by models of future progress in machine learning. I don’t think we’re looking at wrong models; I think we’re looking at no models.
It's more of a hardware to run specific types of algorithms.
And history is also filled with people who said "this can't happen" and died without ever getting the chance to say "I told you so" 2000 years later when [thing] still hasn't happened
I'm guessing from the title there may be a useful point to it, but it could do with some tough love to unearth it.
I think the author is just writing in a frame of mind where he has dealt with a lot of people arguing with him, and he feels the need to stem the tide of common counterarguments to each point he makes before he moves to his next point. To me, that's actually less distracting then when an author ignores an important counterargument and leaves it unaddressed while continuing their point.