The Singularity Is Further Than It Appears (2015)
rameznaam.com
rameznaam.com
And this is what has been nagging me about the Singularity and its associated predictions. Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress.
Futurism, thinking about the future of AI and technology is still important. Long-term thinking, planning, and prediction are always good. But progress will probably be much slower than anticipated.
I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge?
I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world.
That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.
That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons.
And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program.
Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.
[1] http://people.idsia.ch/~juergen/agi2011bas.pdf
[2] http://people.idsia.ch/~juergen/selfreflection.pdf
Edit: to see another way forward, consider alternate/finitistic arithmetics [3] which are neither stronger nor weaker than Peano arithmetic, but which do not exhibit incompleteness. Arguably, Goedel's theorems pass due to the inherent infinities of the successor axiom, but with an arithmetic like Andrew Boucher's system F (not to be confused with system F in programming language theory), no successor axiom exists, and so we cannot assume that the set of numbers is infinite, and we can enjoy full induction without difficulty. There is still a lot of surprising territory to explore in the foundations of mathematics.
[3] https://golem.ph.utexas.edu/category/2011/10/weak_systems_of...
I wouldn't say it's strictly impossible. I would say the limit is environmental, rather than no limit existing. At a certain point, the available signals from the environment are as precise as they're going to get, the mind is as informed as it's going to get, and no further improvements are possible without a whole complicated raft of new experiences.
Kinda like how science often hits the point where we can't actually shift into a new Kuhnian paradigm by pure deduction, but actually have to do expensive, difficult experiments.
I've been reading up on deep belief networks and Restricted Boltzmann machines. Using these tools definitely reflects a deterministic logical process, there just happens to be hidden state involved.
Why does intelligence improvement have to be recursive improvement to the intelligence "source code"?
Why can't it simply involve adding more hardware?
Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful.
That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.
Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.
The hard problems are going to be increasingly easier and easier to solve because they themselves consists of simpler problems. What technology is really good at is to share it's findings across the entire network. What one learn every other entity now have access to.
The mistake in dismissing the thinking is in believing that complex problems are irreducible complex, they are not. They are simply the layer of ever more simple problems.
Thats how we came to create the complexity and the problems that arose from it.
This is almost certainly false. Current state of the art in complexity theory is that some complex problems are irreducible. The expectation is that P != NP and that means there are irreducible problems. Assuming that P=NP is almost, but not quite, like assuming perpetual motion.
The claim was that the interesting problems are much more complex and thus can't be solved easily by machine learning.My point is that most of these complex problems aren't really anything but simpler problems gaining complexity by connection of simpler problems not by being irreducible.
Any engineer will tell you thats exactly how you solve complex problem, by breaking them into smaller ones.
In fact, the model is lower bound approximation to the real thing and if the approximation has certain complexity then the real thing will be much worse. That is the whole point of having a model. You approximate reality with something simpler and hope the solution to the simpler model problem is close enough to the solution of the real problem.
So what real world problems are you talking about? I'd say much of engineering and science is coming up with these solvable models for real world phenomena and so far no one has cracked the nut on coming up with a simple solution to problems that have complicated models. You might argue AGI is not one of those problems but that's a different argument.
"In the real world, the interesting problems are much much harder than that."
In one of the parents.
An example is sequencing the genome. This problem was solved much more quickly than anticipated due to advances in computer processing power. But it also revealed a huge new problem space of epigenetics. Complexity is emergent, not static and hierarchical. Progress is a tree, not a ladder.
It's only how and to what extent they are combined that they are of various complexity.
So once you solve some of those simple problems in some problem spaces you can solve the same level of problems other places.
In other words, you don't have to start over with every single problem. I am not saying it's always like that but I believe enough to make the singularity a realistic thing to plan for.
Right now if I notice one of my programs is doing something silly I come along and stop it, find out why it didn't work and start it again.
What if AI get's stuck on a (silly?) problem for an eternity, who will be there to help it out? Obviously I'm making some generationalisations about how The Singularity could potentially work. But maybe only being able to spend a certain amount of time on something is an important feature of our brains?
Frankly what I'd like is a 1000 foot high overview of all current types of AI technologies. I.e. What exists and what it's useful to solve. I think it's all just training neural nets on large datasets though. I can't help think there's a lot more potential than that.
Whether this provides a few percentage points or an order of magnitude advantage is a different question.
Going into the third dimension would allow you to cram way more transistors into a small volume of space. Once we run out of improvements along the first two dimensions, up might be the only way to go to keep transistor growth going for the future.
This is a general problem in solid state physics. Power density is already a huge problem.
No one knows how the brain computes and the current hardware models are very poor approximations as the article outlines. Stacking more silicon is probably not the way to do it.
We might not know how exactly brain performs some of the tasks, but we are pretty sure neurons are arranged in 3D structures, and heat dissipation is not a problem.
Actually, heat dissipation is a HUGE problem for biological structures. Heat exhaustion is a really easy thing to have happen to you.
Biological organisms operate well only in a very narrow temperature range. The brain is a gigantic heat producing organ coupled to an enormous heat dissipating organ--skin.
There is actually a limit study using fundamental physics that shows that the amount of heat you can remove from a solid per unit time has a limit. Overheating is unavoidable below a certain volume. We're very close to that limit.
Meanwhile 28nm remains the cheapest per transistor process - the first time smaller nodes got more expensive (due to heat, mainly). In the short-term, we might go for larger dies, clocked lower, to get more computation for less power. Perhaps all that wasted hardware will get used more efficiently (eg Metal and Vulkan).
This is THE big one for me. In the classic paper-clip-maximizer-gone-wrong example, there's no fathomable reason why generalized AI is necessary for such a specific, mundane task. But the organizations that have the (enormous) resources to develop such an intelligence are almost entirely focused on small numbers of tightly scoped problems.
It's very difficult to see any marginal ROI for any organization whose end goal is not some form of world domination.
Of course, that just means you'll see AGI efforts from organizations whose end goal is world domination. In fact, in general, half the people I've ever seen or heard using the term "AGI" were basically wannabe supervillains.
It sounds like a viable business plan and good incentive to me. Google builds gAI, sells it as a service to solve any problem.
Honestly, I think we are naive in thinking that GAI will just become another service. From our perspective it will be like inventing magic. Ask it what you want and it will give it to you.
The future will truly be stranger than fiction once GAI arrives. It will be like nothing we ever imagined.
Consider how buggy most software is. Consider that we have nothing approaching a general theory of computation. Consider that we cannot measure the quality of any given non-trivial piece of software. Consider that we don't even know what "quality" means in this context.
Consider there are minor mathematical issues - the halting problem, P != NP, the Incompleteness Theorems - which strongly suggest that all possible symbolic systems must be either incomplete or lacking rigour.
Consider that the thinking around AI is an unholy mess of millenarianism, techno-evangelism, philosophical confusion between unrelated concepts (learning, personality, emotional drives, sentience, self-improvement, general intelligence, "magical" post-human physics...) and wishful thinking.
I'm completely happy that we can build AIs for specific domains that perform better in those domains than humans do - because we already have.
I'm fairly sure we can build learning machines that can analyse a domain and provide a practically useful overview of it to a much deeper and broader extent than we do already.
I'm also fairly sure that process has practical and mathematical limits which are analogous to the transistor size limit for Moore's Law, and that the nature of symbolic computation itself makes it the most difficult of all domains to master.
My guess is there won't ever be a post-singularity god-AI. But there will be AIs that appear smarter than all humans in specific areas. And that these AIs will continue to be dumb slave machines which do what we tell them to, because domain mastery is completely orthogonal to any requirement for sentience, personality, or motivation.
The singularity already happened, in meat space, i.e. Humans. Now those same humans are trying to create AI's smarter than themselves. We still have not accomplished it but we will. All of this looks self evident from my point of view.
Remember, people used to think that humans would never fly. But why would they think that? There was plenty of evidence that flight was possible. Evolution invented flight, then we did the same. Evolution invented GAI in meat space, now is our turn to do it. This is a virtual certainty, unless we destroy ourselves.
The singularity also happened in net space--the Internet.
For all of its faults, the Internet is really effective external species memory. All of our current AI relies on the Internet as its "storage".
What if that area happens to be persuasion?
A GAI will be smarter than the entire human race combined. It really does not matter how much computing power you have if it is not GAI.
That still leaves a ton of organizations!
An off-the-top-of-my-head summary of some key points:
- Don't dismiss diminishing returns. Even if takes 10^6 times more computations to double intelligence, we may very well build a machine powerful enough.
- Units of computing per dollar, in contrast to moore's law, continues to double consistently and shows no sign of slowing down.
- Computational complexity is a theoretical model that makes certain assumptions that don't always matter in the real world, such as optimality and worst-case performance. Approximate solutions can be far cheaper and plenty sufficient. Average case performance is often more important.
- Impassable barriers can sometimes be entirely avoided by solving the root problem in a different way. Self-driving cars don't need human level vision, they just need a way of sensing the immediate environment, which LIDAR allows, despite being technically inferior to the human eye.
Of course none of this says we will achieve strong AI or a singularity. It is primarily a response to the type of arguments found in the linked article above.
Why is this argument about AI? The idea of a technological Singularity has nothing to do with AI. In fact the theory is pretty explicit about the fact the details of future technology are unknowable, and generally never predicted correctly. The theory attempts to explain a global evolutionary model in which global evolution continues despite the speed of localized evolutionary systems, like say biological evolution. Local effects of any particular technology's growth are not predicted, and moreover they are said to be unpredictable.
Even if the op was successful at arguing that AI does not work like every other technological system that is not salient to the ETA of the Singularity.
The point is not that cpus double in transistor density every 18 months, and therefore Skynet. The point is that evolution was exponential, human knowledge was exponential, electrical technology was exponential, digital technology was exponential, biological technology was exponential, and in aggregate exponential growth across all technologies is consistent and predictable. Even as one technology's growth or usefulness tappers off, others supplant it.
If you arbitrary pick something say, vacuum tubes or the printing press, and create some sort of argument that it doesn't in and of it self experience exponential growth, you may succeed in your argument, but you haven't said anything.
I certainly hope this prediction is true. But I think it has more to do with that fact that Kurzweil will be creeping towards the end of his life (gauged by average U.S. life expectancy) in the next 15 years.
The obligatory note: the true immortality is unattainable in principle without indestructibility (which is something that is hard to imagine even theoretically). Otherwise people will continue to die, en masse, from various unnatural causes. My guess as to the half-life of a human in the best of circumstances is just a couple of hundred years...
He's correct, but the reason is obscured: Moore's Law appears to be following a logistic curve, and is leveling out as we speak. If it wasn't, the compounding interest of quadratic transistor increases over linear time could well lead to a (relatively) hard takeoff at the point where a single chip contains a human brain's worth of calculating ability. Granted that concept (a brain's worth of computation) is hand-wavey and poorly defined: but the point is that if 20n's Intel processor has one brain's worth, than 20n+1 has two, and 20n+2 has four, and so on.
It's not that bad. There are two measures - function equivalence in devices using optimal algorithms - things like Siri, self driving cars and secondly trying to simulate a copy of the neurons in the brain similar to the Blue Brian project.
The numbers are roughly 100 teraflops for the first case, about 1000x that for the second.
Moores law gets all the press coverage but the more interesting thing is the amount of computing per dollar keeps doubling, did so long before Moore and probably will for a while yet. https://ourworldindata.org/wp-content/uploads/2013/05/Calcul...
Specific groups of humans - committees, office groups, political groups, etc - may well be dumber than the human average.
For cultural intelligence you need specific processes that generate, share, and try to maximise collective intelligence. Most groups lack those processes, and without them humans reduce to a confused and milling herd which is easily led by charismatic individuals - who are not necessarily the best and brightest.
Edit to add: processes to maximise collective human intelligence (like science, and persistent education) are exactly analogous to the processes that are claimed to be needed for GAI. Humanity as a whole is already an evolving GAI which has completely outperformed the limited potential of individual humans.
"Nothing about neuroscience, computation, or philosophy prevents it. Thinking is an emergent property of activity in networks of matter. Minds are what brains – just matter – do. Mind can be done in other substrates."
Yet the rest of his post is obsessed with "building" minds or AGI and tries to extrapolate based on that premise. There is a school of thought that views AGI as an emergent cybernetic process, "a metasystem-transition" [1]. This has roots all the way back to the concept of "Noosphere" that comes from Teilhard/Vernadsky [2]. Even if one does not feel aligned with such ideas, it is intellectually dishonest to posit AGI solely as the product of directed human engineering.
It is far more likely in my view, that AGI will be a Black Swan event and therefore all attempts to place it on a time scale, fraught with peril.
I mean, if you want to disagree, feel free. But at least understand what the claim is that you're disagreeing with.
But why can't computers be not just silicon? If natural selection can blunder into exploiting unusual physics, why can't we do so deliberately?
The idea that the physical universe is all that exists is so deeply ingrained that it's really hard to get people to even see that they're thinking inside a box...
So what are you proposing, exactly? That it doesn't follow rules?
In short, God. If God exists, if God is someone rather than something (a "he" rather than an "it"), and if God made humans in his image, then human personality can be real. It can be more than just a property of the atoms that make us up.
This may sound like a bunch of mystic woo. But I argue that it is the only thing that explains our observations of ourselves. No matter how much our theories say that we are just matter, that our personality is just the impersonal plus complexity, we still live - cannot avoid living - as if we were persons, not just machines. Why is that? I assert that our experience of personality is evidence that materialistic theories are inadequate.
But you can't help it in a wider sense. You seem to have a view that science tells you the real truth about reality, but that is not a scientifically-provable view. It's a philosophical or religious (in the wider sense) view.
A corporation optimizes for shareholder value, market cap or some other X related to business/market goals. They do not optimize for global intelligence capability. They do not benchmark the company based on quantitative capacity to meet or beat human capabilities across the spectrum of activities. Corporations focus on one or a handful of market specific metrics where they meet or beat their competitors. Full stop.
You could argue that it would be in the company's interest to focus on general corporate capabilities, in theory giving them a major advantage in the market, but they functionally don't do that, and I would argue can't because that's not what they are designed to do. I think the only major company that might be doing something close is Alphabet, and even they are hamstrung by it.
What I do agree with is this part: Lack of incentives means very little strong AI work is happening
Which is my primary frustration with the field. Most people don't even want to discuss it, let alone try and specifically work on it (even if it means working on subsets which could help lead to it).
I think there needs to be a philosophical MOVEMENT to create AGI. I think it will take that to get there in a short horizon. I think it will happen regardless, but without evangelical AGI proponents it's going to take a lot longer.
Really? OpenAI is a thing. Numenta has been a lovely scam for VCs for years. I found an "AGI" company just yesterday that actually publishes peer-reviewed research (http://www.maluuba.com/research/). There's another one that's kinda cranky but sent a guy to give a talk at MIT CSAIL last October or so (http://www.vicarious.com/).
Seems like it's a space where you can pitch yourself as "AGI", but you have to get your revenue from solving real-world machine learning problems. That seems pretty appropriate to me: you have to solve someone's problem to have a business.
Numenta, OpenCog et al... all of their founders want to create AGI, but they aren't evangelical - they are focused on creating companies which can move progress forward on NAI. Which by the way makes perfect sense, that's exactly what I do because it's impossible to fund AGI for it's own sake. In fact I NEVER talk to investors about how our goal is in the AGI space. For a million different reasons.
Seems like it's a space where you can pitch yourself as "AGI", but you have to get your revenue from solving real-world machine learning problems.
In fact it's worse. If you pitch yourself as AGI you'll get 99% of investors to walk away immediately.
No what I am saying we are not hearing is this:
"The purpose of humanity is to build an Artificial General Intelligence, it's the thing that we should all dedicate our lives to because it's the offspring of humanity and the most important thing any of us will ever contribute to."
Nobody is out there beating that drum yet, and it's a shame.
Nobody is beating that drum because it comes off as religious crazy-talk. I see zero reason why it should be our purpose to create any kind of metaphorical species-offspring, especially when you yourself say that any concept of making it safe for us to share a planet with is oxymoronic. That sounds like saying, "our purpose is to build an apocalypse and die for its greater glory". Again, that comes off as religious crazy-talk of the suicidal kind.
It seems to me there are entirely sane reasons to work on advanced machine learning and increasingly general AI. That's just not at all one of them.
Well nothing, we get to determine it. According to what you write, your assumption is that the "grand goal" is to "enjoy life."
I say, that's probably not it - or rather I don't think so (based on my experience/research, general revealed preferences for humans writ large).
So my proposition is that our "grand goal" should be to build AGI.
Such a narrow view of life is a tragic waste, regardless of what you think that destiny is. I expressed my frustration with the narrowness of your perspective in a metaphorical way. I am not literally assuming that you don't go outside.
But you pretty much have to believe the first (our intelligence is just neurons, there is no magic behind the curtain) in order to believe in AGI at all.
Right now, Humanity has no "goal" as it is currently with AGI - we haven't figured out a seed goal for AGI yet. In my view we should agree that the meaning for humanity is to create our successor: AGI. This fits into the evolution progression, only it becomes self directed.
In the sense that they take large amounts of money from VCs, don't ship a product, don't make a profit, and then clean up their books and start another company eventually, yeah. OTOH, if you're one of their investors, you might have a better look at their business model than me, so I could well be completely wrong. They've never gone public, so what would I know?
More importantly, they never really cared about making a profit. Their main goal has always been reverse engineering the neocortex. And regarding "shipping a product", both production-ready [0], and research [1] codebases are both open source, and being actively worked on.
Next time please refrain from making bold claims about companies you know very little about.
[0] https://github.com/numenta/nupic [1] https://github.com/numenta/nupic.research
http://longnow.org/seminars/02007/feb/15/what-if-the-singula...
The singularity has already picked up that flag and waved it.
the singluarity isnt "ai making new ai"
the singularity is ai solving problems that we cant - "greater than human intelligence."
which has basically already arrived. albeit bounded such that we still have ultimate control over what problems we direct ai to solve.
This piece is full of sloppy thinking as well as obsolete. Calling corporations superhuman AIs doesn't clarify the problem; it introduces oranges to a discussion of apples. And even in this irrelevant tangent, he is wrong. As we so often see in government and the private sector, many of us can be dumber than a few of us. Collective decision making has pernicious emergent properties, which means we should consider many corporations as subhuman AIs.
> The most successful and profitable AI in the world is almost certainly Google Search.
This, too, is false. Parts of Google ads might qualify as the most lucrative. But other parts of Google outside search, notably DeepMind, are much more successfully pushing AI forward. Autonomous cars and drones are two very successful examples of tech using AI.
The fact that he even brings up Jeopardy Watson in a discussion of AI shows that he knows little about the state of the art, which is light years ahead of IBM's question-answer system.
Ethical issues will not prevent nation-states and corporations from continuing to pursue the AI arms race.
And there are huge incentives to be the one to get this right. Which is why enormous investments in AI are being made by governments and the private sector alike. Google's DeepMind is going to more than double from 400 to 1000 people, half of whom are AI researchers. DeepMind is obviously a research powerhouse, and that investment alone must cost hundreds of millions of dollars beyond the acquisition price of 400M pounds.
AI advances hand in hand with hardware capacity. Distributed computing and faster chips will continue to progress, and pull AI along with them. A breakthrough in quantum computing will entail a huge step-wise leap in computing power and therefore AI. So progress will be non-linear, but not in the sense he thinks.
What would constitute progress toward AGI if not the ability to solve more and more complex problems? Winning at Go involved high-performance machine vision, and I think we'll all admit that vision is an important part of how an intelligence will operate in the world. It also involved reinforcement, or goal-oriented, learning, another crucial strategy for an eventual AGI.
It's been 20 years away for the last 80 years, right?
Well this is how I feel, so I like asking people to make a prediction so there is some public record people can look back at.
I worked at DeepMind as an AI researcher. I can guarantee you will not see "AGI" or "strong AI" in your or your children's lifetimes. It's fun to believe, it gives us something to look forward to, talk about with friends, and discuss on HN. But in reality, we are so far from artificial general intelligence, even with an exponential curve, it will take us 100 more years. The current deep learning era (or more aptly named, pattern recognition) will last another 5-10 years, at best. Then another winter will come.
RE: Siri and Alexa - Industry deployments of AI are a lagging indicator of what AI can do. AI is moving at two speeds: research and business/consumer applications. The research is moving faster than the apps.
AlphaGo and many other fundamental papers to come out in the last two years have done more than nudge the ball forward. They constitute significant steps, and bundled together they are an even greater achievement.
But I did raise a major objection, which is simply that his post is obsolete. It might have been reasonable to say what he said in 2014/2015. It isn't as we enter 2017.
Other people in this thread have already rightly stated that corporations are not superhuman AIs. This is true. Strong AI will be one or more models that operate as a coherent whole and be composed of machines, not humans. Corporations are mostly wetware. That's why they're not "artificial".
He claims there is a lack of incentive. That is not true. There is a well-funded escalation of research among large organizations that is well known and reported on, to the tune of hundreds of millions of dollars.
Why think that, though? Why not suppose that Strong AI would be distributed? Why suppose it would have a singular sort of intelligence? Isn't that basically a pre-internet way of thinking about AI? Maybe Super AI will be diffused across the environment, not having any sort of singular intent.