Comment by eli_gottlieb | Hacker News Reader

Comment by eli_gottlieb | Hacker News Reader

Parent Full thread

>Exponential growth in our problem-solving capabilities is explosive when the problem-space is linear. But what happens when the problem-space itself grows exponentially with each iteration? Then we're back to linear progress.

I've always wondered: why should recursive intelligence improvement be possible? That is, if you're an agent, you're nominally searching for improved versions of your source-code to self-improve. That's obviously going to be a discrete, combinatorially large search space. Why should each search space have constant or falling entropy, conditional on the agent's existing source code and knowledge?

I don't really think "intelligence" can drive the entropy (or the compute-time) of any given search-problem to zero just by existing, so it seems like it should need to draw some resource other than memory space and energy from its environment. This resource is probably information: you would need some knowledge of the world to improve yourself. Successive improvements would then require successively better, finer-grained understandings of the world.

That makes sense, but implies that self-improvement becomes more difficult with each round you attempt, since you've already conditioned on most of the environmental information you can get. Entropy can't go to zero, precision can't go to infinity.

> I've always wondered: why should recursive intelligence improvement be possible?

That it's possible has already been proven [1]. Obviously there are limits to self-improvement, but we are ourselves limited in similar ways, ie. we are very likely simply finite state automatons.

[1] https://en.wikipedia.org/wiki/G%C3%B6del_machine

>Though theoretically possible, no full implementation has existed before.

And it's called a Goedel Machine because it has to prove, within an initial fixed logical language, that its improved program will be globally optimal. Rice's Theorem says this task is impossible in general (provided we stay within the deterministic, logical setting), and Chaitin's Incompleteness Theorem implies it should be increasingly impossible even in specific: the longer a successor program we search for (to incorporate knowledge that makes it smarter), the greater the chance of hitting a sufficiently complex successor candidate that we can't prove its optimality with our existing Goedel Machine program.

Again, that's within the paradigm of deterministic logics in which proof-checking is semi-decidable. Real brains don't stick within that paradigm (they're natively probabilistic), and thus AGI probably won't stay within that paradigm either.

Really fascinating stuff. I guess an AGI has to skirt the boundary between the purely rational (i.e. evaluating statements of propositional logic), and irrational (i.e. intuition).

What do you think intuition is, mechanically?

Probably a hidden Markov process[1]?

[1] https://en.wikipedia.org/wiki/Hidden_Markov_model

The point is more that intuition isn't something opposed to rationality, rather it is a decidedly deterministic logical process that you merely lack conscious insight into.

I hadn't ever thought of it that way. I guess it's a false dichotomy.

I've been reading up on deep belief networks and Restricted Boltzmann machines. Using these tools definitely reflects a deterministic logical process, there just happens to be hidden state involved.

Exactly, you have to tack on quite a few "provisos" in order to argue it's impossible. There are in fact many ways to circumvent incompleteness upon which every theorem you've quoted depends. Goedel machines have actually been built [1], so their existence isn't just theoretical.

[1] http://people.idsia.ch/~juergen/agi2011bas.pdf

[2] http://people.idsia.ch/~juergen/selfreflection.pdf

Edit: to see another way forward, consider alternate/finitistic arithmetics [3] which are neither stronger nor weaker than Peano arithmetic, but which do not exhibit incompleteness. Arguably, Goedel's theorems pass due to the inherent infinities of the successor axiom, but with an arithmetic like Andrew Boucher's system F (not to be confused with system F in programming language theory), no successor axiom exists, and so we cannot assume that the set of numbers is infinite, and we can enjoy full induction without difficulty. There is still a lot of surprising territory to explore in the foundations of mathematics.

[3] https://golem.ph.utexas.edu/category/2011/10/weak_systems_of...

>Exactly, you have to tack on quite a few "provisos" in order to argue it's impossible.

I wouldn't say it's strictly impossible. I would say the limit is environmental, rather than no limit existing. At a certain point, the available signals from the environment are as precise as they're going to get, the mind is as informed as it's going to get, and no further improvements are possible without a whole complicated raft of new experiences.

Kinda like how science often hits the point where we can't actually shift into a new Kuhnian paradigm by pure deduction, but actually have to do expensive, difficult experiments.

Interesting that you say "provided we stay within the deterministic, logical setting" because Paul Christiano's work shows that if we move to the probabilistic domain (the real world) it becomes possible.

I've always wondered: why should recursive intelligence improvement be possible?

Why does intelligence improvement have to be recursive improvement to the intelligence "source code"?

Why can't it simply involve adding more hardware?

Maybe the resulting intelligence of putting a million pieces of human level hardware next to each other would only be at the level of million human beings working in close cooperation with perfect communication ability. But that seems pretty powerful.

That might not be perfectly omnipotent but it seems like it would be pretty effect, in the fashion the humans are effective and constantly improving themselves, slowly on average, and quickly in the best instances.

Aren't humans already doing this with supercomputers and the internet overall? We can already have a million pieces of hardware communicating. What makes this any more powerful for an AI or uploaded mind? We're already doing it. So either the recursive improvements is already underway, with no need for AGI or uploaded minds, or the future AGI and uploaded minds won't do any better than we are.

>Why can't it simply involve adding more hardware?

Because adding linear amounts of hardware won't give you linear increases in performance. Almost no machine learning algorithms are O(n) in asymptotic complexity. Actually, I can't think of one that is.

Reply on news.ycombinator.com