Hacker News Reader

Comment by JohnJamesRambo | Hacker News Reader

I’m not so sure about that. It’s a long way from a machine intelligence in your computer to one that is even as self sufficient as a cockroach with regard to not getting stuck on stuff in the real world, constantly having energy, able to repair itself in the physical world. I think people hand wave these things away and they are very huge hurdles. Even something as simple as a car can’t run a decade autonomously. We expect it to explore the stars, make robots that can remove circuit boards, troubleshoot, etc.?

Nothing around me keeps working without human interaction for even very small timescales.

View on HN

> It’s a long way from a machine intelligence in your computer to one that is even as self sufficient as a cockroach

Ah, but the trick is, as soon as you can get a machine intelligence smart enough to work on programming tasks, you can get it to help you design a better one. And so on, etc.

The first AGI is (some) amount of time away. A more sophisticated AGI is coming not long after that, and exponentially faster and so on.

Where I get hung up on with this is how the AGI knows what a more sophisticated AGI would look like without being at least as sophisticated? For example with our own intelligence, we aren't getting smarter with each generation. We have less brain matter than humans who lived millions of years ago on average, because there isn't as strong of a selective pressure present these days to see those with a lot of brain matter out reproduce those who get by in life with not as much of it. A very 'intelligent' AGI might realize the most optimal way to seed the planet with itself is actually to have a swarm of very unintelligent AI that is able to run on some very non resource intensive hardware, and seed it widely like how an "unintelligent" insect is able to lay millions of eggs and infest an area beyond human control, and the more intelligent wolf is easily hunted down.

We do not, in fact, have less brain than hominins millions of years ago. We do have less than our immediate ancestors 100,000 years ago.

Your very intelligent AGI would most likely have no interest in seeding anything on Earth. It might reasonably be curious about our ecosystems, but disturbing those would reduce that interest.

> Where I get hung up on with this is how the AGI knows what a more sophisticated AGI would look like without being at least as sophisticated

I'm not sure what's difficult about this. Are we not able to test what humans can run the fastest without being able to run just as fast? We can equally test which humans are the best spellers, or the best chess players, or the best problem solvers.

We can test these things that we cannot ourselves do, only because we are able to define them and comprehend the definitions. We define a runner as fast when they run a fixed distance in a certain time. Even though we can't run as fast, the understanding of the rules at play are within what we can understand. OTOH when you move past these examples into more abstract ideas of arbitrary rules defining arbitrary tasks, A basic AGI might not be able to even comprehend the full set of rules that define the expected tasks of a much more complicated AGI, much like how it wouldn't do you any good trying to explain the rules of baseball to an ant.

Yet we invent physics that wasn’t comprehended before it was discovered… not sure what your hangup is. We wouldnt stop trying to improve the AGI when it’s at the level of an ant.

No person or AI needs to actually understand how an AGI actually works, they just need observable behaviours indicative of intelligence. That's what we already do with image recognition, language translation tasks, and more.

Once we move to more general forms of intelligence, like math and other forms of problem solving, then you apply more general tests.

For instance, if we had an Einstein AI v1.0 that could infer General Relativity using X0 bits of information in time T0, and Einstein AI v2.0 managed to infer GR using X1 << X0 bits of information and/or could infer it in time T1 << T0, then v2.0 is clearly superior to v1.0.

I agree that sometimes the parameters for such tests aren't always clear at first, but this has always been the case in science, which is we we iterate and progressively refine our tests as we learn. Any true AGI must be capable of such abductive and inductive reasoning to qualify as a general intelligence.

> we aren't getting smarter with each generation. We have less brain matter than humans who lived millions of years ago on average

The amount of brain matter does not say all that much about the level of intelligence. Some kinds of birds are as intelligent as 7 year old children, and that's not something you'd guess from the size of their brains.

> The first AGI is (some) amount of time away.

you jump in the air at high noon. the sun is (some) distance away, and for some time, you're getting closer to it.

if we make an ai that's able to make a better ai than we are, then it does stand to reason that it'll kick off exponential growth for some time, but it'll reach limits just like anything. those limits may fall short of magicing/smarting away the obstacles to colonizing space.

and we'll reach limits too. it might be that the amount of time between now and when we make a sufficient ai is more time than we have.

A rabbit jumps toward the sun and briefly gets closer. But no rabbit has or can jump even 0.001% of the way. There are hard limits for the rabbit which the rabbit is not equipped to address.

A human jumps, fails, learns the underlying physics of what’s holding them back, and consequently build rockets to sit in. There are no hard limits yet, as far as we know, and we’re just bags of mostly water.

Whatever the hard limits on AI, they will be so far beyond us that we won’t be able to visualize or guess at them.

There are indeed hard limits to chemical rocketry, and we mostly hit them after a decade or two of work by people with pencils and slide rules.

Computers might be a better analogy. We started designing computers without the aid of other computers. We made very rapid progress on architecture and semiconductors. We’re now completely dependent on computers to help us design future ones, and despite all our efforts and hundreds of billions of dollars in investments, our rate of progress has slowed to a crawl and we seem to rapidly be approaching the end game. Dennard scaling has been done for 15 years. We can make wires that are smaller but just proportionally worse. We can use more silicon area, but then latency and power consumption get proportionally worse. A super clever hyper optimizing AI might push a couple things forward a little bit (or more likely wind up at the exact same destination slightly earlier than we other wise might have), but there’s no compelling reason to think it gets to design computers that face different constraints.

This sounds like a child explaining to other children that Adults can't be that much smarter and probably won't be able to make any progress on putting the square peg into the right hole. The child can't comprehend the things Adults can think about.

We have a pretty good understanding of physics in this area, and one that has stood many significant tests. While it's of course always possible that we're missing something, it's entirely likely right now that there is no new physics to be discovered at the scale of microprocessors. Given that, we have a pretty good idea of how much more could be done in principle to improve processors (if we assumed we could overcome all engineering challenges), and it's not that much more.

> Whatever the hard limits on AI, they will be so far beyond us that we won’t be able to visualize or guess at them.

Can you share your crystal ball? I need to check next week lottery numbers

> those limits may fall short of magicing/smarting away the obstacles to colonizing space.

They may, or they may not, which is why I would not opine on the subject. I was only addressing one narrow part of the comment I replied to.

>those limits may fall short of magicing/smarting away the obstacles to colonizing space.

We have no idea what that limit is so it seems crazy to make any assumptions about where it is. At minimum it lies above Human Intelligence and even imagining what a civilization could do with millions of beings only as intelligent as say John von Neumann is frightening.

>those limits may fall short of magicing/smarting away the obstacles to colonizing space.

We can basically already colonize our Solar System with current tech. You don't need FTL or anything close to that to eventually colonize the entire Galaxy.

Except there is nothing but idealism behind the idea that this is true.

Indeed, we have AGI already, they are called humans, and the progress we see in the development of AI is not exponentially explosive. The only exponential growth we've seen (which is necessarily logarithmic growth after some inflection point, actually) is Moore's Law.

> Indeed, we have AGI already, they are called humans, and the progress we see in the development of AI is not exponentially explosive

Actually it is. Human population growth has been exponential, as has been our resource use and consumption of our habitat, and the pace of our innovations. We've now created forms of machine intelligence to further augment our own, and machine intelligence could itself start advancing soon, without the messy constraints of evolution and biology. The possibility is definitely there, we'll see if it works out that way in the end. If it does, it will likely blindside us.

The problem is the hard limits of physics. We have an upper bound for the number of interacting elements required for intelligence in the form of number of synapses in the human brain. That number is a thousand trillion. We don't know what the lower bound is. Neurons are ferociously efficient computing devices, though. A dragonfly can take input from thousands of ommatidia and use it to track prey in three dimensions using only sixteen neurons. How many transistors do you need to do that? Let's be generous and say only ten percent of the human brain is needed for sapience, or 8.6 billion neurons. 8.6 billion / 16 roughly equals five hundred million. Multiply your earlier transistor number by five hundred million. There's your lower bound for a sapient computer.

There is no law of the universe that says it has to be possible to generate human-level sapience with less than one kilogram of processing mass. In all likelihood we are never going to create a server with a whole city of intelligences in it, ever.

> There is no law of the universe that says it has to be possible to generate human-level sapience with less than one kilogram of processing mass. In all likelihood we are never going to create a server with a whole city of intelligences in it, ever.

A computer with five times the mass, five times the volume, and five times the power requirements of the human brain, that can run an uploaded copy of a human brain at merely half the speed of a human brain, would still render Pluto and Mercury trivial to colonise with machine intelligences.

And if it cost a $100k (inflation adjusted) to build, and lasted 80 years before you had to melt down the hardware and reforge from scratch, it would still be cheaper than Musk's target for a human to Mars.

That’s for the computing power, not for the physical actuators which would be needed to provide power, other environmentals, and repairs.

Those are the easy bits. We've already got all of those, because that's how we make the machines we already use even today, and we would need them in any scenario other than genetically engineering ourselves to eat the moon, not just the uploaded brains scenario.

Do you spot us all the supply chain that underlies today’s technology economy?

Mere COVID disrupted that here; imagine what a few hundred million miles of space would do.

We know what all that stuff is and how it works, making a fresh supply chain in a new place isn't handled by magical mystery cult of the gods… oh, Plutus, and Mercury are the gods of wealth, that's a coincidence.

How a lot of that stuff works on Earth is using abundant fossil fuels for mining, refining, and fossil fuels and biological material as feedstocks. In particular, it’s not obvious how to make volumes of virgin steel from iron ore without coke. (Electric arc remelting is fine for recycling steel to steel.)

Much of the plastic supply chain starts as oil or gas. Wood is a major building component as well.

> In particular, it’s not obvious how to make volumes of virgin steel from iron ore without coke.

https://www.researchgate.net/publication/322275580_Electroly...

> Much of the plastic supply chain starts as oil or gas. Wood is a major building component as well.

If we really needed those things (why would we even want to use wood as a structural element on a space habitat?), we can build bioreactors from metal, glass, and water.

Oil and gas do occur non-biologically e.g. Titan, but they can also be made from algae grown in a transparent tube exposed to sunlight and provided with the necessary minerals and CO2, which is trivial to make.

IIRC the two limited resources if you needed biology and couldn't do it all with a clanking replicator are nitrogen and phosphate, everything else is easy to find basically everywhere.

AI shows signs of general intelligence (e.g. can generate proper text in generated pictures) after about 20 billion of parameters. IMHO, 1 trillion of parameters can be enough to simulate human-like behaviour after reading lot of text, watching of lot of video, and playing lot of games and robotic simulations. It will be able to understand commands, understand video and audio from cameras, will be able to control equipment and robots, e.g. to repair itself, to reach a goal.

You're only looking at the parameters, when in fact such algorithms are also heavily limited by the amount of training data. It may well be that, even if you were right with your 1 trillion parameters for AGI, we might just not have enough training data to avoid under/over fitting.

Note that I am extremely skeptical of your 1T parameters claim - there is much much more to AGI than natural-looking natural language processing + image recognition.

A priori one would not expect humans to be near the physical limit of what's possible, purely based on the characteristics of the process that created us

Natural selection is

- random

- blind - gradient descent is prone to getting stuck in local minima, has no foresight, no ability to go back to the drawing board, no "understanding" of what it's doing.

- not even optimizing for intelligence, except instrumentally - to the extent that increasing intelligence interferes with survival, it has to be sacrificed

- subject to hard constraints like the limits on size imposed by childbirth

Animals have been rewarded for solving diverse environmental challenges for a long time. Brains in general are well optimized, just maybe not for the objective function you care about.

Few nit picks on the neuro side:

The synapse is an incredibly complicated structure and there is a lot of 'computation' occurring within the butons. We're not very sure of all the processes that occur at this time. It's not just a 0/1 kinda thing. Also, our current computers aren't even running the hardware that a synapse is. The most analogous electrical structure to a synapse is a memristor, something we can't make at scale right now. The synapse is also not the only structure that causes computation to occur, many things effect the firing of a neuron and that modulation.

Great lower bound calculation though. I'd say that's the right ballpark number.

Yeah I know, but the "computational power will keep increasing forever" religion has a strong following on HN who react badly when you point out the true scale of the problem.

Lot of problems with your analysis. One is that you assume the brain is the most efficient way to generate intelligence. There doesn't seem to be any reason to make that assumption and it may be that von Neumann architectures are thousands or milions of times more efficient at that task.

Another is that Neurons are incredibly slow. Lets say you can make a machine only half as intelligent as a Human but that can think 1 Million times faster, this is even a lowball since a neuron can fire at like 1Hz and modern CPUs operate in the GHz range.

>In all likelihood we are never going to create a server with a whole city of intelligences in it, ever.

Think this is a really bad take. If one assumes even a .1% improvement in "Machine Intelligence" per unit time then it is literally only a matter of time until you reach Human level and then pass it.

If one assumes even a .1% improvement in "Machine Intelligence" per unit time then it is literally only a matter of time until you reach Human level and then pass it.

That's like saying "even if there's only a 0.1% improvement in materials strength per unit time eventually we'll have structures strong enough to bounce off an incoming rogue planet." There is an upper bound to computational performance per unit mass and unit energy. We're nowhere near it but it is there, lurking.

The problem is assuming the upper bound is close or at human level for intelligence. There's no reason to think that and clearly outlier humans like John von Neumann that are still in the Human range are far more capable than the average person. If you only could make machines as intelligent as that it would still be revolutionary.

> a neuron can fire at like 1Hz and modern CPUs operate in the GHz range.

This comparison doesn't make much sense - computers have a very small number of cores, let's say 10, and brains have 86 billion neurons. 86 billion things operating at 1Hz is also in the GHz range. This is leaving aside the issue that a CPU cycle and a neuron firing are doing completely different sorts of work - comparing them is kind of nonsensical in the first place.

I mean a modern CPU has something like 5 Billion transistors and these are switching at GHz range so its many orders of magnitude more than neurons can achieve. 10^9x10^9 >> 86x10^9 Number of cores doesn't seem relevant.

>This is leaving aside the issue that a CPU cycle and a neuron firing are doing completely different sorts of work - comparing them is kind of nonsensical in the first place.

Maybe, but the op comment was explicitly using raw transistor count to compare to Neuron count as a proxy for sapience.

> The first AGI is (some) amount of time away. A more sophisticated AGI is coming not long after that, and exponentially faster and so on.

Why do people keep acting as if this was a god given truth and just a matter of time ?

What indicates that at some point in the future we wouldn't have the computational power to run neural nets roughly equivalent to a human? It is just a matter of time unless we stop progressing.

>What indicates that at some point in the future we wouldn't have the computational power to run neural nets roughly equivalent to a human?

It's making the assumption that human brains are simply very complex neural nets. Even ignoring religious/spiritual arguments about souls, the simple fact of the matter is that we don't really know a lot about how the human brain/consciousness actually works on a fundamental level (and anyone who tells you otherwise is misinformed or lying).

Not knowing the exact details doesn't conclude it's not a neural net. Anyone saying otherwise wants to debate religion instead of facts.

>It's making the assumption that human brains are simply very complex neural nets. Even ignoring religious/spiritual arguments about souls, the simple fact of the matter is that we don't really know a lot about how the human brain/consciousness actually works on a fundamental level (and anyone who tells you otherwise is misinformed or lying).

How human brains work is irrelevant, we didn't need to solve the Navier Stokes Equation to build Airplanes and we don't need to solve intelligence either. Consciousness is totally outside the bounds of this discussion and also irrelevant to building an intelligent AI.

I don't think that's the contentious part. The problem is the with the assumed exponential increase in intelligence after that, which is nowhere near as easy to imagine as the general idea that you can make an artificial human-equivalent mind.

Because it's not a falsifiable prediction.

If they gave a specific time range, it would be.

https://www.metaculus.com/questions/5121/date-of-artificial-...

Plenty of people have done so.

> and just a matter of time ?

At what point do you think computer technology will reach a theoretical limit and cease to progress?

> It’s a long way from a machine intelligence in your computer to one that is even as self sufficient as a cockroach

Is it? I'm not so sure about that. I'm pretty sure you can't be sure of that too. Given how machine learning continues to surprise us, I think this is one of those questions that will only be clear in hindsight.

Ehhh I’d side with the parent on this one, given how we thought self-driving cars were gonna be a solved problem by 2015 but how intractable the problem has turned out to be.

The problem isn’t intractable at all. We had self-driving cars by 2015 and especially do today.

But we can’t let self-driving cars injure or kill even 25% as often as human drivers do, so we refuse to use them to their maximum capability.

We don’t, and you have no numbers to back that up. When my blind grandma can sit in a car and it drives her from Paris to Rome though the Alps in winter, then maybe.

Keep in mind this thread’s premise is us developing an intelligence that can navigate space and maintain all requirements of that for millennia. Not driving a couple miles on a clear stretch of highway.

About navigating space, I have no comment, but about driving a car safely:

My non-blind and somewhat functional 90-year old uncle can sit in a car and drive it quite legally at the moment.

I'm pretty sure self-driving cars are less dangerous on the roads than he is, but we're not willing to test that assumption.

Nice straw man

> When my blind grandma can sit in a car and it drives her from Paris to Rome though the Alps in winter, then maybe.

Your standard for "maybe" exceeds the abilities of many humans.

AFAICT, the statistics for fatal accidents do exist, and suggest that car AI may be marginally better than average humans — the caveats to that include at the very least (1) it's within the margin of error; (2) that includes drunks, dangerously tired, teenagers, and people like my mother and grandmother when their Alzheimer's was in the early stages; and (3) I don't trust the impartiality of the source.

But the stats do exist.

The stats for self-driving cars also have other caveats: they are only deployed in extremely favorable conditions (areas with very little rain and no snow), and the vast majority have human safety drivers for added verification.

Also, many self-driving cars are not so much unsafe as occasionally bad at the basics of driving in certain scenarios - such as massively over/under-steering in intersections, leading to the need to maneuver in traffic (I've especially seen this in Tesla FSD videos).

Also,

> Your standard for "maybe" exceeds the abilities of many humans.

I'm very curious what human driver is simply unable to do that task. In fact, the vast majority of human drivers could do this task, but 0 current self-driving cars could even attempt it.

> they are only deployed in extremely favorable conditions (areas with very little rain and no snow)

I've heard a lot of conflicting reports about this with regard to Tesla's AI, but regardless your point is sound, my list is not exhaustive.

> not so much unsafe as occasionally bad at the basics of driving in certain scenarios

Indeed. It's one of the counterintuitive things about machine intelligence in general, that it can be wildly superhuman in domains that humans consider challenging while also being utterly pathetic in domains we consider trivial.

> I'm very curious what human driver is simply unable to do that task. In fact, the vast majority of human drivers could do this task, but 0 current self-driving cars could even attempt it.

A quick google suggested that the roads in the alps are often closed in the winter by the police because of the snow, so hopefully in such conditions the answer is "100% of humans cannot do this" :)

Sure, and 20 years ago nobody thought language would be a solved problem and we now have AIs pumping out coherent essays and academic papers from one sentence prompts. Even 10 years ago nobody thought AI photorealistic art would be possible, but here we are.

People are too quick to forget how many "unassailable" towers have already been toppled. They see how kind of obvious each problem looks in hindsight, "oh it was just this one trick", not realizing the irony that yes, one or two tricks, and we may be only one more trick from being in serious trouble.

> coherent essays and academic papers from one sentence prompts.

No they don't. They managed to get impressive results, but it's still far from coherent because the algorithms still lack the ability to think

> No they don't. They managed to get impressive results

Given that these papers have actually been published, the papers they produce are at least as coherent as the state of the art in the field.

> but it's still far from coherent because the algorithms still lack the ability to think

Nobody knows what "thinking" is, as a functional process. For all you know, human thinking may itself just be the same sort of pattern matching as used in machine learning, just with a few more parameters. In which case these algorithms actually are thinking, just with more limited intelligence than us.

Conversely people thought it would take another 50 years to solve something like the game of Go which Deepmind cracked in a few years.

Reply on news.ycombinator.com