The Myth of a Superhuman AI

This will invalidate fantastical accelerationist woo until such evidence is presented, and -in the current state of the art research within relevant fields- it's not even certain we can know what would constitute such evidence _in principle_.

Reproductive fitness requires many attributes which obviously compete with intelligence, and evolution is a blind hill-climbing search which cannot easily escape deep local minima.

To me it's obvious that the development of superhuman AI is inevitable assuming we don't wipe ourselves out before we have time to develop it.

Agreed.

Dunno why he thinks this: "At the core of the notion of a superhuman intelligence — particularly the view that this intelligence will keep improving itself — is the essential belief that intelligence has an infinite scale."

The notion of self-improvement is both powerful and not assumed to be 'infinite'.

I figure it goes like this:

Q. Is AI potentially dangerous? A. Yes.

Q. Will it be human-like? A. Probably not.

People tend to couch these discussions as a series of anthropomorphisms.

We show that it is not only implausible that machines will ever exceed human size, but in fact impossible.

[...]

The term “supersized machine” implies a machine that has crossed some threshold, which is often denoted “human-level largeness.” However, it is not clear what “human-level largeness” could refer to. Has a machine achieved human-level largeness if it has the same height as the average human? If it has the same volume? The same weight? [...]

When one begins to consider these questions, one quickly concludes that there are an infinite number of metrics that could be used to measure largeness, and that people who speak of “supersized machines” do not have a particular metric in mind. Surely, then, any future machine will be larger than humans on some metrics and smaller than humans on others, just as they are today.

[1] https://arxiv.org/abs/1703.10987

Well, neither is there any reason we know that it's possible.

You're just demonstrating your apriori biases ("the brain is just a computer"), nothing more.

- we don't understand enough about the brain to emulate it.

- we don't know how to model the brain.

- even assuming we had the required scientific knowledge, there would likely be major engineering problems.

So, in other words, it's not impossible in the same sense FTL travel is not impossible, our scientific understanding does not categorically exclude it, but making predictions as to when it will happen is kind of silly, because it will likely require several fundamentally unpredictable scientific breakthroughs.

Square-cube law. As the number of processors grow how do you hook them together? If you want to minimize latency you end up with a solid packed sphere. O(cube root) latency. In that situation, heat dissipation only scales with the surface area of that sphere. So there is a limit to the scale. If you choose a topology that permits cooling, then you accept ever increasing inter-processor latency which eventually bottlenecks your computation.

Computers are not magic. They exist in a physical reality and have to deal with thermodynamics and the speed of light. Just because we feel like we have a lot of them doesn't mean we have enough for "magical super AI" to work.

So I'd say there's great utility in an AI being smarter than any human... and then the returns diminish.

But thanks for bringing up this link!

Conceptually the possibility of an AGI does present a real challenge (& one worth grappling); in practice, we haven't made any progress really toward realizing that problem. We've become good at feeding "large" datasets into algorithms that are many orders of magnitude less advanced than a lizard brain. Then we get to the issue that our large datasets are only large when thinking of classical statistical sample sizes. They nonetheless are lacking in redundancy, interactivity, stakes or any centered perspective such that there is no chance of anything resembling intelligence, agency, or plain old intentions arising.

The notion that we ought to take everything seriously that is 'not impossible by the laws of physics' is entirely stupid. Because virtually anything is possible, the question is how likely it is, and what evidence we have for it that is quantifiable and meaningful. Or are you currently prepping for an alien invasion?

Myth is the absolutely correct label for 'superhuman AI' fantasies, because the difference between a real threat and a myth is that the former is grounded in empiricism while the latter is a product of psychology and storytelling.

And superhuman AI is a very old story, it's simply the Golem myth. The religious idea that humans, in their hubris, create a tool beyond their control. It's Goethe's The Sorcerer's Apprentice and it's The Terminator.

Also, Kevin Kelly the author of the article is the founder of the magazine.

Incidentally, this is exactly what the AI superintelligence people are terrified of, and what the paperclip problem is meant to illustrate. https://www.decisionproblem.com/paperclips/index2.html

https://en.wikipedia.org/wiki/Kevin_Kelly_(editor)

To the second half:

Dunno. Good question.

Sure, he talks a lot. He makes grand statements. Is there any reason to take him seriously in this context?

He sure seems to believe, and to want to believe. But actual evidence that he's going to be right any time soon? That seems to be missing.

This is satire, right?

For those who didn't notice, nuclear plants have, in fact, melted down, more than once. Biosafety level 4 labs have, in fact, had leaks of pathogens before, more than once. An agent AI will tend to develop a survival instinct by default, even if we don't give it one, because it is a convergent instrumental goal. Thus, it would be motivated to not simply let you yank it, would fight you if you tried, and may even take preemptive precautions.

Something very weird would have to happen for it to be that hard.

The basic idea is that, the smarter you are and the faster you can think, the more information you can extract from weak signals. It involves creativity, and building models, and some minimum amount of in-built bias, but the upshot is that the amount of information a super-intelligent AI could infer from very small amounts of experience might absolutely stagger the imagination.

Do you think everything taught in an oral form (as opposed to hands on) is completely useless?

Are all military strategy books used by military academies a waste of time, since you can only learn by leading a real army?

In AlphaZero for example, there were 44 million training games total for 700,000 steps of training for the full 9 hours.

Turning that to human-scale numbers, 44million games with on average 60 moves, at 1 second thinking time per move,

> 44000000*60/60/60/24/365 = 83,7138508371 years of training experience in 9 hours

The whole field of Reinforcement learning has agents training and playing games for many orders of magnitude more time than a human ever will. In-fact, we can scale this to over 100k of actions per second, in a single machine:

https://github.com/alex-petrenko/sample-factory

Then, there is also distributed Reinforcement Learning, where hundreds of agents can play at different machines and share experience, see AlphaZero, LeelaZero, R2D2 agent, R2D3 agent, Apex, Acer, Asynchronous PPO.

> but the data isn't useful without the context of experience

The experience is the data in Reinforcement Learning.

> and all processing power can do it overfit model without experience.

That is wrong, the agents perform what is called exploration to avoid getting stuck in simple strategies.

> Even if we put AI into an army of robots running around and experiencing things, there are still scaling limits to encoding and communicating knowledge and understanding.

True, but machines scale better because they speak the same language, or they can learn to tune their language to get their message across.

> Human organizations are a great example of the scaling limits of intelligence.

Human organization is a testament to how far we can get with something as limiting as the commonly used language. The language that we use to communicate is subject to misinterpretation due to our subjective experiences, this limitation is not shared by machines.