Comment by rayalez | Hacker News Reader

Comment by rayalez | Hacker News Reader

Just because you can't figure out how to define/test superhuman intelligence, doesn't mean it can't exist.

> If you think you're specific business case might require an AGI, feel free to hire one of the 3 billion people living in South East Asia that will gladly help you with tasks for a few dollars an hour.

How can a person writing an article about AGI fail to understand what AGI means so badly?

You're complaining that the kind of intelligences you can hire for $2/hour aren't useful/powerful enough. What does that have to do with anything?

By your own argument, people have already created very powerful intelligences - Elon Musk, Alan Turing, etc.

100 of those, with unlimited memory, internet access, and faster thinking speed would already be extremely powerful, enough to take over the world and do whatever they wish with it.

The whole idea of human beings being the pinnacle of intelligence always seemed ridiculous to me. We're the dumbest animals capable of discussing the concept of intelligence. What, the first few somewhat intelligent apes happen to also bump into the theoretical limit for intelligence?

Even minor variation in our genes are enough to account for the difference between Elon Musk and a Joe from the gas station who spends his life in a trailer park, sniffing glue and trying to seduce his cousin. And just because there are enough Joe's already, artificial superintelligence can't exist or be useful?

Having said that, even Joe-level AI's would transform the world more dramatically than any invention before that - they would replace human labor.

> 100 of those, with unlimited memory, internet access, and faster thinking speed would already be extremely powerful, enough to take over the world and do whatever they wish with it.

Why? What makes you confident in this assertion?

To whoever has downvoted: I doubt it. Surely if you think this is so obvious that my question seems disingenuous, you can clearly articulate why the claim has substantial credibility?

Existence of Elon Musk's brain proves that Elon-Musk level intelligence is possible.

The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world.

Imagining a team of 100 Elon Musks with unlimited memory/energy/etc/etc taking over the world is not such a stretch.

Why is it not a stretch? This is not a rhetorical question (in the Socratic sense) - what I'm asking for is precision.

If you literally mean infinite memory, your claim is trivially correct regardless of whether we postulate a team of 100 Elon Musks or a team of any 100 replicated adult humans in the world. Even a modest human intelligence with access to infinite memory can eventually do anything possible in a computational sense. So let's pare that back a bit, because infinities break thought experiments.

Do you mean something like 100 Elon Musks, each with a datacenter's worth of GPUs? Is that sufficiently many to conquer the Earth and do what they wish? If so, why? How would they do it? If you don't have a realistic example for how they'd do it, why are you confident they can?

All I'm saying is that much dumber people have been known to get much closer to taking over the world.

Obviously "infinite memory" and "taking over the world" are exaggerations. By "infinite memory" I mean "much more than any human have ever had, enough to memorize the internet". And by "taking over the world" I mean "be able to amass enough power to become a serious threat".

Humans, with human intelligence have already been way too close to taking over the world or causing apocalypse or whatever.

If we're talking about non-aligned AGI, we're talking, at the very least, about an ElonMusk+ powered sociopath. Isn't that already a cause for concern?

Or, from a different perspective, imagine a business that employs 100 Elon Musks. The article claims that AGIs already exists and they're "useless". Do they? Are they?

"All I'm saying is that much dumber people have been known to get much closer to taking over the world."

Like who?

This entire comment thread is populated by people who did not read and comtemplate the posted article.

James Clerk Maxwell was a genius, likely more intelligent than Musk, who did not take over the world.

If James Clerk Maxwell was born 10,000 years earlier, he would likely be a respected shaman or an intelligent warrior, but he would not have derived the laws of Electromagnetism without 10,000 years of scientific development.

Likewise, Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day.

Paraphrasing the linked article, a superintelligent AGI will do nothing because physical resources and crystallized empirical knowledge is the bottleneck to human progress. Not intelligence.

Now I would like to see AGI fanatics address that core thesis.

Unless AGI can do 1000s years of human thinking in a few weeks. Or, you know, read wikipedia.

I’m not an AGI fanatic but I will try to address it.

Due to several reasons, based on ethics, the possibility of violent reaction, etc., there is an upper bound on how much power (read work / time), or more generally performance ( results / resources) we can squeeze out of us and other fellow humans. This upper bound limits the crystalization process that you mention.

Technology enables us to push that upper bound, and it has been applied in a lot of areas - including the improvement of thought assistance (programming, excel, etc.).

But the actual thought is an untapped area that resists automation; and, following Amdahl’s law, the more you optimize a part, the more the non-optimized part weights as the reason we don’t go quicker. You can see this in the growing demand of intellectually intensive jobs in the last couple of centuries, and the huge salaries that elite workers in finance or programming command.

If we reach thought automation that is at least on par with humans, we have just opened the only area that was resisting optimization, and we can suddenly improve entire dimensions that were out of bounds.

And, as a free bonus, we can now more easily convert capital to labor (just deploying more machines and “cloning workers”, instead of raising children).

This could create a runaway effect, or at least have an immense multiplicative effect.

Until now, your central point still stands:

> Elon Musk is nothing without the experimentation, empiricism, and the sum total of human knowledge accumulated to this day.

That /is/ true. But! If AGI is reached, you a) can buy your own Elon, b) can copy it, improve it, experiment with it and generally manage it in ways we currently can’t; and c) you start to accumulate knowledge capital at increasing velocity that is yours and doesn’t decide by itself to die, quit and go elsewhere, or play politics with its power instead of diligently working. So from where we stand, it seems reasonable to expect a “winner takes all” scenario.

In sum: a huge part of the initial value doesn’t come (nor needs) from god-level power, but from “enslaving” thinkers the way we “enslave” production lines, cars and computers. None of this requires something a thousandfold more intelligent than the average human. Not at first at least... ;)

> The fact that Elon Musk (and similar brains) already have a ridiculous amount of power and are using it to change the world proves that Elon-Musk level intelligence is enough to gain a lot of power and change the world.

Surely the null-hypothesis should be “being rich makes it easier to get richer, and their initial breakthrough was luck”? Personally I would guess that there are a lot of smarter or equally smart people who never get close to that level of power.

You're completely missing the point, not seeing the forest for the trees.

Take any reasonably intelligent human, say a valedictorian from your hometown high school.

Now consider a system with the same intellectual capacity for reasoning about the world as your valedictorian and give that algorithm a data-center's worth of computational resources. Keep in mind that this system has perfect recall. Keep in mind that the speed at which the system can perform intellectual labor is not constant as it is in humans, but bounded only by the amount of computational resources that the system has access to.

So this system is just as intellectually capable as your valedictorian with one data-center's worth of resources and it has perfect recall. Now what happens if we double those resources and give it two datacenters? Does it become twice as intelligent? Or twice as fast at accomplishing some given intellectual task? We can't say for certain because the answer depends entirely on how such a system might work and since no such system yet exists, we can only speculate. But as it turns out, we don't need to know exactly which dimension it would improve in, only that it would improve in some relevant dimension.

And that isn't even considering the fact that such a system could improve its own architecture further improving in some given dimension relevant to its ability to act intelligently.

Assuming intelligence is capped at the human level is as naively anthropocentric as the old belief that the Earth was the center of the solar system.

The point appeared to be that intellectual capacity was power. I’m not disputing that AGI or ASI should be possible — indeed, I’ve written about the insanely transformative nature and plausible timescales myself [1] [2] [3] [4] — but I am disputing that brainpower is the most important metric for get-stuff-done power.

As an aside, I had to Google “valedictorian“, as it’s a cultural reference that I have no experiences of. We get our results in a much more boring way where I’m from.

[1] https://kitsunesoftware.wordpress.com/2018/10/01/pocket-brai...

[2] https://kitsunesoftware.wordpress.com/2017/11/17/the-end-of-...

[3] https://worldbuilding.stackexchange.com/questions/51746/when...

[4] https://kitsunesoftware.wordpress.com/2017/11/26/you-wont-be...

I'm a bit late in my reply as I don't have notifications setup for HN but it appears you may, so here goes.

I think that it's clear that brainpower is at the very least an important metric for determining get-stuff-down power as the vast majority of human beings to ever accomplish anything of note (financial, scientific, or military success) all possessed above average intelligence in way or another. I suppose the case might be made for different kinds of intelligence, but presumably something considered worthy of the title of ASI would be competent in any conceivable dimension of intelligence.

And if it is true that brainpower is at least an important component in whatever might make up get-stuff-done power, and assuming that the level of brainpower under a hypothetical ASI's command was effectively (in relation to a human being anyway) unbounded, then whichever other (external?) metrics potentially attributable to get-stuff-done power could presumably be compensated for by the overwhelming weight of the brainpower.

I'm curious what other metrics for get-stuff-done power you might have in mind?

Reply on news.ycombinator.com