Hacker News Reader

Comment by icky | Hacker News Reader

It's surreal to see this even in the NYT's orbit.

I was thinking the same thing. It wasn't too long ago that the only mainstream coverage of this stuff was the occasional Kurzweil-centric fluff piece along the lines of "Hey, this crazy guy says we'll live forever soon." I wonder if the singularity debate's finally leaking into the public consciousness, or if the Spectrum special issue just forced a blip.

Well, I don't think anyone can argue that if we create an AI smarter in nearly every way than any given human, several or a large amount of those AIs could create an even smarter AI, and so forth recursively. The only question is if we indeed will, and if so when (even 100-150 years isn't that far off if you think about it). Given the huge impact it would have on society (for better or worse), it's perfectly reasonable to be a matter of public debate.

Actually that point isn't necessarily obvious to me. Maybe we can create an AI which is 10% smarter than ourselves, but maybe a 10% smarter AI still isn't smart enough to create an AI 10% smarter than itself. Maybe building a 121% AI that smart is such a hard task that even our 110% AI can't manage it -- maybe the best it can do is a 115% AI.

Analogy: Out of Lego, I could build a robot arm that's capable of building a slightly larger robot arm out of Lego. But I can't just keep on doing this 'til I have a robot arm the size of the Burj Dubai, because there are limits to how big a Lego-based robot arm can be before it collapses under its own weight.

I'm not saying that AIs are like this, just that they might be.

Your analogy is particularly apt if you consider that two of the means of achieving AI are emergent and thus not engineering and not at all friendly to reverse engineering. If we can simulate the brain's processes in bulk, we still don't necessarily understand how it does what it does. (Like making a photocopy of ciphertext.) Likewise, if we create some sort of architecture that lets us evolve AIs, there is no guarantee that we will be able to understand how the results work. There's no guarantee that the resulting AIs will be able to either.

I think Godel's Incompleteness Theorm implies the machines can't grok the code they are made from. I'm not sure though.

How do you know we'll be able to create an AI smarter in nearly every way? What if we just create an AI that's like an average human, but is able to pay attention without distraction? What we'll have are a bunch of ideal slave laborers, not a runaway exponential increase of intelligence.

Does this assume that at some point humans will have successfully mapped the entirety of the brain and its functions?

Reply on news.ycombinator.com