Comment by xianshou | Hacker News Reader

Comment by xianshou | Hacker News Reader

Here are a few more excellent resources on the potential and the dangers of machine intelligence. Even if you don't expect to read Nick Bostrom's Superintelligence - a deep, provocative, and thoughtful book, but also very verbose - the above links will give you an excellent primer on humanity's prospects if and when we develop a true general AI:

[Wait But Why]

The AI Revolution, Part 1: How and when we achieve machine intelligence, e.g. strong AI - http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...

The AI Revolution, Part 2: The species-level immortality we can hope for, and the extinction we have to fear - http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...

[Resources on Friendly and Unfriendly AI]

Unfriendly AI: What AI looks like if it does not act expressly in our interests; e.g., a universe tiled over with paper clips - http://wiki.lesswrong.com/wiki/Unfriendly_artificial_intelli...

Friendly AI: Strategies for designing an AI that respect human morals and metamorals, including what we'd want if we were as wise as a superintelligence (coherent extrapolated volition) - http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...

Here is the thing about all of those, none of them give concrete step by step examples of how we would get to "unfriendly" or "friendly" AGI respectively. Now, you might say - well of course because we don't know how an AGI will be built yet. Right! That is exactly the point.

At this point all of the MIRI (Former SIAI) staffers have stopped trying to explain paths to unfriendly AGI and have been just assuming them for their solutions. Just look at Yudkowski's work a few years ago on CEV and you'll see how any attempt at formalization falls apart.

Superintelligence just takes from the starting point they seeded in Global Catastrophic Risks, but doesn't actually address it in any more detail IMO. "Our Final Invention" tried also to do this but failed as well.

To be clear the AGI community understands thoroughly the risks inherent in AGI - I would venture to say far more-so than the non-researcher. So they are worth exploring, but at a certain point you have to say "We aren't sure if the atmosphere will catch on fire or not but lets try it anyway because it is the next logical step."

At some point we need to have the hard conversation about what humans will do in a world where we are less relevant.

Or just stop pursuing it altogether and put a moratorium on AGI research. A terrible terrible idea in my opinion.

thanks for these links. They seem extremely interesting.

Just curious, why the downvotes ? On the welcome page, it says it's okay to say thanks, am I missing some unspoken rule about Hackernews ?

Reply on news.ycombinator.com