Hacker News Reader

Comment by mariusz79 | Hacker News Reader

I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.

True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.

View on HN

> I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment.

Given [0], I think that it's the only plausible way we can have to make it follow human values.

> True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.

That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1].

> I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.

Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious.

[0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

[1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis

The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously.

The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.

Reply on news.ycombinator.com