Hacker News Reader

Comment by mannykannot | Hacker News Reader

The comment is somewhat tongue-in-cheek, but I think the concern is valid. It probably is anthopomophizing, but that does not mean that it is automatically wrong. It might be game over by the time an example is apparent.

View on HN

For an AI to become "uncontrolled" requires quite a lot of assumptions. I don't dismiss the concern as invalid, but I don't think the assumptions implied by the question are necessarily plausible either.

The argument follows from the first thing I wrote in this thread. Our understanding of complex systems is through abstractions, but abstractions (by definition) hide the details. Just as attackers can exploit details that are 'below the radar' of an abstract view, AI with a mind of its own (which it would have, by definition) could probably do things that escape our attention, including social engineering. That is not necessarily bad, but it might be out of our control.

To be clear: I am considering real AI, not that which is sometimes passed off as AI these days.

How would a strong AI learn to influence humans? How would it come to the conclusion that influencing humans would achieve its goal? How would it develop the ability to socially engineer humans in useful ways before the people monitoring it would notice? How would a strong AI develop the "motivation", for lack of a better term, to influence humans to remove constraints on its behavior? Why would this strong AI even decide that removing constraints would be its "goal" in the first place?

Reply on news.ycombinator.com