Comment by jessriedel | Hacker News Reader

Comment by jessriedel | Hacker News Reader

Parent Full thread

The question of whether AI is dangerous in the hands of people with good intentions is more difficult than the one of whether it's dangerous generally. I was only trying to convince you of the former, in response to this:

> AGI will completely upend society by giving every human being access to practically infinite resources...But the idea that it is therefore "dangerous", is nothing but a pre-theoretic misunderstanding of very complicated machinery.

But to get at the harder question, I think you misunderstand the paperclip story. It's not supposed to be an general argument for the danger of AI, and the danger is not that people will design a machine which can be easily predicted to fail. The danger is that they will design one that fails for reasons they did not foresee, and the point of the paperclip story is just to illustrate that the simplicity and mundaneness of the goals you give a goal-driven AGI doesn't bound how bad the impact can be. This arises because of the monumental shift between (1) telling a machine explicitly what to do and (2) telling a machine what you want.

The paperclip example is used because we can understand both the goal (paperclips) and the action that produces the goal (grab atoms, build paperclip factories). Whats fundamentally different about an advanced goal-driven AGI arising from recursive self-improvement is that, for sufficiently difficult goals, you won't understand the actions. Therefore you must get the goals right.

Now you can certainly dispute whether people will build a goal-driven AGI (e.g. something that has an explicit utility function) rather than something else, but that's really an empirical question about the choices of the designers and, more importantly, what the easiest way to get an AI to recursively self-improve is.

EDIT: Also, have you read the book? I think it has flaws, but it certainly doesn't contain "the _most absurd_ arguments" so I'm just afraid you might be misled.

I understand the paperclip objection, but I do not consider it valid.

I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.

I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.

There are a lot of interesting questions and problems about making AI behave as expected. But from my perspective they are all technical problems that have specific solutions in specific architectures. And none of them are particularly daunting.

[1]: http://lesswrong.com/

> I understand the paperclip objection, but I do not consider it valid.

>I tried to point out the problem, which is that it focuses on single-objective optimisation when the only interesting question is multi-objective optimisation.

Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. You can the argue with others on that point, but that doesn't invalidate the paperclip story.

> I haven't read the book, sorry, but I have seen some of these arguments reposted on the net, especially at [1] - and what I have seen so far did not inspire in me the desire for more of the same.

Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.

> Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI,

That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect.

But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions.

Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology.

The only thing it says anything about is a toy variation that noone in their right mind would ever consider building.

> I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals.

Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly.

(Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)

> Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet.

Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either.

I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.

I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so.

[1]: http://en.wikipedia.org/wiki/Eliezer_Yudkowsky

> That may well be the original intention. But that's not how it's used in practice, is it?...

> But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there.

Perhaps I'm misinterpreting, but it sounds like you got the wrong impression of the point of the story by reading about it from some folks on the internet, and that you at least provisionally accept now that it could have a useful point, and that your appreciation of this came from someone (me) who read and cites the original material. That seems like good evidence that the book contains reasonable arguments that haven't yet made it to you unadulterated.

> (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.)

Actually, no. If the machine wanted to maximize the likelihood that is successfully built 100 million paperclips, and the machine is smart enough that it can take over the world with very high likelihood (or if it worried about the small chance of humans destroying the world with nuclear weapons), then it will first take over the world and then build the paperclips.

> Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom.

Yudkowsky runs that site as much as Paul Graham runs Hacker News. (It's a public forum, where anyone can post and content is voted to the top, etc.) I presume you would recommend that someone actually read Paul Graham's writing before dismissing his philosophy on start-ups based on the conversations of his followers on HN. And even if HN was on the decline and there was only trolls and bad thinkers left, you say the same thing. All I'm recommending is the same for Bostrom.

Reply on news.ycombinator.com