Comment by tvural | Hacker News Reader

Comment by tvural | Hacker News Reader

Parent Full thread

People are worried about AI risk because ensuring that the strong AI you build to do X will do X without doing something catastrophic to humanity instead is a very hard problem, and people who have not thought much about this problem tend to vastly underestimate how hard it is.

Whatever goals the AI has, it will certainly be better at achieving them if it can stay alive. And it will be more likely to stay alive if there are no humans around to interfere. Now you might say, why don't we just hardcode in a goal to the AI like "solve aging, and also don't hurt anyone"? And ensure that the AI's method of achieving its goals won't have terrible unintended consequences? Oh, and the AI's goals can't change? This is called the AI control problem, and nobody's been able to solve it yet. It's hard to come up with good goals for the AI. It's hard to translate those goals into math. It's hard to prevent the AI from misinterpreting or modifying its own goals. It's hard to work on AI safety when you don't know what the first strong AI will look like. It's hard to prove with 99.999% certainty that your safety measures will work when you can't test them.

Things will not turn out okay if the first organization to develop strong AI is not extremely concerned about AI risk, because the default is to get AI control wrong, the same way the default is for planets to not support life.

My counterpoint to the risks of more limited AI is that limited AI doesn't sound as scary when you rename it statistical software, and probably won't have effects much larger in magnitude than the effects of all other kinds of technology combined. Limited AI already does make militaries more effective, but most of the problem comes from the fact that these militaries exist, not from the AI. It's hard for me to imagine an AI carrying out a military operation without much human intervention that wouldn't pose a control problem.

--------- Edited in response to comment--------

I am feeling like perhaps you didn't read the article? Many of these arguments are the exact lines of thinking that the author is trying to contextualize and add complexity to.

These are not bad arguments you are making, or hard ones to get behind. There are just added layers of complexity that the author would like us to think about. Things like how we could actually 'hard-code' a limit or a governor on certain types of motivation. Or what 'motivation' is even driven by at all.

I think you'll enjoy the originally linked article. It's got a lot to consider.

> Whatever goals the AI has, it will certainly be better at achieving them if it can stay alive. And it will be more likely to stay alive if there are no humans around to interfere.

This is a sequence of deductive reasoning that you brought up there. Quite natural for human beings, but why would the paperclip maximiser be equipped with it?

Seriously, the talks specifically argues most of the points that you brought up.

Shit is complicated yo. Complicated like the world is - not complicated like an algorithm is. Those are entirely different dimensions of complicated that are in fact incomparable.

> to do X will do X without doing something catastrophic to humanity instead is a very hard problem

This scenario I agree with. For instance: the AI decides that it doesn't want to live on this planet and consumes our star for energy, or exploits our natural resources leaving us with none.

The whole AI war scenario is highly unlikely. As per the article, the opponents of AI are all regarded as prime examples of human intelligence - many of them have voiced opposition to war and poverty (by virtue of being philanthropists). Surely something more intelligent than humans would be even less inclined to wage war. Furthermore, every argument against AI posits that humans are far more important than they really are. How much time of day do you spend thinking about bacteria in the Marianas Trench?

> AI control

My argument is with exception to this scenario. By attaching human constraints to AI, you are intrinsically attaching human ideologies to it. This may limit the reach of the superintelligence - which means that we create a machine that is better at participating in human-level intelligence than humans are. Put simply, we'd plausibly create an AI rendition of Trump.

>Surely something more intelligent than humans would be even less inclined to wage war.

The default mode for a machine would be to not care if people died, just as we don't care about most lower life forms.

> Furthermore, every argument against AI posits that humans are far more important than they really are. How much time of day do you spend thinking about bacteria in the Marianas Trench?

Exactly.

Which is why worrying about ourselves in a world with superintelligence is not wasted effort.

The extreme difference in productive abilities of superintelligence, vs a human population who's labor and intelligence has been devalued into obsolescence, suggests there will be serious unrest.

Serious unrest in a situation where a few have all the options tends to lead to extermination, as is evident every time an ant colony attempts to raid a home for food crumbs.

The AI's might not care whether we live or not, but they won't put up with us causing them harm or blocking their access to resources, even if we are doing it not to hurt them but only to survive.

> if you are given a strong AI randomly selected from the space of all strong AIs

Why would this ever apply? We're building them, not picking them out of a hat.

I think the current state of deep neural networking designs and research funding pouring into generalizing the simple neural nets we're working with now suggests that we are, in fact, pulling them out of a hat.

Right now we're just discarding all the ones that are defective, at a stupendously high rate as we train neural nets.

I can't speak to what method would generate the first strong AI, but I suspect the overall process - if not the details - will be similar. Training, discarding, training, discarding, testing, and so on. And the first truly strong AI will likely just be the first random assemblage of parts that passes those tests.

That's not how neural network training works. It's not magic or guesswork; it's essentially glorified curve fitting (over a much more complicated space than the polynomials). It's also not random in any respect. The space of all neural networks plausibly generated from a training set is very, very small compared to the space of all networks of that size.

I get the feeling that not many people in this thread actually know how AI and related concepts work.

The initial conditions of neural networks are almost always chosen randomly or pseudo-randomly. Data sets are sometimes, although not always, presented with random sample order or selection.

Either way, the randomness in initial conditions means the solution found is one of many different solutions that could have been found, and depending on the problem, different initial conditions can result in very different solutions even on the same data.

Reply on news.ycombinator.com