Hacker News Reader

Comment by OMFGscience | Hacker News Reader

Obviously we don't actually need a superintelligence to make paper clips, it's just an example to illustrate that an AI won't do what you want it to do if you program its motivations incorrectly. If you haven't programmed an AI to place sufficient importance on human life, and then tell it to maximize paperclip production, it could take that as its primary goal and see humans as made from raw material it could use to make more paperclips. It may know that you want the paper clips for human use, but does it have any reason to care if it hasn't been made with that as part of its goal system?

View on HN

> If you haven't programmed an AI to place sufficient importance on human life (...)

And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so.

The point of those examples being, the only AI that is safe for us is the one with values extremely aligned to our own; make a little mistake and you end up dead. "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.".

All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize these resource and learn from our mistakes.

It's a complex topic but the primary danger is with self-improving AIs. That pretty much implies there is going to be only one - the first one that starts to improve itself to be smarter than us. So it will look exactly "that our software will be progressively getting smarter, and our databases larger" up until the point some kind of critical mass is reached, and after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.

You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you). But there is no reason why this has to happen, and even if we did succeed in copying exact human values to a human-level AI, we'll likely be soon locked in an inter-species conflict over access to resources - after all, we don't care about other species living on Earth, so why the AI should care about us?

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.

That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc.

>You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you).

No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask. Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it.

> it will be smart enough to understand that it's pointless.

Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0].

[0] - https://news.ycombinator.com/item?id=8387771

His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!

You're being sarcastic, right? Since when our universe is governed by capitalist economy?

I'm quite serious. Our universe is governed by laws of conservation and efficiencies. For AI to exist, it needs power. Power must come from somewhere in the universe and, at least here on earth, it has been shown that there is competition for those resources. Whether this alludes to business or not is irrelevant; it's a fact of our reality that we are all competing against each other for resources.

But competition here has absolutely nothing to do with free-market economy that we have today.

> The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!

One, AI will not necessarily have to pay for anything. It may, or it could convince us to donate resources to it, or it could take them by force.

Two, why would you ever think that it has to market paper clips for anyone (assuming anyone is left)? It optimizes for paper clips, so it makes paper clips. Marketing is a way humans bullshit each other into donating resources so that they don't have to fight for them.

Three, what shipping costs, what buying? Again, AI wants to maximize amount of paper clips in the universe, period. It doesn't want to sell them (unless it leads to more paper clips). The only "shipping cost" it cares about is how much energy it takes to reach another planet, not some arbitrary dollar value people attach to movement of goods.

My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws. The only time an AI would care about it would be if - and only as long as - interacting with humans was a best strategy for that AI to achieve its goals - which is likely only at the very beginning, when it needs to gather enough power to subdue us.

It sounds like your version of AI has a way of getting power to run itself for nothing. I disagree with that premise. Marketing is raising interest through some means. Shipping costs are how much energy it takes to get to another point in space.

> My problem with your answer is that it lifts market economy from something completely irrelevant to the level of physical laws.

I don't actually think you have a problem with what I'm saying, but I definitely do think you have a problem with the topic. It is upsetting to think about AI 'taking over' and my points only served to discuss the path by which it might or might not be able to do that. At the end of the day it will it competing for resources with us. It doesn't get to do that for free anymore than we do.

> Since when our universe is governed by capitalist economy

@temporal our universe is governed by physical reality. AI can't start converting people into paper clips without power, and resources. The AI won't be able to just go and start taking over the world building power plants, and necessary manufacturing without anyone noticing.

Even AI has limits.

We're designing the AI so it's safe to assume that the first versions of it will think like a human. We don't know how it will think after it starts to evolve but for the Paper Clip AI scenario that' irrelevant.

The AI, even the Paper Clips AI , will need some basic understanding of our world to make the clips. It needs to understand what paper clips are. It needs to understand what they are made from. It needs to understand the process of making paper clips. where to get the material for the paper clips. how to pay for that material. It has to think like us, or at least understand our thought process. It will need to know how many we need, what colors, sizes. Where to ship them.

We can't think of AI as a god. It won't suddenly know everything and understand what we want without our input. It will be limited by the resources given to it.

Even if we tell the Paper Clip AI - "Go make me some paper clips" do you really expect it to start calling banks to get the loan for the material, arrange shipment and unloading it. Will it also order the machines, call people to have them connected, hire workers to do it? If it does all of that will it also take care of sales? Marketing? Will it understand the supply and demand? To do all of this it needs to understand us, and our world. If at this point it decides that it makes sense to convert us to paper clips, then perhaps it's right and we deserve this.

You assume both that this notional AI is intelligent beyond the limits of human capability, hence 'super-', and that this same AI is not intelligent enough to realize that making paper clips for human use is pointless if it recycles all the humans into paper clips so there's no one left to use them.

Moving from the specific to the general, you share the common error of the AI philosophers, which is to assume that a system capable of cognition on even a human level, to say nothing of a superhuman level whatever that might mean, would nonetheless depend entirely on human input to define its relationship with the world and the purpose of its actions therein.

We ourselves don't need that kind of input; an adult human chooses his own path, sets his own goals, constrained only by circumstance and by the extent of his innate capacities. Why blindly assume the same won't be true of a superintelligent AI made by humans, which will be in every meaningful sense a child of humanity and as such an inheritor of the human legacy, despite happening not to be biologically human itself?

I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment. Simple Paper Clip AI (SPCAI) will not be smart enough, nor powerful enough to prevent us from stopping it when we see that it's starting to make paper clips out of humans. True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.

True AI may decide that the humans are unnecessary and remove us for other reasons. I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.

> I don't think AI will be "programmed" in a traditional sense of this word. AI will need to go through a learning process, where it will have a chance to learn all about humans, and the environment.

Given [0], I think that it's the only plausible way we can have to make it follow human values.

> True AI (tm) on the other hand will be smart enough to know that making paper clips out of humans is simply stupid.

That's exactly The Misconception the whole movement around Friendly AI is trying to debunk. See [1].

> I'm sorry but if the paper clip AI doomsday scenario is the best this institute could come up with to explain why AI could be dangerous then I'm not impressed.

Well, paperclip maximizer is just a toy example made up for explaining why general AIs are dangerous to those for whom this fact isn't obvious.

[0] - http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

[1] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis

The paperclip stuff is a hackneyed cliché, old gray goo in new nanobottles, the sort of thing high-flown thinkers do when they want to explain the meat of their high-flown thinking to the intellectual hoi polloi, who can't be expected to get around genuine complexity. The trouble is that only someone, who is so high-flown a thinker as to be completely divorced from all common sense, could possibly imagine anyone taking such a simplification seriously.

The other trouble is that these particular thinkers, who are so high-flown as to be in metaphorically geosynchronous orbit, consider this simplification an accurate statement of their basic concern. Those of us who haven't departed the atmosphere entirely, on the other hand, can see that they're assuming their notional golem to be both far smarter and far dumber than any possible human. Presumably that's the sort of conundrum you need a philosopher to untangle, which is why I'm glad they're busy doing that, instead of getting in the hair of people who have useful work to do.

Reply on news.ycombinator.com