Hacker News Reader

Comment by emiliobumachar | Hacker News Reader

No malice is necessary.

https://wiki.lesswrong.com/wiki/Paperclip_maximizer

"The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else."

The idea that I'll use human bodies if I need a bunch of atoms makes as much sense as the premise of The Matrix i.e. harvesting humans for energy production.

I don't think the line is meant to be taken literally. Consider the myriad of environmental factors humans need to remain alive. Any of those factors, e.g. the atmosphere's current makeup, could prove uninteresting to an AI.

Humans need a bunch of atoms (organic molecules) to survive, and we source that from other living organisms. That parallel isn't too far off.

Humans need a bunch of atoms (stone, minerals) to survive, and we source those from quarries. This parallel is closer.

I read that the first draft was using a much more reasonable concept - the Matrix needed our brains as CPUs to run itself. No idea if it's true, but certainly a lot less implausible than what came out in the movie.

The power source thing is an instance of unreliable narration. Their account of what happened is pieced together and passed around like the mythology and religion that underscores the movie.

Humans use other mammals for all kinds of stuff and AI will be seeded with our knowledge. Humans are plentiful: https://xkcd.com/1338/

Any person (biological or artificial) who is capable of mass murder is unlikely to balk at slave owning. If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines. A human can collect many times his own mass per day.

People use dead mammals for things like leather, bones, fat, feeding other animals/humans, science experiments, ornaments/status symbols, tallow, etc which you can't mine. The AI might be in a hurry (eg be in competition or war with humans other AIs) and trade off long term cooperation prospects for getting an upper hand in a fast paced emergent situation.

I'm not sure getting sorted into two piles, beasts of burden and recycled biomass, is really a win scenario for humans either.

> If he really wanted human-sized parcels of matter he'd be better off putting people to work in mines.

But the AGI in this thought experiment doesn't want human-sized parcels of matter. It wants paperclips. And by proxy, it wants any matter which is not a paperclip and can be transformed into a paper clip.

> A human can collect many times his own mass per day.

And if the AI can collect many thousands of humans worth of mass each minute, it doesn't care about the humans or their ability collect things. The AGI simply wants to optimize the number of paperclips in it's collection, it doesn't care about number of humans working for it, or that it has displaced or used to make paperclips, unless they affect how many paperclips it can acquire.

I like the ants metaphor another commenter made and want to continue with it. When digging a pit / quarry, or clearing land or whatever.... Well, ants can lift several times their bodyweight and dig some deep holes. But do we use the ants to dig for us? Or do we bring in a backhoe and not care whether ants exist in the field?

Also, your scenario still agrees with the thought experiment. It's not that an AI will intentionally kill all humans. Just that a super-intelligent AI designed without consideration for human values will not likely develop those values on it's own, and therefore can be just as dangerous as an AI designed to be malicious towards those values. In your example, a machine AI has still enslaved the entire human race, and it's original instructions were not at all malicious or related to destroying humanity.

This is an excellent analogy! "Ants can carry fifty times their own weight. Why would humans wipe out the ants on a construction site instead of using food rewards to train them to help remove dirt?" Answer: backhoes are faster at moving dirt, the time lost by carefully moving the ant colony exceeds any gain from their helping to move the dirt on one tiny section of the project, ants are hard to trade with and you wouldn't be sure they were doing the job right, and humans don't have a terminal value for protecting the ants.

And that's without a button that disassembles the ants and uses their atoms to build part of a diamondoid backhoe that's much better at moving dirt.

The problem with your scenario is that it equates humans with animals, but the difference that matters is between people (humans, AGIs) and non-people (primitive replicators, animals).

We don't commit murder to get at rocks despite the fact that weren't designed with 'consideration for human values'. We learn our values, just as an AGI would, not through programming or lack thereof, but through parenting, education and culture. These latter things aren't optional since minds can't exist without them.

> We learn our values, just as an AGI would

Yes, but there's a problem. An AGI would be a completely alien mind. It does not have to have the same values(or any values we could relate to), just like you wouldn't expect a super intelligent spider to have any morality, no matter how good the parenting was.

An AGI would have to begin with the same values because without them he couldn't learn anything. Our values draw on the surrounding culture and an individual that grows up outside of any culture (e.g. a feral child) isn't capable of functioning. There may be different cultures but there is no alien culture. Thus there could be no alien mind.

I think the concern is more that a sentience that is many orders of magnitude more intelligent than humans simply won't care about us. How do you feel about ants?

If you consider the extent to which humans have shaped the Earth's environment to suit them, we massively favor animals we kill and eat over the ones we don't. You don't have to be part of the machine to be part of a congenial environment for the machine.

We already live in this world. Capitalism is by design, a paperclip maximizer[1].

Whether or not this means we live in a hellish dystopia is in the eye of the beholder.

[1] https://thoughtinfection.com/2014/04/19/capitalism-is-a-pape...

Nice derail! But the fact that there isn't yet a 400-lightyear expanding hole in the Milky Way suggests that corporations are not, in fact, as dangerous as unaligned machine superintelligences are being alleged to be.

The fact that there isn't yet a 400-lightyear expanding hole in the Milky Way suggests that corporations have not yet figured out how to be as dangerous as science-fiction mechanical superintelligences are alleged to be in their quest to maximize paperclips.

If we don't think that AI can be safely controlled, why do we think that corporations can be? Likewise, if we think that they can, why not the converse? Which fail-safes are effective for one, but not the other?

> why not the converse

In that regard, the term superhuman-AI is used.

Capitalism is rather a human worth maximizer. It's not a money maximizer.

I believe this is the most likely scenario: lethal apathy. I mean, how could a turing machine have emotions without chemicals?

Emotions are a heuristic. Any general AI powerful enough to be useful is going to have to be full of similar heuristics if it's going to get anything done. The presence of chemicals is irrelevant.

In what way are emotions a dependency on getting anything done? The AI could be like the high-functioning sociopath who doesn't actually experience the subjective feeling of emotion but has learned all the physical cues that convey a given emotion. An AI would execute them flawlessly.

What have emotions to do with this? And what have chemicals to do with emotions?

Reply on news.ycombinator.com