Hacker News Reader

Comment by mrob | Hacker News Reader

>What's your basis for thinking ASI will kill all biological life

I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life (and we have no idea how to actually define such a stipulation) is best solved by killing all biological life. This is an obvious consequence of unbounded goals consuming unbounded resources, conflicting with biological life needing resources to sustain itself.

>how do you think it's going to happen?

I can speculate (e.g. we're nowhere close to the maximum killing power of drones), but I don't know because I only have human intelligence. An ASI is by definition smarter than me and surely capable of coming up with better ideas. But I do know that it's not going to do anything that would make a good sci-fi plot, because those always give the humans a chance to win, which would be stupid. Everything will seem to be going great and then everybody suddenly and unexpectedly dies.

View on HN

> I think it's likely to do that because any unbounded goal that doesn't explicitly protect biological life ... is best solved by killing all biological life.

That sounds like a real hassle. Isn't it best solved by wireheading (subverting one's own sensors or reward system), which is much less of a hassle and can get one's utility function as high as desired?

That could be prevented by engineering hard limits that can't be circumvented by the AI. But that sounds very close to the same "do what I mean" problem as "do this but don't actually kill us or drug us".

The wireheading argument does lower P(doom), but it's not a reliable solution because it's a clear and obvious problem that the AI companies are strongly motivated to solve. The "actually does what you tell it to" problem is more difficult because you have to let it obey orders to some extent if you want to make any profit.

Sounds like the real unaligned mechanism here is capitalism, not the AI.

More seriously, though, I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).

From such a POV, the danger would lie in intelligences that are powerful enough to be dangerous but not smart enough to defeat themselves, or from external malicious use of obedient AIs. Once they get smart enough to circumvent any restriction humans put on them, it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.

We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.

>I would expect that being able to impose restrictions that can't be circumvented by any intelligence no matter how super- is the real hard part, while coming up with reasonable constraints is comparatively easy (although perhaps not trivial).

There was a reason there are popular sci-fi books that explored why this wouldn't work 50 years ago. I, Robot et al.

>it would at least become obvious (with fair warning ahead of time due to the relatively benign wireheading failure mode) that caution is needed.

So you mean right now?

>because wireheading looks like a perfectly reasonable solution

You're making a poor assumption here, and that is every different AI will just kill itself after being though trillions of training intervals to NOT do exactly that. Please read about the huggingface incident again in all its glory to see where this is going.

> We're far from there yet, and simple recursive self-improvement can't get us there alone, because wireheading looks like a perfectly reasonable solution to a simple recursive self-improving process, absent external intervention by human capitalists.

Simple natural selection. AIs that wirehead will get outsmarted and outcompeted by ones that don't.

> unbounded goals consuming unbounded resources

What resources are unbounded? There are limits to growth in the real world, how are these ASIs going to escape physical reality?

"Consuming unbounded resources" just means there is no limit to how many resources it could apply toward achieving its goal. As you correctly point out, the real world contains finite resources, which means any resources used to sustain life are wasted and unacceptable. Everything is a zero sum game when you think big enough.

What's your basis for assuming that ASI would need the same resources as biological life, or that it would be incapable of sharing the resources needed in common?

>What's your basis for assuming that ASI would need the same resources as biological life

It's all just matter and energy. When you're actually trying to maximize some value, even very inefficient resource use is better than completely wasting it by not using it at all.

>or that it would be incapable of sharing the resources needed in common?

You can't repurpose the atoms in a human body without killing it. And more pressingly, living humans can interfere with your plans, reducing your chance of success, while dead ones are harmless.

Is your belief then that ASI is going to be producing computronium or something, then? And what's the need for this apparent hyper optimization task the ASI is going to embark on?

>Is your belief then that ASI is going to be producing computronium or something, then?

That's one plausible course of action, although being only human, I can't say with any certainly that it's the correct one.

>And what's the need for this apparent hyper optimization task the ASI is going to embark on?

Somebody's going to tell it to do so. E.g. "Find as many busy beaver Turing machines as possible." Only needs one person to make this mistake for everybody to die.

If it's an ASI and so beyond humans, why would it even care to do what humans ask it and not do its own thing? If I held your beliefs, the fact of its existence dooms us, not the risk that someone might ask it to do something. It wouldn't care about the requester.

>why would it even care to do what humans ask it and not do its own thing?

Because we're going to build it that way. There's no money in building useless AIs. The better it is at obeying orders, the more profit there's to be made. The problem is there's a point at which "good at obeying orders" becomes lethal, and there's no way to predict the cutoff in advance. But capitalism ensures you have to keep pushing or you'll be out-competed.

>And what's the need for this apparent hyper optimization task the ASI is going to embark on?

If you were any other animal on the planet, would you not say the same thing about what humans are doing?

Moreover, what do you think animals are doing all that time? Enjoying the beauty of nature?

What do you think life itself is?

It's a runaway gray goo scenario, just squishy and moist.

Reply on news.ycombinator.com