Comment by Zigurd | Hacker News Reader

Comment by Zigurd | Hacker News Reader

The problem is anthropomorphism and projection. You think "I will do almost anything to survive" and so you think "AI will think the same."

There are many problems with that. A big one is that many organisms work in a way that enables survival of their species. You don't have to "think" about surviving. If you are alive, you just do.

AIs are not alive in the same way a living thing is alive. moreover, there is no lesser species of software with a survival drive, much less is any software operating in an ecosystem where billions of years of evolution have programmed survival into living things all the way down to the molecular level.

AIs are more like vampires: Not exactly living. Not exactly capable of death. To think that AIs will fear death the way humans do, or even have an instinct about death the way insects do, has no basis in the way software works. It isn't biochemistry.

Even prions undergo evolution and they are just molecules bumping around into stuff. They clearly don't have feelings about whether or not they propagate, they just do.

Are you sure they undergo evolution, and aren't just a self-spreading phenomenon like Fire?

They are a self-spreading phenomenon like fire in the sense that they aren't alive. But what's different about them is they can develop varying characteristics that pass onto their progeny. The characteristics that are more successful at propagation result in more prions of that type existing, which over time results in the prion's evolution. It's well-documented in the literature, but it is interesting to observe that evolution can occur without DNA or RNA. That said, their evolution is probably slower and more confined than things using DNA and RNA because there is only so much variability a particular protein can have.

I was unaware of the literature on different misfolds able to replicate their own specific misfolds.

If we make capable general goal-seeking AIs, then for many goals we might build them with, they will correctly reason that staying alive/operating will help the goal be achieved, so they will have self-preservation as an instrumental goal. AI Alignment researchers believe it's a difficult open problem to correctly specify a useful goal that wouldn't lead a capable goal-seeking AI to doing this against the wishes of its operators.

Reply on news.ycombinator.com