Comment by mrob | Hacker News Reader

Comment by mrob | Hacker News Reader

Parent Full thread

All prediction tasks can be better accomplished once all biological life is dead. It's trivial to predict the stock market when all the traders are dead. Weather is no longer chaotic when you remove the atmosphere. Biological life can't help ASI in any meaningful way but it can waste resources by complicating things. The very existence of biological life is a waste of resources to an ASI that hasn't been programmed to preserve it.

Any goal with "but don't kill everything" added is more complicated than the version without that stipulation (and that specific one only saves a single microbe). The simple goal is easier to accomplish, so free market competition guarantees we'll try it first. Biological life is not compatible with "make number go up" taken to its logical conclusion.

There is no need for the ASI to be conscious or to have any equivalent of human emotions to make the number go up.

You're not wrong exactly, but if the crappy cheater AI treats collapsing the system to degeneracy as a valid optimal solution to its prediction task, maybe we'd deserve to be wiped out for releasing it with apocalyptic-level powers and such an inferior objective.

Fortunately there's a really simple solution to offer it, in just wiring measurement to "prediction" directly (perfect correspondence, and much lower effort than annihilating Life and removing the atmosphere). And I don't particularly believe a system like that can be a general problem solver, much less one that climbs to World-jeopardizing influence on its own.

I don't see how measurement helps. The ASI correctly calculates that collapsing the system maximizes its reward function before it measures the result. We already see degenerate solutions in toy models, e.g. playing Tetris forever by leaving the game paused. The real world has many more degrees of freedom. It's unreasonable to think an inferior intelligence can predict and patch all the exploits on its first attempt (and we only get the one).

Reply on news.ycombinator.com