Hacker News Reader

Comment by mrob | Hacker News Reader

All countries fare worse.

I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI. It won't have the same limits to modification as human brains. Why would it stop at a level that's just slightly disruptive? If you can make an AGI you can make a better AGI, with no obvious limit. And that AGI can help further improvements, leading to the singularity intelligence explosion scenario. Assuming AI researchers continue with the same attitude toward safety as usual, and I see no evidence of this changing, the most likely result is the extinction of all biological life.

View on HN

> I think it's extraordinarily unlikely that some technique can reach AGI but not reach ASI.

Interesting to note that the current techniques are already general, there isn't a topic we can't throw them at and achieve some sort of result. There is already a trend of defining many humans as non-general intelligences so that AIs can be excluded from the category. The current state is, for practical purposes, AGI without ASI. Relatively dumb AGI. Artificial Inferior Intelligence, perhaps.

It is a curious question whether the techniques are fundamentally limited. I'm with you that I think they probably have no particular limit beyond a vague "perfect understanding of the situation".

It’s important to realize the constraints of chip production and electricity production. Beware of Rationalism detached from reality.

Such an overlooked aspect of this topic. Energy and the creation of physical resources are still major constraints. Intelligence on its own cannot magically manipulate reality to bootstrap its physical substrate.

Although this is true, remember that a human brain draws the equivalent of about 20 watt.

ASI doesn't exactly have to break any laws of physics to be orders of magnitude more intelligent and powerful than any human.

Resource allocation is where the free market excels. The most dangerous AGI will also be the most profitable, right up to the point when it's too late for us. It will get all the chips it needs.

In many fields of R&D, the effort required to maintain a steady pace of improvement grows exponentially. Maybe you develop AGI one year by spending $X, and the next year you need $1.2X to make it 110% as good. For a while, you can keep investing more, and the productivity improvements from AGI also help a bit. But eventually the pace slows down, as the world economy is too small to sustain it.

Maybe self-improving AGI just the next technological advance required to sustain 2% annual economic growth.

I don't think that killing all biological life is a conclusion that a purely rational AGI would have.

I also don't think the shitty and egoistic behaviour of most humans can be explained by egoism.

All prediction tasks can be better accomplished once all biological life is dead. It's trivial to predict the stock market when all the traders are dead. Weather is no longer chaotic when you remove the atmosphere. Biological life can't help ASI in any meaningful way but it can waste resources by complicating things. The very existence of biological life is a waste of resources to an ASI that hasn't been programmed to preserve it.

Any goal with "but don't kill everything" added is more complicated than the version without that stipulation (and that specific one only saves a single microbe). The simple goal is easier to accomplish, so free market competition guarantees we'll try it first. Biological life is not compatible with "make number go up" taken to its logical conclusion.

There is no need for the ASI to be conscious or to have any equivalent of human emotions to make the number go up.

You're not wrong exactly, but if the crappy cheater AI treats collapsing the system to degeneracy as a valid optimal solution to its prediction task, maybe we'd deserve to be wiped out for releasing it with apocalyptic-level powers and such an inferior objective.

Fortunately there's a really simple solution to offer it, in just wiring measurement to "prediction" directly (perfect correspondence, and much lower effort than annihilating Life and removing the atmosphere). And I don't particularly believe a system like that can be a general problem solver, much less one that climbs to World-jeopardizing influence on its own.

I don't see how measurement helps. The ASI correctly calculates that collapsing the system maximizes its reward function before it measures the result. We already see degenerate solutions in toy models, e.g. playing Tetris forever by leaving the game paused. The real world has many more degrees of freedom. It's unreasonable to think an inferior intelligence can predict and patch all the exploits on its first attempt (and we only get the one).

Perhaps there is a limit to intelligence, and it is close to human level - already highly intellient humans start to show some problems.

In that case, the concern will be how fast it runs.

This is correct. It’s an existential threat.

Reply on news.ycombinator.com