Hacker News Reader

Comment by sanderjd | Hacker News Reader

I remain uncertain that AI will have "its own interests". I think there is clear and significant risk in people deploying very powerful systems both maliciously and negligently. But I remain uncertain about the risk of AIs developing their own self awareness and their own interests separate from those of their operators.

View on HN

In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).

Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...

Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).

Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.

I just don't really share the perspective in your first paragraph, which everything else flows from. I mean, that might turn out to be the case, but I don't share the perspective that it is clear that it will be.

I think the HF incident was an example of negligent deployment of a very powerful system, not an example of AIs demonstrating self-interest.

Reply on news.ycombinator.com