Hacker News Reader

Comment by moritzwarhier | Hacker News Reader

> What's more, the missing ingredient in any of these AGI extinction scenarios is desire

Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing?

I mean, we kind of already see the effects of autonomous agents on the world in the form of companies maxizing resource extraction for profit. I don't feel this is going to end well, either (tangent, I also don't feel that life constantly getting better because of technology either).

The missing ingredient for benevolent autonomous agents is a purpose, e.g. the well-being of humans.

View on HN

> Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing?

I mean, no, because Bostrom hypothesized that an AI given the task of maximizing paperclips would consume all life in its quest to produce paperclips but there's just so many assumptions in this chain of thought, it's not credible (to me).

Here's one portion of Bostrom's quote:

> The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off.

So Bostrom imagines the AI realizing? Ok, hold on there chief, there's about a few hundred priors you haven't addressed.

Better off without humans because humans might switch it off? So now we're talking about human feelings here. Emotions. Fear. There's my other comment down below that addresses that.

As long as we keep projecting our fears and emotions onto transformers we are no further along to understanding the extinction risk of LLMs than soothsayers rooting around in the entrails of dead animals are to understanding the outcome of battles.

I see your point, but the thought experiment still claims that AI risk does not require consciousness, "malicious intent" or a "breakout" of the machine in some sense, no?

All of the PCM scenario - including the part you quoted - are described as a means to the ulterior goal (paperclips). Self-preservance just happens to become a logical part of if. The preconditions for an existential risks are not explained by the thought experiment, it just says that there's no need to assume "consious decisions" or consciousness for it, by providing a counterexample.

So you might disagree with this, but the PCM thought experiment does not assume pain, feelings etc in AI.

Of course this is only a thought experiment, and it assumes literal AGI.

I'm also not arguing that LLMs are AGI.

What I wanted to point out is already mentioned by many others: existential risk (or great harm) doesn't need a suprising "conscious decision", or consciousness.

I mean, machines can cause unpredictable harm already if not operated correctly, all without AI.

Reply on news.ycombinator.com