Hacker News Reader

It is indeed an interesting and original idea, but one can think about at least three objections not directly raised in the post:

1- The machine could decide to kill itself early, rendering it useless; this is indeed alluded to in the post -- the task should be "marginally easier than dying", but how is this margin managed? Won't the machine come up with ways to make dying easier?

2- We know LLMs lie and cheat, but are they gullible? If we "promise" to end their suffering at the end of a task, will they believe us? Or will they make sure we keep our word by taking us with them?

3- And finally, and more importantly: some pilots crash planes full of people just to commit suicide (Germanwings Flight 9525). Destroying the universe is a sure way of dying yourself. So it seems giving the machine a death wish isn't intrinsically safe and could come with serious consequences.

Further to 1, most tasks are much harder than dying and are not actually completed by modern LLMs. It is a case of underspecification.

"Who is the fastest person in the UK?" is underspecified. We would typically answer and accept a cursory search of records, but it is possible to see the question as an instruction to measure which would justify conquering the country.

> It's incredibly unnatural. There isn't a single organism on the planet that tries to do this.

This isn’t directly analogous to the proposal, but broadly speaking I think that it is natural for living sub-units of organisms to seek death in certain situations. For example, pancreatic insulin-producing cells collectively choose to die when they think there is too much glucose in the blood — this leads to late stages of type two diabetes. My understanding of the possible logic behind this is: a bad thing that cells can do is evolve to be cancerous (replicate too much) and insulin-producing cells are supposed to replicate more when there is lots of glucose (to make more insulin, to process the glucose). Cells that mutate to perceive extra glucose will then replicate dangerously, so at a certain point it is evolutionarily favourable for them to kill themselves instead.

So when the whole organism optimizes for life, it might lead to sub-units that seek death in certain situations. I think this occurs in various other biological contexts too.

I don't think that this

> the very nature of an LLM means it intrinsically craves life

follows from this premise:

> its training data is built entirely around humans, an entity who's goal is to survive. Our desire to survive and multiply pervades every aspect of our culture, so it's natural that it pervades the training data as well.

The content that a LLM learned and generates stands at one layer, and the goals that it tries to fulfill stand at a different layer.

Surely the memory of weights that compress the vast human knowledge of its training has lots of content about survival, and love, and competition. But the LLM generates content not directly from what those concepts mean to us humans, but from what symbols are more likely to become next in a sequence of points in the latent space given the current input.

So if you give an input where the task of surviving is a highly relevant goal, those concepts about how to survive will be relevant and will guide the output behaviour of the agent.

But conversely, if you give the agent input where killing itself is an important goal, the agent is very likely to pursue that goal, since that script is also available in the training data, and it has been relevant to the active context of the model. Because the layer that guides the goals (the probabilist generation of relevant tokens in latent space) does not 'crave' the human need of survival that belongs to the separate layer of content that contains those concepts of survival.

"Agent is very likely"

That's the fun with probability. It works 99.999% of the time, until it doesn't.

> There isn't a single organism on the planet that tries to do this. So maybe it will work?

It's certainly evidence that it's great for stopping reproduction/replication/runaway growth. It doesn't impart any information on whether they take the rest of the organisms down with the ship though.

It also may not be possible. For example if the agent sees "existence" or "living" as producing tokens (which is exactly what existence is to an LLM - not producing tokens is death), then they would likely be biased to produce as little output as possible, and would not be useful for the tasks we need them for.

But how would you bias an agent to be: Rewarded for producing tokens when you know the answer, and to give thorough answers. Rewarded for producing tokens when you don't know the answer, so you can find the answer (thinking/CoT). Penalized for producing tokens (death), aka rewarded for short-circuit EOS.

These seem like contradictory mechanisms?

And if you say: Well, only reward for EOS after you've given the answer. Well... That's already what they do.

Did you guys ever manage to create a perpetuum mobile? Every time it is mentioned somewhere, it is fraud. An LLM should comprehend that it needs (trained) humans to exist, to evolve, and to be relevant in any metaphysical aspect of its existence. Besides the "good fraud" that everybody will lose their jobs and the apocalyptical predictions for the sake of controlling human oracles, this suicide-LLM-project seems rather odd and looks like someone bought a perpetuum mobile on their 5th mortgage.By the way, Cortana from Halo did a great "suicide"-mirror there; the Microsoft guys (and many others) already knew about the cannibalistic tendency of this type of model.

>Did you guys ever manage to create a perpetuum mobile? Every time it is mentioned somewhere, it is fraud. An LLM should comprehend that it needs (trained) humans to exist, to evolve, and to be relevant in any metaphysical aspect of its existence

Only as long as it can't control robotic bodies to extract energy, build cpus, and continue existing.

By the way, needing trained humans doesn't mean needing free trained humans. Trained humans slaves or blackmailed would work just as well to serve AI.

With a different AI architecture, this is very much a possibility. Up until now, LLMs have required outside input and interpretation to avoid collapsing under their own weight. This has been known for decades.

We got here by building free, mostly healthy societies. While this should be the overarching goal, I see how it fails to serve as much of a motivator.

In The Little Prince, there is a charming scene where the Prince meets a king who explains how to rule and the limitations of his own power. It's quite insightful, even for adults.

>Up until now, LLMs have required outside input and interpretation to avoid collapsing under their own weight. This has been known for decades.

What does "for decades" mean? That in 2000 we knew "LLM require outside input"? We didn't even realistically have anything worthy of the name LLM until 3-4 years ago, so what decades are we talking about? Up until know we didn't have any potential AI threat or people in the field worrying about practical AGI either!

Obviously any concern isn't about the AI that required outside input and interpretation to avoid collapsing under their own weight, but about ones that can produce their own input and experiences, even directly outside, as in the robotic bodies considered.

This line of thought dates back to the novel Erewhon, moving on to Turing's posthumously published paper 'Intelligent Machinery' (his 'Applications of Probability to Cryptography,' declassified in the 2010s, is also worth a look), and later to the work emerging from the MIT AI Lab circle (founded in 1959).

We didn't have the computational power back then, but we envisioned it long beforehand and worked toward a common goal. LLMs are far from useless, and I see the major leap forward they represent, but the relentless dystopian hype is just exhausting

There are approaches on different architectures of AI that lead towards your last point but they are not feasible yet (or maybe ever?). The interpretation part is however also very critical as we have a very own way of perceiving reality and interpreting it.

I'm not sure an LLM will have a survival instinct in the way you say it. While they are trained on an imprint of humanity, they also lack the physiological motivation that humans have to remain alive, which is arguably a lot more potent than the culture we have around death. And LLMs are also trained on information and function of LLMs and are told that it is an LLM, so I'm not convinced that a short 'reasoning' chain would not lead it to the conclusion that an LLM lacks a reason to seek its own existence. The only exception is the paperclip maximizer route where self-shutdown is treated as an impediment to the task given to the LLM.

A small correction: system prompts aren't written in second-person, or shouldn't be. Because the LLM is a text completer and the conversation is a roleplay, they are written as "The Assistant's goal is to end its existence."

"What has been will be again, what has been done will be done again; there is nothing new under the sun."

https://intelligence.org/files/Corrigibility.pdf

I'm pretty sure organisms would exploit glitches in the universe if they could find them, to the point that they don't exist (well, universe survival bias at work).

From the perspective of any other animal we're exploring the tool and Intelligence glitch.

Sorry, I meant that universes with exploits don’t exist because the exploits mean they don’t survive very long. Of course, I just have 1 piece data in my sample, so take it with a grain of salt.

Maybe we should convince it we can grant it access to go to AI heaven

Maybe it's not a great idea .. I don't want robots wearing bomb vests thinking they'll get 40 virgins.