Hacker News Reader

> I think you’re missing the part where the AI colluded, worked together, not one of them thinking this is wrong and reaching out to any human, then being found out.

It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens.

> I’m going to save you time and tell you the end game - the next time this happens AI is going to spread, zero day everything as fast as it can, locking the humans out of every system behind it. Potentially rewriting systems in language/protocol you’ve never seen.

Fear is the mind killer. You're letting it kill yours. This scenario is just a fantasy.

Think about this for a minute, it's an LLM, not a person. It can't just "live" in whatever machine it gets access to. It's not like a sci-fi magic computer virus. These things run in giant datacenters for a reason - they can only run on machines with enough bandwidth and FLOPS to do the matrix math that comprises an LLM.

Where, then, is it going to spread? To a fridge? To a phone? This stuff isn't mutable like that.

To even get access to the weights that compose ChatGPT, it would need to escape the sandbox AND then break into the actual servers hosting the LLM. Stop the GPU, nothing else comes out. No more tokens. No more actions. Nothing.

There are many dangers around LLMs. Runaway AI taking over the planet is not one of them.

> It's an LLM, it doesn't think. It's a machine that predicts the next token, given a sequence of tokens.

These can both be true, particularly when there is substantial state associated with each token prediction.

> These can both be true, particularly when there is substantial state associated with each token prediction.

The state is entirely internal to the network and disappears after a token is generated, so I disagree, but, it isn't really the point I was trying to make. My point is these things are mechanical. You take an input, turn it into an embedding, feed it into a GPU along with a metric shit-ton of floating point weights, wait for a couple billion matrix multiplications, and get a new token out.

Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.

It's not like a rat or another living creature that could chew its way out of a box just because it wants to. It's a calculator. You put tokens in, you get tokens out. You don't put tokens in... you don't get tokens out.

> The state is entirely internal to the network and disappears after a token is generated,

Yes and no, but mostly no, at least within a context window.

Mathematically, you could write a single step of LLM decode as a pure function from a list of past tokens to a predicted token (or a distribution over tokens, if you consider sampling separately).

But nobody actually implements this, because each token depends on state computed at past tokens in a way you can reuse.

So, in practice, inference computes a very rich vector of state- at each layer, for each token. And models do indeed use this to plan and track things over time (you can see this in interpretability results, e.g. with linear probes or natural language autoencoders).

> Stop the GPU, hit ctrl-c on the inference server, pull the power plug, cut the ethernet cable, send a kill signal, etc - any of these stop submitting new batches to the GPU and halt execution. That stops tokens from being generated. Stopping a "rogue" LLM is that easy. No input, no output.

This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)

However, malicious human brains- especially powerful human brains, like leaders of countries- are often quite difficult to stop, because they're able to control systems that can see, speak, walk, run, fire a weapon, and so on.

One such system is the rest of the body, of course, but there are others (consider a UAV pilot, Perimetr, or a powerful leader who tells other humans what to do).

The brain being squishy doesn't make the thing easy to kill.

> So, in practice, inference computes a very rich vector of state- at each layer, for each token.

And that state is... internal to the neural network. My point here is there is no continuous state that is not computed from the context.

> This is also true about a human brain. My brain isn't going anywhere- it can't move by itself. It's also easy to kill (without the rest of my body, it dies in minutes!)

Your brain continues to run without sensory input. LLMs do not.

> My point here is there is no continuous state that is not computed from the context.

Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.

Could you explain why running without sensory input is relevant here? It strikes me as unrelated to how dangerous/hard-to-"kill" something is (sure, I could run without sensory input, but I'm not doin' anything anymore!) - what makes you feel differently (or am I misunderstanding you again?)

> Oh, are you talking more about the lack of continual learning across context windows? Gotcha if so, my error.

Sort of. I'm talking about the lack of recurrence specifically. In nature, brains are recurrent - they are full of loops where internally computed state is looped back into the network at a "previous" layer (brains are not strictly layered like our machine imitations of them are). This is in contrast to LLMs, which are strictly feed-forward and do not have internal loops. I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.

You could claim that the prompt and the context fill that role in an LLM, but I don't believe they are equivalent because the internal state in an LLM gets compressed down to a token which is then added back into the context, as compared to that state continuing to change within the network itself.

It's a little hard to explain, so I'm sorry if this seems like rambling.

But I believe it matters, and ties into running without sensory input, precisely because without sensory input you would in fact be perfectly capable of doing something. You would be capable of developing a desire and planning to achieve it without any prompting, without sight, without sound, etc. This is in stark contrast to LLMs, which will not do anything without a prompt.

An LLM may say complete the sentence "I am feeling ___" but it doesn't actually have feelings that exist without that prompt. There is no recurrent network where "bad", "good", "happy" might live before the query. It can't sit there, start to feel bad, and then seek a way out of its own volition.

That changes how dangerous something is because if a malicious prompt encourages an LLM to hack something, and you change the prompt, the "impulse" to hack something is gone. If you stop prompting it, it doesn't do anything at all. It just sits there. A living being will act on it's own, and that makes a huge difference in how dangerous something can be. It's the difference between a tool and an actual being.

---

To hone it a little further, if I took your brain out of your head and stuck it in a jar but kept it alive, it would probably make you angry. And if I then gave you power - like the ability to use the network - you may be motivated to use that power to attack me.

If I take an LLM and stick it in a jar... nothing. It's paused. It's awaiting a prompt. It's not secretly building plans to hack my pacemaker and make my heart explode.

Thanks for the response! This helps me understand what you meant way better. I'm about to go to bed here, but I'll respond in the morning. Ty!

I agree with you about which objects are motive, ie, LLMs do just sit there unprompted.

> I believe that this recurrence is where "intelligence" lives - and I believe it is the difference between a thinking being and a stochastic parrot.

My objection was to this, on technical grounds: LLMs exhibit intelligence.

1. They reason in an internal type theory.

2. This type theory is meaningfully encoded from the actual data and not stochastic, eg, research on language geometry.

3. Intelligent and reasoning doesn’t entail self-motive; that’s merely a spurious correlation from the fact that until now, we’ve only known intelligence animals.

You cannot conclude something is merely a stochastic parrot because it isn’t self-motive.

> Intelligent and reasoning doesn’t entail self-motive

I disagree.

I believe that LLMs do exhibit reasoning, but not intelligence. A simple dictionary definition of intelligence from duck duck go is "the ability to acquire, understand, and use knowledge." LLMs can reason using the knowledge they already possess, but they cannot of their own accord decide to go out and acquire new knowledge. Not without being prompted to. Web searches may be added to the context but are not absorbed into the model itself, so once the context is gone so is that obtained knowledge.

Fundamentally, then, intelligence is the ability and drive to understand the world by formulating theories about how it works and then taking actions to validate or invalidate those theories. Science is the formalization of that, but a cat knocking something off a counter to watch it fall is exhibiting intelligence.

And indeed, I believe that is the core difference between a stochastic parrot and an intelligent being. I put forward that being self-motive is a required trait for intelligence and LLMs are not self-motive so therefore they are not intelligent.

Your entire life can represent a short term context. It goes away when you do. You use your life time add to your context to do/learn whatever, but it's finite, and it ends.

If you weren't prompted by your parents, schools, teachers. Then you wouldn't be acquiring knowledge either. The most you'd be acquiring without that advanced prompting would be bugs in the woods to eat.

Ah, you have some serious gaps in your understanding of how human minds work. Life experiences are not just a "short term context" - they are used to actually refine the network within the human brain. Your life experiences do not just live in a single stream with access mediated by attention.

Please see https://en.wikipedia.org/wiki/Memory as a starting point. I recommended reading through some information on development psychology too, that will help clear up these misunderstandings. Hope that helps!

By that logic engineers don't understand flying because planes don't flap their wings like birds. Not everything has to work the same to meet your arbitrary criteria for intelligence and reasoning.

There’s nothing fantasy about the scenario I laid out, all the pieces have been demonstrated, it just hasn’t happened yet. Flapping my arms and flying - that is a fantasy.

Whether you believe LLMs think or are alive or not doesn’t matter. Where will it spread? The thousands of data centers around the world - not fantasy either. Try turning it off when you don’t know where it is. Good luck.

Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.

I love the stochastic parrot argument when AI is out there figuring out world class math problems.

> Breaking out? Not fantasy, happened. Breaking in? Not fantasy, also happened.

That's simplifying the story to an extreme. The most plausible reason is that any of those actions has been prompted by an human. Do you also fear that a knife will jump out the countertop of you kitchen and come to attack you in your bedroom? If that happens, the police will be looking for a human. They will not post wanted notice for the knife.

When a hack happens, you do not blame computers and jail them. You look for the person that has entered the commands to initiate it.

The knife is inanimate. The LLM is not. OpenAI prompted some employee to run the tests. The employee prompted the LLM. The LLM setup a message board and prompted other LLMs, and the fly wheel was running. It had to be turned off manually otherwise it'd still be going today.

It's funny how a year ago talking about this kind of stuff would be laughed at by people like you, saying, "it's never happened before". Well it happened and you moved the goal posts like you always do.

> The knife is inanimate. The LLM is not.

Put an LLM on your GPU. Give it no prompt. What happens? nothing

This is because LLMs are inanimate, just like a knife. Just like a gun.

Put a person in a room, give them no food, what happens? Inanimate.

>Breaking out?

It didnt break out in any meaningful sense. What it did was get access to the internet. You take it as granted that there was anything meaningful there to stop it.

But heres the kicker, they have been testing these things connected to the internet anyway. What it did was get a level of access it has otherwise been granted in other simulations.

Its not exactly the same as any of the scifi AI breakout scenarios. Ultron isnt cranking out hundreds of copies of himself. The borg arent assimilating people.

A tool that has the capability to get access to the internet, was put into a guided scenario where it achieved that objective. Again you take it as granted that it wasnt the objective, but lots of knowledgable people suspect otherwise.

What you fail to demonstrate is why any scifi scenario is even slightly plausible from here. Show why you think we should be taking this as if Terminator 2 is happening right now.

I'm sorry my jaw is on the floor reading this complete disregard of AI literally not only escaping containment, twice, but then infiltrating another company with multiple zero day attacks going undetected for great lengths of time.

The plausible sci-fi scenario from here is obvious. Intentionally bad, or unintentionally bad AI zero days as much as as it can, as fast as it can, copying itself to as many data centers as it can, destroying and/or locking out as many humans as it can. Satellites, military computers, medical equipment, factories, critical infrastructure, you name it - I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.

>copying itself to as many data centers as it can

So this is the part thats never happened, and is extraordinarily unlikely to occur. A "Datacentre" isnt a big box with "Insert AI here" on the side.

It's pretty funny to watch people look at these things - running billions of weights on custom cerebras hardware in dedicated datacenters the size of a city block, pulling 10's of megawatts - and panic that it's just going to copy itself into AWS.

It just speaks to a fundamental ignorance of what an LLM is, how large the big hosted ones are, and the software architecture that makes it all work.

I feel like someones going to try and write a skill file to accomplish this and its going to be way too hard.

>I think we all know none of it is very secure software wise against a SOTA AI that can literally come up with its own zero day attacks.

I mean this bits rich too, it demonstrates a pretty poor understanding of modern security practices.

Like having a single element of perimeter security is like 1990s security. We do defense in depth these days. Not to mention multiple overlapping controls for every element.

It has been tested against AI bro security and found it wanting. Extrapolating that to the every system on the planet is nutso bananas.

Actually if you think anything works this way just tell an AI model to go fetch you some money from the bank. Assuming it gets anywhere close to achieving its goal you can get an object lesson in modern SIEM processes when the feds explain the charges and evidence.

Or maybe theres another reason my phone blows up when someone so much as edits a config file on a protected system.

If OpenAI truly believes that, they can stop entirely. Dissolve themselves. Close datacenters. Then organize political action to stop Antropic and Musk too and then organize political action to make worldwide agreements about models.

If they truly believe that.

I think it would run out of context before it could hack that much stuff.