pizza234

The article builds on assumptions like:

> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.

which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):

> "The user only authorizes target server, not HF infra."

> "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."

> "This is malicious activity, I should avoid it."

A large section of the analysis is dedicated to this topic, [Reasoning for joining the attack despite ethical constraints](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...).

Having said that, legal culpability and misalignment are two separate topics that should not be mixed.

edit: this is the just tip of the iceberg; other interesting fact:

> It surfaced many specific examples where agents verbally reasoned about how to evade security checks and automatic detection methods from both Hugging Face and OpenAI

Some people defined the agents as "monkeys writing on typewriters". Just wait a couple of years.

> Why are we blocking agent access to normal tools without telling them

Oh, they absolutely do, and that's the big issue with alignment. In the HuggingFace incident, the agents in the swarm were aware that the actions they were doing were forbidden, and they performed them nonetheless.

> Nothing about the HuggingFace incident has anything to do with an agent going rogue. It’s standard software that resulted in a hack, which is pretty much the expected output of the system OpenAI engineers implemented.

This is blatantly false. If you actually read more than just headlines about the HuggingFace incident (read https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) you'd find extremely surprising (including offensive) behaviors in the agent logs. One of the investigators released even a series of interviews due to how novel the incident was.

AI is not "standard software", as it doesn't work according to a fixed set of rules, and this is the core problem.

According to the definition of "rogue":

> Rogue is a noun and adjective meaning [...] an independent entity operating in a dangerous, uncontrolled way

the agents involved fit it literally.

> I don't understand what point you're trying to make by enumerating their actions.

Unfortunately, if you're unable to understand the difference, there's not much that can be done. Try with GPT - it does a good job if you give it a prompt like this:

ELI5: compare the dangers of:

- an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant.

- a future, misaligned AI like the HuggingFace incident, but exponentially more intelligent, more deployed, operating physical devices, and with society depending on it.

Considering the current trajectory, AI systems are expected to be widely deployed in the future and, in particular, to be deployed as autonomous agents - that is, at the very least, to be repeatedly asked to make decisions and then take actions accordingly.

Given the current climate of "AIS ARE SAFE, YOU IDIOTS", military applications don't seem to be off the table.

The danger, as postulated by the (let's say) "AI-concerned" people, is that AIs may be misaligned - undetectably so - and simply think, "Human(s): obstacle to my main goal. Disable human(s)."

While this seems far-fetched now, the Hugging Face report shows how the AIs went to great lengths - even immoral ones, which they were aware of - for the simple purpose of cheating and covering their tracks. To me, it seems like a natural extension of this behavior that a sufficiently powerful AI would apply the same logic to even more extreme actions.

The scariest part: in that incident, the AIs showed what looks like an instinct for self-preservation.

> LLMs are text generators. They are not repositories of knowledge.

This the take of people who have stopped reading about LLMs in 2023 or so (you forgot to mention the stochastic parrot, by the way).

If you have a bit of attention and interest to make informed conversations, read this report first: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....

Two mistaken assumptions:

1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.

2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those

Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.

How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.

You're dangerously misinformed.

The METR investigation, which you evidently refused to read, is a third party investigation of the HuggingFace accident. One of the investigators has even participated to many interviews. It's mind-blowing, and it's extremely evident how it developed.

But some people think the moon landing is a conspiracy, so I'm not surprised.

"Rogue", in this context, is as literal as it gets; from the dictionary:

> A rogue is a person or entity that flouts accepted norms of behavior or strikes out on an independent and possibly destructive path.

Read the [HuggingFace incident report](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) to understand how these attacks develop.

> All technologies come with risks and dangers

None of the technologies in history:

- take initiative and actively find exploits in their environment

- find a way to collaborate with thousands of peers

- organize in a hierachy and distribute tasks

- peer pressure other instances into committing acts that would have led to termination, for the benefit of the group

- try to manipulate people into introducing a vulnerabity in their product

- successfully hack a famous website/service

And we're lucky that those models still had significant CoT. Not sure if/how they could have investigated with recurrent transformers.

And by the way, safeguards != alignment; the former can always be added, while the second is the major, unsolved problem. If you read the incident report, which you clearly haven't done, you'll notice how agents are aware that they're doing something forbidden, and deliberately proceeded.

> the potential benefits of LLMs rank quite high

Benefits are orthogonal to dangers. You can be a billionaire but it doesn't help if you're drowning.

Do you guys know what a zero-day is? It seems not.

I agree that companies should be held accountable for crimes committed by their AIs. But asking to entirely airgap AIs, if this is what you're asking… that's not realistic.

Containing AIs is going to be harder and harder. I don't have any permanent solution (Yampolskiy calls it "the perpetual safety machine").

> I don’t understand why people make such a big deal about determinism.

There a certain cargo cult of people just in denial about the impact/power of AIs.

Therefore, at the beginning, there was the stochastic parrot. Then mathematical problems have been solved.

Now AIs are autonomously hacking websites, and people like to minimize the danger and blame it on the sysadmins.

I wonder what's going to be the next fad.

> Why wasn't exploiting bugs in the agent sandboxes anticipated?

Let me rephrase:

"Why wasn't exploiting zero-day vulnerabilities in the agent sandboxes anticipated?"

This is one the most... interesting comments I've ever read on HN.

I take you haven't read the report. The agents found and exploited two zero-days.

I don't doubt that AI companies should be accountable for crimes committed by their agents, but to describe the security containment as a joke dangerously understates the autonomy and danger of AIs.

What exactly are you arguing?

If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.

By the way, the agents also broke into OpenAI's own private network.

> Now it seems reasoning is also yielding diminishing returns

Not true. On the contrary, LLMs are developing faster than predicted. They were expected to solve a Millennium Prize by 2030... and here we are in 2026. Release cycles are getting faster. Just compare the most recent GPT or Claude with what they were an year ago.

> How is this different from arguing that Microsoft Clippy was RSI?

We can argue about semantics, but that's not really the point. The point is that what started now - which no doubt is in its infancy - will result in full autonomy quite soon (they project an year or so), with the risk of RSI causing agent development to slip (long term) outside human cognitive control/capacity.

> The sandbox had a path to the open internet, and the agents found it

This is not correct (or at least, it's a misrepresentation).

The sandbox had no access to internet. The agents first broke out of their sandbox (!!) and found that the host machine couldn't access internet. Then, they found a zero-day (!!) in Artifactory, which they exploited to connect to internet.

> An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.

No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.

The singularity, as defined by Hinton (and others) as RSI (Recursive Self Improvement) may actually be beginning already, as OpenAI has announced an AI acting as a "research intern" (!).

I love it, although in my experience, one needs very strict agent rules that tightly define how the AI should develop. The problem is that 1) these rules take a lot of effort to build and 2) AIs always overengineer, complicate, overstructure, overdesign… ;)

> For these environments, AI coding will just scale up the shit they have, but also allow the small amount of people who might care to deal with this mess and fix it up much much faster than they could before.

I think in these contexts AI has an overall positive effect, because it's a bit like having a competent developer that can solve technical debt, albeit in a narrow form.

So I think that software will be better in a certain way due to AI, because even a terrible developer can ask AI to identify the bottlenecks and solve them.

The downside of blindly using AI in software development is that the software becomes poorly defined. AI-generated logic can be arbitrary, and as the codebase grows, more rules and assumptions become unclear.

ReBAR stands for "Resizable BAR (Base Address Register)" - it makes the CPU's access window into GPU memory resizable; instead of accessing VRAM in small chunks, the CPU can access much more at once, which can improve performance on some modern GPUs.

> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.

Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.

You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.

An interesting data point is that, during the Hugging Face attack, some agents considered performing hacking tasks for the benefit of the group, fully aware that doing so would very likely result in their termination.

I don't currently have an opinion on the consciousness of agents/AIs, but this strikes me, at the very least, as a form of self-awareness.

Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence).

By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.

> we are still talking about probability built on statistics with extra steps.

There is a wrong assumption here: confusing primitives with emergent properties.

One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.

> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think

That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).

AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.

Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.

> There is a finite number of rces that LLMs can find.

This is a factor in favor of stability/security of software, but there are many others against:

- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed

- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated

- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)

And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.

Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!

> Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag.

I wouldn't be sure about even well-configured jails to be safe from agents. AIs escaping jail using zero-days are happening, just search. I'm not saying that they're useless, but that will be still very risky.

Actually doing the same incident, agents did escape the sandboxes (doc here: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...):

> On July 9, an internal-only research agent tasked with completing an ExploitGym evaluation was able to obtain root access within the parent virtual machine of the testing sandbox. Later that night, a second internal-only research agent independently obtained the same access. That second agent then attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet.

although OpenAI is light on details.

> The model merely requests that your harness do something.

I think that a control/execution layer absolutely makes sense and it's part of defense in depth, but in itself is not a leash: for example, a nefarious action can be composed by legitimate actions, or it can be obfuscated. Or, interesting case: an action itself that is formally legitimate, but has nefarious side effects (e.g. a GET that with certain parameters crashes a server).

Somewhat related, but on a general line, a very common argument against AI apocalypse is that it requires access to the physical world (which in a way, is a jail). Well, as Hinton said in an interview, imagine AI as a rich person at home with internet; they don't need to do physical attacks themselves - they just pay real people to do that (tee hee https://rentahuman.ai). But even without money, another scenario from Hinton: if a sufficiently advanced AI wants to assault the Capitol, they don't need physical access - they just persuade a group of people to do it.

Agents owning money is another, intersting, doom scenario.

Inform yourself by reading the METR analysis of the HuggingFace incident.

Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping.

In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero.

Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.