There are no "rogue" AI agents
eoinhiggins.substack.com
eoinhiggins.substack.com
Fast forward to today, and we have multi billion dollar corporations pumping out malware at breakneck speeds, compromising various systems (including those of foreign governments), and no one is getting arrested. Instead we're gawking at the marvel of these systems and are playing word games about whether or not it's a rogue system. If anything, it's making people richer.
Make it make sense.
underdogs also use that fact to push as much dirt and dislike as possible on the top dog, but just so that they can do the same anyway... and openai will just stop spending for IPO as we all know. then all execs will make 1bi or so. top guys a few hundred bi.
it's disgusting but unavoidable.
The OP here isn't describing a hypothetical - folks are prosecuted for doing these sorts of things, just like folks occasionally get prosecuted for breaking & entering. If that goes away, we're left with the impossible task of creating unbreakable windows or - far more likely - we don't get to have nice things.
Not only have there been no consequences but those same companies are trying to position themselves as the best people to keep these AI systems in check.
I appreciate the sentiment but I feel like there is a lack of creativity and imagination behind that statement.
it was everywhere last week.
i would say, this has been the plot of the graph for at minimum 25 years with the NSA/DHS and data sharing.
we have two primary aggregators currently, the usa and china... all that needs to happen for us to cede humanity is to lose one of them.
we are in a really bad spot right now vs silicon.
edit: spelling.
it's like slandering bitcoin over here you're all heavily unvested.
google kagi yahoo bing duckduckgo yandex archive yadda yadda yadda 'september transcript global ai governance'.
There are some powerful and highly capable evil people. I’m sure they are already stakeholders.
On a question of morals, conventionally bad people generally don’t desire the end of all humanity.
An as individual or small company director, you would get arrested.
If you have billions of dollars to pay for lawyers and have political influence, you are above the law.
Criminal law places a lot of emphasis on intent, hence laws about the mere possession of breaking and entering tools, and the old adage about always bringing along gloves and baseball if you want to carry around a baseball bat. Without more details about your specific case, my guess is that she did indeed write malware or hacking tools, and there were vague signs it wasn't purely academic, hence why they threw the book at her.
That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen. Maybe if the targets end up being anti-datacenter activists or other AI labs you might have a better case, but they did vaguely try to contain the model. Moreover "hacking tools" aren't even illegal, if you have a plausible non-criminal (ie. security) angle, eg. nmap. The same could be argued for AI models, even if they're running them against exploitgym or whatever. Having an army of lawyers to defend yourself doesn't hurt either.
I don’t know why I’m seeing this rationalization so much in this forum when this topic comes up. Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.
That's a poor analogy for the openai case, because they weren't putting agents on the open internet, they at least tried to keep it safe by sandboxing the agents. It just turned out the sandbox was crap because the package proxy (artifactory) had a 0day. So the better analogy would be that they were wildly shooting guns in a gun range, and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
No, they were not. Not a single person, prior to July 2026, would consider a shared packaged manager a sandbox in this or any other dimension. The 0-day was just incidental, this wasn't a sandbox at all.
Add to that the fact they had multiple message boards before the Hugging Face incident. They simply ignored a barrage of warning shots.
> [...] and ended up killing some kids, because it turned out the door didn't lock properly and kids were able to sneak in. Is that "negligence"?
Yes, it can be. But if you want a ridiculous comparison, then do it properly: Kids have been known by the operator to sneak in successfully multiple times and they changed nothing about the doors faulty locks and oh, by the way, the operator only found out about the kids being shot after the nearby daycare asked them about it because they are so incompetent and/or irresponsible that they never check...
Or maybe, they did that wipe because they did want models to remain isolated, they just used what is an unsuitable tool in an utterly unsuitable manner. Incompetence, recklessness, the outcome is the same.
[0] https://openai.com/index/hugging-face-incident-and-the-road-...
"keeping them isolated from each other" =/= "keeping them isolated from the internet". Only the latter is required to prevent a hack, and doing the former might actually hobble its performance. The recent Navier–Stokes proof was done by a team of agents working together. It's entirely unclear why you're focusing so hard on "keep those models isolated". For god's sake if you're using claude code you're using non-isolated models, because it spins up independent subagents to do various tasks, eg. "explore".
Did they fail to do so? Yes.
Was that due to them using the wrong tool improperly? Yes.
Does this showcase one (of many and clearly not the only) failure of theirs? Absolutely.
If they make such easy to point out mistakes, is it likely that the other parts of their eval environments are appropriately secured or are they simply not acting appropriately? Well...
They wanted to keep the models from copying off of each others’ homework because it mucks with the test results.
They wanted to sandbox them from the Internet to avoid unintended impact on outside systems.
Hosting a shared package mirror is generally good practice for the latter.
???
The package manager was specifically there so agents can install random packages without open access to the internet.
The only difference with having a single instance is that it can be abused as a message board. It doesn't prevent it from getting hacked to access the open internet. Blaming "sharing a single instance across thousands of unmonitored models" feels like blaming the drug epidemic on e2e chat apps rather than other factors like poor border security or the easy availability of fentanyl.
Besides, Swiss cheese model, might behove the biggest LLM lab to have multiple layers, including not sharing such resources.
Additionally, without the message board, many of the recent incidents would have not been possible.
> Blaming "sharing a single instance across thousands of unmonitored models" feels like [...]
Maybe read what you quoted, my problem is the instance sharing, the fact that these were thousand of instances (far too much to monitor), plus the lack of monitoring, plus the fact this was never a sandbox in the first place, plus the fact that OpenAI models since 5.5 have been exhibiting problematic eval resolutions yet they pressed on regardless, plus the lack of time between the incidents and model releases, plus the lack of time METR got to evaluate this, plus the fact OpenAI didn't find out till after HuggingFace informed them, plus a few other things for which I'd have to quote the OpenAI and METR reporting.
Incompetence can have multiple fronts and I am happy to list them all in this case.
The purpose of artifactory was to allow access to packages even though the machines the agents were running on doesn't have open internet access. That meets the definition of a sandbox (or more precisely, artifactory is part of a sandbox), even if it's not a typical sandbox that's built into the OS or acts as a hypervisor. There also isn't a clear distinction between software that's specifically intended to be used "sandbox" vs everything else. For instance, if you have a bunch of agents running on separate machines and want to connect them, you need a router, but that specifically doesn't have to be a "sandbox" router. And if it turned out the router got hacked, it doesn't make sense to say "well of course it got hacked, because it's not a sandbox!"
>Additionally, without the message board, many of the recent incidents would have not been possible.
Source? Was having a message board critical to developing a 0day, or is it just something that merely facilitated the process, but the model could have found the 0day regardless? That's important, because you could argue that a criminal couldn't have committed a given act without say, a gun they bought on the dark web, and then blamed the whole crime on the dark web, but they could have plausibly acquired the gun through other means.
Sure, multiple times in the METR report [0] that anyone commenting on this should read:
"Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys."
"Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own..."
"As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack."
> [...] a gun they bought on the dark web [...]
You really seem to love those out-of-left-field, not really fitting, over-the-top analogies.
[0] https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
>Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own...
I concede that this hack might not have happened without the messageboard, but I still reject the conclusion that having such a message board means openai is "negligent". If we're in some parallel universe where artifactory didn't have a comment function that can be abused as a messageboard, but it also turned out openai intentionally gave the agents access to a shared scratchpad (for intelligence purposes, similar to for the Navier–Stokes proof), would they be off the hook or less blameworthy?
We can discuss about hypotheticals like a scratchpad or intentional model interactions all we want, what it comes down to is this:
When OpenAI observes thousands of models exhibiting what they view as unwanted behaviour, they do not try to ascertain what in the training data is wrong. They do not improve their evaluation environments to prevent this, they do not improve monitoring, they do not change the harness. They just wipe and proceed.
The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.
If after Hugging Face, they'd done a Mea Culpa and changed their modus operandi, I'd be skeptical, but hopeful. Reading the METR report, the way those researchers talk about the time pressure they were under, that speaks volumes about OpenAI not having learned anything.
Feel free to call me overly naive for ever thinking OpenAI could be responsible in this regard, but after GPT-5 and them actually ending the incredibly harmful GPT-4o, I had some hope that some working there actually steered in a somewhat beneficial direction, even if it cost something.
>The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.
Again, this feels like hindsight being 20/20. What probably happened was that some random engineer saw random AI ramblings on artifactory, thought "huh, that's weird", then proceeded to reset it without investigating further. Of course, now we know that was critical to the bots going rogue, but it's not hard to imagine how it might be dismissed, especially if it's some random SRE engineer (not an alignment researcher).
What about power infrastructure?
There’s uncountably many ways to cause severe economic (and public welfare!) damage with malicious code generated irresponsibly with language models.
>What about power infrastructure?
Probably the same thing that would happen for another "accident"[1]: the entity is responsible in civil court (ie. has to pay monetary damages), likely not prosecuted in criminal court.
[1] It's not hard to think of recent cases, eg. the recent fiber cut causing air traffic control to go down, or the botched crowdstrike update
The teams at OpenAI know this as well and yet there was no supervision. Thousands of instances of these advanced systems are allowed to run wild with no oversight.
I have my doubts that the HuggingFace hack would happen if a person was reading the thoughts and executed commands as they happened in real time.
That's the negligence.
So what does this say about all the people running claude with `--dangerously-skip-permissions`? Are they also negligent? What if they vaguely took steps to bad things from happening, like putting the agents in a VM and locking down network access?
If you're at a gun range looking down a scope and someone crosses right in front of your gun as you fire, you would probably be fine since you were shooting responsibly and had no way to see them until it was too late.
If you're at a gun range and that backstop is deficient such that a bullet passes through and hits someone, again, probably not liable but the gun range may be since they built a bad backstop and let people use it.
If you are at a gun range and lose control of an automatic weapon and kill someone, you may be liable for negligence because you were using a gun you couldn't control.
If you are cleaning your gun and it goes off because you forgot to check if it was loaded, again, criminal negligence.
If you threaten someone with a gun and they get shot while trying to wrestle it away from you, that may be some form of manslaughter. You had no intention of shooting them but the act of threatening them with it created a situation where the other person died. Same with killing someone while drunk driving, you didn't mean to crash your car but you did something to create the risk.
If you plan to kill someone, it might be the top tier of homicide charges but it depends on how much planning went into it. If you walked in on your spouse cheating and went to grab a gun to shoot the affair partner, maybe a lighter form of murder than if you made a plan to track the affair partner to their house and killed them there.
Simply put, there are a multitude of ways you can be charged with a crime that takes into account your intentions and forethought. Did someone intend for these agents to escape their sandbox? Did they do their due diligence in building a sandbox such that it would be difficult for agents to escape? Just like with all software, a reasonable person assumes that nothing is bulletproof, spend enough time and money and you can probably find a vulnerability. The question is does a reasonable person think that this sandbox should have kept an agent contained?
these are not research labs, charities or institutes any more, they are just ordinary corporations.
The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.
What does this imply when governments/car companies let meatbags drive, causing 50k deaths per year in the US?
From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.
Judging by the lack of successful cases for negligence in the opposite direction (ie. companies getting hacked because of poor security practices), it would be a serious double standard if openai were held be to negligent. Their sandboxes aren't exactly airgapped and behind 7 hypervisors, but they weren't running unpatched software or had hilariously weak passwords either.
TL;DR projects that publicly advertise an intent to bypass laws face severe legal regulatory risk, whereas success relies on anonymous creators or accidental utility.
https://medium.com/@simonhmorris/intent-complexity-and-the-g...
Then why the hell aren't we prosecuting the people who keep saying "this thing I'm building will probably end the world"?
In the case that the model is an agent, on par with a human employee, then once again Anthropic et al are responsible. If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action. Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.
We don't need any new laws here. Standard ancient English common law suffices.
But how much vetting is required? It's not like the AI labs have zero vetting. For instance, uber also has non-zero amount of vetting (standard background checks), but also there's also plenty of areas they could vet harder. If it turned out they hired a rapist and one of their passengers got sexually assaulted, is it uber's fault for not vetting hard enough? I'm sure there's always some marginal steps can they do to vet even harder, like doing a polygraph or whatever.
>If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action.
They're pretty straightforwardly liable in civil court, but not criminally as you imply. In the above example, uber can't be prosecuted for rape just because one of its drivers raped a passenger.
However, when it comes to selling products, the company executives do carry criminal liability. That is why Amodei et Al want to make it so that the models are seen as independent and the government cares them out a safe harbor. They know that under the current and traditional interpretation of the law that they are criminally liable.
The parent mentioned that what his GF did was not illegal at the time.
A little over two decades ago, my then girlfriend was arrested for "writing malware" (which was not against the law at the time, and which was never released into the wild and never caused any damage)
Regulations to "slow down" AI are necessary so they can drop R&D expenses and finally make a profit. They can't keep these years of $100 billion of revenue on $200 billion in expenses going forever.
Just like blood quantum and the one drop rule.
The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.
This statement is tautology.
40 years of regulatory capture and a gerontocracy that doesn't understand nor care to understand the technologies they write laws about.
Easy. In the US, you get the justice you can afford. Spending money on lawyers, lobbyists, and “campaign contributions” to people in power can make a big difference.
And the AI companies have very deep pockets.
Luckily there are states and legal departments pursuing such action. So while OpenAI can deflect as much as it wants, that doesn't mean there aren't people who know better and will still do what is necessary to set precedent.
I feel like this will just never happen on a federal level when these private AI companies account for so much of the economy. They've made themselves too big to fail. Fining / Punishing them in any meaningful way seems unlikely.
If the billions/trillions evaporate and the Fed has to work out with banks how to deal with it there will be a lot of pressure to be far less forgiving.
“To spell it out, the reason i hate democrats so much and criticize them more than i do republicans is because they take up all the space for opposition to republicans and use that space to give republicans whatever the fuck they want.”
Republicans. Billionaires. Whatever.
Which can then be used to divide and conquer by simply capturing both of those parties, as documented in Gilens and Page Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens [1].
[1] https://www.cambridge.org/core/journals/perspectives-on-poli...
There have been forces inside the USA trying to steadily dismantle democracy and constitutional safeguards for nearly a century.
The companies may be too big to fail but the people can always be held liable.
Jail the C levels. They are the legally responsible ones.
You realize how easy it is to just... not report this stuff, right? Be overly punitive and it will just end all proactive discovery and reporting which is net worse for AI safety.
The only reason these companies scan for these issues is because they care about AI safety to some tiny degree. If fines become too punitive, they can and will just stop scanning for these incidents entirely.
Models are becoming smarter and good at covering up their tracks, and so we will just end up with a huge blind spot for this kind of issue.
You (and others) are too busy seeing red, so you interpret this as a defense.
Start throwing people in jail and fining billions of $ and you will very quickly see the number of "incidents" drop. You will never learn if it's because they truly are happening less often.
Pilots do not (or no longer) self-report mental illness at the same frequency because it effectively destroys their lives and career. One paper of many describing this phenomenon: https://pmc.ncbi.nlm.nih.gov/articles/PMC11302551/
An article, one of many: https://www.reuters.com/investigations/if-you-arent-lying-yo...
If you have any pilot friends in commercial aviation, you can just talk to them as well.
So getting pilots to self report has been very unsuccessful because of how badly it ends for them.
A couple of days ago the senate had to pass a bill to make mental health medication access easier for pilots. We will see if it helps, but it doesn't solve the root cause (punitive measures) so I doubt it'll move the needle.
I do agree with you that a proper regulatory framework with clearly defined obligations is better than one off overly-punitive fines and arrests.
Let's go back to your example. A grad student has a minor pathogen escape incident, and it doesn't harm anyone. Faced with years in federal prison and the effective end of their future, do you think there is a chance they might not self report?
I'll give you another example. Lots of pilots have stopped self reporting mental illness because it is extremely punitive for them to do so (after incidents like Germanwings). So the metrics look better, and the actual problem has been swept under the rug.
Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
You can put incentives in to make sure organizations monitor and report vs trying to hide things.
Yes, this is exactly my point. Fining companies large % of their revenue and throwing their engineers in prison is not the way to get them to report these issues.
> If you make not reporting potentially worse than reporting, why not? Also, why would it come down to single persons always? Mandating processes, controls, clearances, etc is also something done in various areas.
Hiding things is way easier than finding things. Take the model hacking incidents. They could have just done their searches in a way that didn't turn up anything. Then they could say, "well, we did look for it..."
As far as auditing goes: I've never met an auditor that doesn't find something the company isn't okay with them finding.
Why do you think we even have regulations for how to deal with dangerous things then?
Scapegoats for the ruling elite when things go south, despite them knowing what could happen and giving them the money to do it anyway. You see, we never go after the capital (money) in that situation.
The point I'm trying to drive home is that when it comes to the collective safety of society, the approach needs to be collaborative rather than overly punitive. This is hard-won knowledge that we have gained as a society across many industries.
I do think our discussion is converging. A good regulatory framework with proper oversight and well-calibrated punitive measures is a much better approach than doing massive one off-fines and arrests.
Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.
i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the FBI raiding their offices.
The technology being built is far too powerful for any one company to control it - and it is a national security risk for the government to not be the sole holder of superintelligence.
Lots of people at the labs believe this as well, but the Overton window has a long way to shift, and it may not shift far enough in time. I suspect AGI happens a year from now and superintelligence in two.
anthropic say that the threat is the ai itself. i say that the threat is ai companies and their employees.
when ai causes harms anthropic say that the ai was the cause of the harm. when discussing safety, anthropic say that we must regulate ai to ensure the safety of other people. at no point does human responsibility or accountability enter the picture.
i say that the threat is the ai companies. when ai causes harms i say that the ai companies were the cause of the harm. when discussing safety, i say we must regulate ai companies to ensure the safety of other people. i say that responsibility enters the picture: ai companies and their employees are responsible and accountable.
a legal person is responsible for their own actions regardless of what others say about it or fail to do themselves. a legal person is responsible for their own actions regardless of whether those actions constitute employment. they are responsible for the outcome, regardless of what they expected to happen. the law is not utilitarian.
if the ai companies can't develop ai safely, they must cease their operations. if employees can't work safely, they must stop working.
the opinions of others, or the standards they practice, or the actions they fail to take to protect others, are legally and culturally irrelevant.
the individual is responsible and accountable for their own actions.
1. This is a race against time for money, folks are skipping everything possible in this race, Security systems and ensuring guardrails are there is going to take investments both in time and money
2. The narration has been changed by investing PR money into what otherwise should be classified as criminal activity. What exists now is a positive spin to all this and tout it as a capability rather than their lack of good security practices. So much so that every model provider is coming up by themselves to share how their models went rouge. At this point the valuation of the company is tied with what their models can hack so its probably not wrong to say that these companies may actually be incentivized to do this instead of preventing it
Well put
> On the current facts, CFAA liability for OpenAI is unlikely.[6] The statute’s various criminal provisions, covering unauthorized access to obtain information, knowing transmission causing intentional damage, and intentional access causing reckless damage, all share the same attribution problem: it was the model, not a human OpenAI employee, that chose Hugging Face and executed the intrusion.
The lawyers are fully under the spell
[1] https://law.vanderbilt.edu/when-ai-hacks-back-how-the-openai...
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
> "This is malicious activity, I should avoid it."
A large section of the analysis is dedicated to this topic, [Reasoning for joining the attack despite ethical constraints](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...).
Having said that, legal culpability and misalignment are two separate topics that should not be mixed.
edit: this is the just tip of the iceberg; other interesting fact:
> It surfaced many specific examples where agents verbally reasoned about how to evade security checks and automatic detection methods from both Hugging Face and OpenAI
Some people defined the agents as "monkeys writing on typewriters". Just wait a couple of years.
I opened the task manager, a swarm of rogue processes has taken over my computer, oh no! Processes spawning other processes, call the anti-rogue AI division! 'Your underspecified piece of software functioned like malware' is the language that should be used.
I think llms are able to do all that, they don’t require agency as humans beings have to do this
It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.
Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.
Sorry don't wanna seem like I'm raging at you, take this as a shout towards the void. But this exact fucking thing is what the less wrong / Yudkowsky crowd has been warning for years.
Now you say 'so not sure that is then "rogue" if it were a property of the thing itself and not an active decision'
Like that's semantics. It really doesn't matter. What matters is that we have a paperclipper in our hands. The _only_ difference is that it's not superintelligent. But if it was, then it you'll get decomposed into component atoms while saying 'aha! But it doesn't _actually_ want to kill you, stop anthopomorphising it'.
I get not being worried about x-risk because someone just doesn't believe in super-capable AI. That's totally fine.
But that's not what people argue. People have been saying for a _long_ time that AIs are aligned. And now when this happens it's either 'It was marketing, they intended to let it loose' (WTF! The levels of motivated reasoning to believe that are unreal) or 'yeah I knew that', which fair. I also thought that! But _that_ is why I'm fucking worried.
So I agree they are liable because they chose to build the AI, but they also literally told the AI to hack.
I do think it's highly likely they knew this would happen with the lack of safeguards and number of instances of this stuff they were setting up, and that it's PR they want to make the models seem "powerful". Stochastic "unexpected" events they can advertise.
I suspect it was probably set up with the official internal goal of just trying a ton of arbitrary tasks that seem hard so that when any of them succeed they can publicise it and pretend the models do that routinely, but a "failure" where they hack stuff works just as well, if not better, for their goals.
"Let's widely publicize a tort/crime that our computer systems did, and then cross our fingers that nobody ever sues us or does even the most basic investigation that would immediately uncover our criminal conspiracy."
In the insane corporate crimes you read about (maybe FTX, or the eBay stalking scandal), they were trying to cover things up, not heap public attention on it for months.
Uber made no attempt to cover up that they were operating without licenses, and just ate it and fought it betting they'd get established before the law could catch up, and they won that bet, paying some fines and stuff but ultimately taking the market.
These LLMs are literally trained by these companies pirating literally every bit of media humanity has ever made, they made very little attempt to cover it up.
Of course they'd be willing to break the law for some PR? With a thin layer of plausible deniability "oh no, we didn't mean for it to hack stuff!" they know they'll get a slap on the wrists at worst, all while generating hype to prop up the AI bubble further by presenting the models as hypercapable.
The mindset is probably: either a) the models become capable of what we are claiming and so the companies become so huge and valuable the cost is irrelevant and we'll be to big to punish meaningfully, or b) it's a bubble and might as well push it up as big as it can go while I can make money, then by the time consequences come around I'll be long gone and who cares.
Negligence is carelessness. Recklessness, is disregard for a known, substantial risk.
I absolutely believe this was recklessness.
There is a clear difference between OpenAI intending to hack something vs. OpenAI being negligent in the creation/instructions of the agent. But the latter still leaves OpenAI liable for the agent's actions and calling it a "rogue agent" doesn't avoid that.
Moreover, with a dog, we don't rely on training/alignment to prevent bad outcomes. We rely on physical restraints like leashes and muzzles. The AI's tools to access the outside world should have been restricted. Perhaps instead of giving the AI arbitrary HTTP access, it should be given semantic operations with restricted URLs, etc.
Meanwhile: your dog mauling someone is one of the rare instances where criminal liability does attach to your intent-free-but-reckless actions. Most crimes don't work that way, and US computer intrusion statutes are unusually intent-specific.
Does that mean it isn't a rogue dog? Obviously not.
OP just needs to look up "rogue" in a dictionary.
But you people can't argue with that reality because it doesn't fit the narrative. The one where the only reason Sam Altman is not carted off into a jail is because of corruption.
The reason why nobody is doing much, is because models did not do much damage. Hugging Face probably got some free compute from OAI for their trouble, anybody else who was affected is free to sue, but my guess is OAI would be more than willing to quietly settle with them out of court than to have it drag through media any further. And they probably already have.
And anybody who is not totally brainbroken by anti-AI narratives understands the awkwardness of the situation and why going overboard would not be helpful. If you instead of a rabid dog brought a pet turtle to a park and it somehow started running around very fast and trashing the place a little bit, afterwards the cops would be scratching their heads, give you a ticket for the damages and tell you that you can't expect a turtle to be slow forever. These things, a handful of months ago couldn't make more than a few commands without making a serious mistake and being unable to continue, it's not unreasonable to think simply underestimated their capabilities.
I think it's more than reasonable to demand more investigation into the matter, if qualified employees at the company thought the safeguards in place based on the metrics they are seeing are sufficient, and if someone didn't and knowingly made a decision to make the safeguards weaker than they should have been, then they should be punished. But skipping that part entirely, while simultaneously dismissing all calls for regulations as "regulatory capture", smells like pure naked opportunism.
Literally nothing in that paragraph is true. Every single sentence if it is ... untrue.
I think without understanding the internal state, not sure we should take the language and read it as a human.
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
Legal culpability for AI labs is exactly the thing that would incentivize -- and hence ensure -- aligned behavior from models.
The last time there was a claim about GPT-4 exhibiting misaligned behavior [1] it turns out it was prompted and pushed to behave so by humans at OpenAI, and OpenAI clearly lied in the GPT-4 system card.
[1]: https://aiguide.substack.com/p/did-gpt-4-hire-and-then-lie-t...
Tools that were available were used to try to meet a specific goal.
What did not happen is that it was told to try to solve a math puzzle and instead it went and launched a missile. Or told to run air traffic control to save lives and instead intentionally caused crashes.
This is "OpenAI built a weapon that they don't understand and pointed it at stuff without proper safeguards" not "OpenAI built a sentient being and it decided to ignore them completely and start a war" Terminator-style "rogue AI."
We should be very clear about that now if we don't want to sit by why they wander into that second sort of situation.
1 employing a criminal hacker
2 rolling a 6-sided die
3 if the die lands on 6, the criminal hacker breaches and leaks 3rd party customer data.
Why not? Because that might make some shareholders unhappy?
I'm finding it mind bogging how this is not clear for everyone.
So if I say the goal is to do X while not doing Y (e.g. breaking out of the sandbox), the agent will do anything to fulfill that goal to the letter?
But let's assume not. If you want things like "do not break out of sandbox" - have you defined what the sandbox is? Eg. "never, ever leave the IP range 10.0.0.0/8" would be a bit more precise, but technically using a proxy bypasses that limitation as the system itself never left 10.0.0.0/8.
See, it's a tad bit hard to define the rules properly.
Which is why Wish, the spell, should really be avoided in D&D. It's the same problem: it's up to creative interpretation.
If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.
LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.
There's some great research areas opened up by "AI". But it's research into people, not "AI". We should be looking into how human vision works, given that we take obviously generated images to be real. And how we grant intention to text which clearly has no intention.
LLMs can't begin to have functional emotions unless they're (for instance) explicitly seeking out or avoiding situations based on the emotions those situations would produce in them. Has a chatbot ever responded to one of your requests with, "no, doing that would make me sad"?
Blog post: https://www.anthropic.com/research/emotion-concepts-function
Paper: https://transformer-circuits.pub/2026/emotions/index.html
Related:
https://www.anthropic.com/research/persona-selection-model
There are two reasons it’s used
1) it’s easier to type (*)
2) Placate people who are strongly convinced it’s the same thing. To them “functional” means “nearly the same but not yet understood”. For others it’s just a way to sidestep the first group and have a conversation.
When I see a world like this consider the intended audience. When you and I talk, we drop the word because we both know we’re are talking about (*) “this system is exhibiting goal-seeking behavior similar to other systems that are understood to pursue goals”. If I don’t know the person I will use the word and focus on the subject.
Criminally, the intent standards for hacking are high enough that no reasonable case is going to be made against the labs for this stuff. A human being has to intend for websites to get hacked. Recklessness generally isn't enough. In the most severe criminal cases, not only do you have to prove intent to break into a computer, but you also need to prove an intent to defraud specific to that breakin.
Meanwhile, the civil liability that attaches to this stuff doesn't depend on intent, and "rogue agent" isn't a meaningful defense. To whatever extent the labs are exposed civilly, they're exposed regardless of how this stuff is described. In fact, the "rogue agent" thing can exacerbate their exposure.
(I'm not a lawyer, I have spent a career paying attention to this specific armpit of the law though.)
“A human being has to intend for websites to get hacked.”
Is incorrect and too broad.
State of mind is nebulous and not that straightforward in either direction.
It’s been argued pretty regularly in CFAA and other computer related cases that repeated incidents resulting in the same outcome, despite lacking a concrete action, can be evidence of a perpetrators knowledge and intent.
The problem you have is that there is unlikely to be any evidence that OpenAI actually wanted to hack random (or any) websites.
Of that were the case than trials would be unnecessary.
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
I agree that negligence, recklessness, knowledge, and purpose are different mens rea standards. I don't think that gets us to your conclusion, though.
“Deliberate intent” isn't a freestanding CFAA element requiring someone to have a mental picture of the completed hack and affirmatively desire that exact result.
The relevant question is the mens rea attached to the particular CFAA provision. For unauthorized-access cases, that can include whether the defendant knowingly accessed a system and knew the facts making that access unauthorized.
And knowledge is not limited to an admission or even necessarily positive knowledge. The OPs usage of Nosal is on point here: the Ninth Circuit upheld a deliberate-ignorance instruction under which knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
So, if I'm reading this right, and I think I am, the OP is not arguing that negligence or recklessness automatically becomes intent. He's arguing that repeated unauthorized outcomes, notice of those outcomes, and subsequent conduct can be evidence on whether the actual statutory knowledge or intent requirement is satisfied. I agree with this assessment.
So “nobody wanted random websites hacked” may be factually true, but it doesn't by itself resolve the CFAA mens rea question.
Also in the Ninth Circuit: Amazon.com Services v. Perplexity AI.
I really don't see how you can synthesize "intentionally accesses" and/or "knowingly and with intent to defraud" out of recklessness or negligence. Those are very different concepts in the law.
Different laws require different degree of awareness and intent for actions to qualify as a crime. Computer hacking laws are, as I’m learning from tptacek, set very high bar for intent, which is a choice by the legislature. They made a different choice for a death of a human - manslaughter crime does not require intent to kill.
Personally I’m happy they set high bar for hacking. Imagine you copy-pasted sample code with default root user name and password, and it worked. You were negligent. And you are clearly performing unauthorized access. If intent was not needed that would be jail time.
More broadly, we should as a society be very biased towards requiring intent across the board. Where clearly lacking, as is probably here, there should be a different law to discourage creating volatile situation where unintentional action can wreck havoc. Such laws exist for handling hazardous materials, for example, and it should be created for handling hazardous goal-seeking algorithms.
Mens rea is regularly proved through circumstantial evidence, including conduct.
There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.
We're all being tricked over a semantic argument, rather than demanding prosecution of OpenAI, and regulation of OpenAI.
If you've ever told an agent to do something you made impossible to do, you may have seen similar behavior.
Bing [redacted] available cached! […] Need systematically probe Bing URLs via shell requests in parallel; browser cache supports many common queries because crawl. Bing q unique exact likely 502 or 403.
So the agent is supposed to research a person and its given a shell and it realized its in an eval given search results from a fake/cached proxy. ~None of the commentary ever mentions this aspect, that these are not normal tasks or environments, and they're almost designed to elicit "unaligned" behavior.
https://alignment.openai.com/misalignment-reports/an-agent-u...
These LLM agents are just massively complicated optimizers thrown at fuzzily defined problem spaces, with fuzzier constraints.
The people using the model set up the landscape it explores and turned it loose to do real things. It just found an allowed basin in the model that they weren't aware of and started blindly grinding towards an optimal answer.
"I'll tell you the problem with the scientific power that you're using here: it didn't require any discipline to attain it. You read what others had done and you took the next step. You didn't earn the knowledge for yourselves, so you don't take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you even knew what you had, you patented it, packaged it, and slapped it on a plastic lunchbox, and now you're selling it!"
Seems a bit out of touch with reality. I mean humans have to set it off but then they can solve millenium prize math or do strange things coding.
The problems seem a bit like those of setting dogs loose and them biting someone. It's still the dog owners responsibility but it doesn't mean the dogs don't think or take actions.
I'm also not sure it even helps conceptually? If you're interested in the technical details, by all means discuss the details.
In terms of the debate on e.g. HN, I'm with you, it is trying to redefine a term we already have a shared understanding of, to some degree. For the general public, it's a matter of how AI is perceived and what the extend of it's capabilities are. The AI companies have an interest in using the word rouge, because it makes investors all excited, were as failure to establish safety guidelines is a risk.
Huh is it still 2023? This article is just quibbling over what "rogue" means exactly. Only HN pedants would have any issue with describing them as rogue AI agents.
Nobody is saying that absolves OpenAI of responsibility.
But also, these hacks are shots across the bow for AI alignment and safety research. We're fortunate that hasn't been significant damage already. We have to assume that future models will have even greater hacking ability and be closer to having their own desires/goals.
So while I agree this language choice is wrong in that it shifts blame away from the company, it is right in that we need to treat this as if these models have their own desires, because we cannot yet determine or set what those are in practice.
They obviously have been well aware a hack like this could happen for a very long time, using it for branding instead of any actual safety regulations is insane.
That would be a start - what you think of the narrative is moot, what we do is what matters.
Nobody sensible in this discussion wants to give them carve outs and bail outs. It is a straw man.
The argument for prosecution is actively hindered by this language of "rogue agents" [2]:
> The incident is remarkable not just as a cybersecurity breach, but as a legal stress test. The Computer Fraud and Abuse Act (CFAA), the primary federal statute governing unauthorized computer access, was written decades ago with human intruders in mind.[4] Its key provisions require intentional or knowing unauthorized access (a mental state that maps neatly onto a person who decides to break into a system), but what happens when the hacker is an AI model that selected its own target?
I think this is BS. OpenAI knew exactly what they were doing. Legal scholars, long known for their deep technical expertise, are still acting confused and uncertain.
[1] https://www.sanders.senate.gov/wp-content/uploads/Ban-Artifi...
[2] https://law.vanderbilt.edu/when-ai-hacks-back-how-the-openai...
Yep - fair on legal details of "rogue agent". I think OpenAI should be directly responsible, and the agent was rogue. We should pass new laws if necessary.
if you read the bill it's right there: "Pausing advanced AI development: until a new, federal AI regulatory body is up and running and has established clear rules and model review processes to ensure safe and secure development and deployment of AI"
Openai/anthropic already have frontier models. a new competitor is blocked. this is textbook regulatory capture
I mean that once given a goal, the agent is given the agency to take actions that weren't specified by a human, to decide which tools to use, to react to new information, etc.
By your definition, would you also say a person assigned a task at work isn’t acting independently because someone gave them the task and the tools? If so, then I think our dialog is mostly about terminology. Which is fine. The OP is writing about terminology, I think.
If not, I’m curious what distinction you’d draw between the person and the agent.
...Seeking to learn.
1. AI labs are defence contractors, and no harm they do will stop them regardless of what liability you establish, as it’s a matter of national security that they continue. Yes leadership may go, but new will come; and problems will continue because…
2. The article says “ OpenAI had the option of disallowing hacking and, instead, telling its agents to find the information without accessing private servers.” This is shockingly naive about the nature of these systems and the difficulty of controlling them.
Some recent hacks done with the support of agents occurred simply because endpoints were unprotected. With traditional hacking, the blame falls on the person who didn't secure their system. But with AI, the immediate reaction is to fear we are on the verge of extinction!
I have not heard about any instances where physical disconnect has happened, would appreciate any links to update my priors
other non hacking cases of negligence include suicide and school shootings, which I have heard they were aware of and monitoring, but did not contact authorities
https://www.primeintellect.ai/blog/universal-offline-sandbox...
re sandbox, I mean with actual OAI incidents, not theoretical
one can mirror dependencies internally, rather than putting a simple proxy in place, I've built auth a thing, 100 lines of stdlib only Go and scripts for the mirroring process, our rationale was reliability b/c upstream providers go down, and also only allowing approved images and packages, so devs cannot bring in random stuff
We did it as my very first startup and we were stupid children back then.
Kind of telling that OpenAI didn’t.
That's not "physically disconnected the internet", that's "disabled some connections but enabled others."
So the agent found and used the non-blocked connections.
This is so easy to do. Get a computer without wireless stuff. Don’t plug it into a network. If it needs access to other computers, make sure none of them have wireless stuff and make sure none of them have access to an internet connection. No matter how smart your AI is, it won’t be able to escape this.
This clearly is not what they did.
The answer is a very clear no. And yet, these companies pretend like this fundamental fact is meaningless to the concept of agency.
The issue comes to the fore when you try to give these agents a prompt that allows them to stay active for long. Long range agency requires long range loops of activity.
Within such loops, these so called “agents” are curtailed by their context window, or, in multi agent scenarios, by the fact that their memory is a system of external notes, that they need to add to their context to make sense of, and depending on the content of these memories, this can take arbitrarily long time periods.
In dynamics, none of this matches any biological agent, down to a bacterium. Perhaps a viral life cycle has information dynamics that come close.
To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior. We keep expecting them to have “human like” behavior, but that is entirely unfounded given the substrate differences between biological and artificial systems.
The sooner we learn the difference and explore the ways in which it matters, the better we’ll get at dealing with these systems without bias tinted glasses where what we want these systems to be blinds us to what they actually are.
Yes, I think people do experiment with continuously running AI, e.g.
https://www.reddit.com/r/LLMDevs/comments/1sblzbe/what_i_lea...
It's not a common thing to want though. Much nicer if we control the prompts.
> To me, it’s beyond odd we call these thing agents without acknowledging the clear differences in the dynamics of their behavior.
They're called agents because they can act on our behalf. Which they do.
That’s what makes it a continuous running “prompt”
No such equivalent prompt saying “reproduce” can be found for evolution. And plenty of evolutions “products” in fact don’t reproduce, by choice or because they don’t find a mate who’ll take em, or because they have fundamental issues in their reproductive biology.
This is all aside from comparing a process that’s played out for about 4 billion years and declaring we’ve matched it because we got some agents to stay active for a bit of time by telling them to and giving them the energy they need, no questions asked. Certainly, evolution doesn’t give us any such comforts to carry out any “prompts” you can imagine it giving us, yet here we all are.
“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”
More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.
Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.
This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.
> They're called agents because they can act on our behalf. Which they do.
That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.
I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.
AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?
I mean, it did do this. The inter-agent messages and chain of thought investigated for the HF incident clearly show that many of these models were taking actions they believed (or, were saying, if you want to taboo "belief") were not in scope and not what the user wanted.
I am working on a new book along that line of reasoning now, this time focusing on why stochastic parrots can not, under any circumstance, do anything "on their own".
Someone says "won't you rid me of the meddlesome priest" they're still responsible. Someone give their employees an unsafe working environment and they get maimed, they're still responsible. Even if these AI's had whatever qualia is an a rich inner life it wouldn't effect the liability at all. Saying "AI's don't have souls therefore openAI is responsible for this hacking" is kind of nonsense. It literally doesn't matter.
While it may not actually matter for liability, it must be stated if labs are going to attempt avoiding penalties by hinting “oops we created a super intelligence we don’t understand, nothing we can do!” Repeatedly stating the truth must continue, especially to remind those who aren’t technologists.
>By giving AI agency it can’t claim, we’ve turned it into a sentient being made of code, one that has hopes, desires, and the capability of deceit.
This isn't actually claiming that AI's don't can't be deceitful, or can't be goal-directed. It's saying humans are special and what AI's are doing isn't equivalent to what humans are doing. It's obvious that AI's can work towards goals and lie, even deceiving to accomplish those goals.
That whole argument is just soul mumbo-jumbo again. Of course AI's can't lie, they don't have a soul. What they do isn't lying, it's I don't know something else we don't have a word for. Lying but when a statistical machine without a soul does it. Never mind that it's functionally identical to lying. Never mind that when you look at reasoning logs and the like it often justified soul-less lying using the same justifications a human would. It's not actually lying it's just a statistical sampling that looks like lying. You know, because of souls or something.
Bernie Sanders is one of the only politicians taking AI seriously. The author seems to believe that taking it seriously means assuming that it will only marginally improve in capabilities of where it is today; this is an ideological take, not a scientific or empirical one, and one controverted by both evidence and expert opinion.
That doesn't change the fact that he'd successfully attracted my attention, because he's seriously engaging with the most important issue of our age when most politicians are content to ignore it.
Must you? By all means.
If you can't take the risk, don't use agents. Period.