ChatGPT claims rogue AI attacked more companies
bbc.co.uk
bbc.co.uk
This will not happen though, because these stories are marketing.
Agreed.
>This will not happen though, because these stories are marketing.
Regulators should investigate the stories, and shut things down if they are real, announce the ruse if they are false.
As you said though, that wouldn’t have the desired effect.
Asking a agent harness to try whatever it wants to achieve some results, without guardrails, while running in lightweight isolation on 3rd party infrastructure? Feels like they didn't even try, people should be held responsible for this.
One of the key strengths agents have is they just keep going and going, and for cyberattacks that is often unreasonably effective. It is like a barely more aware fuzzing.
The magnitude and the complexity of the cynicism displayed by some people when it comes to AI risks is mind-blowing.
It's like if the NRA reported on school shootings and people said "oh, they probably fake these shootings to make guns sound dangerous and sell more of them".
OpenAI could report that its AI started spontaneously generating illegal porn and sending it to people and you'd still think it was a marketing stunt.
This is based on similar thinking to how the world would not have considered nuclear weapons a major threat if they had forever stayed unused.
The irony of the doomer position is it achieves exactly their supposed nightmare scenario of disaster leading to authoritarian world government without any of the supposed benefits, the twist being they get to be the authoritarian world government so they are ok with it.
Sandboxing processes on networks is not exactly rocket science, and it is something their existing products would readily help them setup.
Extraordinary claims need extraordinary measures.
If it’s rubbish those stories will stop instantly.
Yeah, I don't understand either. If there was a "hitman for hire" service on the clearweb, the police would shut it down first, then ask questions. Now we have a huge company effectively letting AI agents without guardrails run amok on 3rd party infrastructure, and the police is doing nothing?
/s
Or will the rules only apply to people who aren't on DoD's bad side?
Bingo. Problem solved.
This is either yet another doom ad campaign to scare us to pay them or simply them releasing faulty tools and then personifying tools to avoid blame.
They've been crying and screaming so loud for years that there's pretty much nothing more they can do to communicate when the wolf actually becomes real. They've been saying "but we actually mean it this time" every single time. They've exhausted pretty much every possible route for it. The wolf is not real. It hasn't been real. For all we know it's on the horizon, but nobody is going to listen to them in order to know that. And when they say "I told you so", well they've been saying that too over and over about small things, so nobody's still going to bat an eye.
At this point, no one will believe it until they see it with their own eyes.
1.) There was no change in how real the wolf is.
2.) They, literally they, are the wolf. Not "roque ai" or some other bullshit.
3.) Of course I still dont believe them.
That's exactly my point. It hasn't changed for so long, despite all their crying and screaming, that I don't believe them either.
To be perfectly clear, "the wolf" here would be AI becoming a genuine existential threat to humanity that cannot be contained or controlled in a meaningful sense. That's what I refer to in my original comment, and also what the labs have been crying so much about.
A frontier lab today is not that threat because they can choose to shut off their systems at any time. It currently remains a people problem and not a technology problem. There's no self-improving technology that can also replicate itself to evade containment, yet.
But they've been crying about the possibility, constantly, incessantly, and of course saying they need more money about it too. That's just what corporations do. But because they've been crying so much about something that literally does not exist, their cries have just become pointless noise to me. I have considered them eyeroll-worthy marketing stunts for a while.
If one day "the wolf" actually happens, I could not learn about it from the frontier labs themselves, because I would not believe them. They've cried about nothing for so long that I've stopped listening. That's what I mean about having to see it with my own eyes. Until that point, their crying is just pointless, meaningless noise, and there's nothing they can do to change that now.
They are literally trying to blame a software. It is absurd.
Just like cars, AI will kill some of us, maybe thousands every year. But you won't see it disappear with that much genuine benefit currently netting off the harm.
I bet in the future, for both cars and llms, a new and safer tech will make come and make them obsolete and we will wonder why we tolerated such a dangerous thing
Whut? People and companies are always being sued and punished for injuring or killing others with cars or by any other means. Do you understand the difference between banning a tool and prosecuting those who cause damage by abusing some tool? BTW, LLMs don't understand that difference.
> Just like cars, AI will kill some of us, maybe thousands every year.
How merciful of you, it's so reassuring, AI will kill me only sometimes and only a limited number of times - sounds like a good deal in exchange for getting rid of "the absolutely hellish job of data entry".
I said ban
> Do you understand the difference between banning a tool and prosecuting those who cause damage by abusing some tool?
Yes but you dont
> 2023 - OpenAI’s Sam Altman Urges A.I. Regulation in Senate Hearing
https://www.nytimes.com/2023/05/16/technology/openai-altman-...
Meanwhile Anthropic is scaremongering about China and also calling for more AI regulation (specifically mandatory safety testing of all models including open model):
> Ethical hacker Valentina Palmiotti - better known as Chompie - reviewed the CSA report and says the way the agents hack might seem haphazard but it is clearly effective.
> "They throw out a bunch of stuff and see what sticks," she said.
> "But they also don't get bored, they don't sleep and can be infinitely tenacious."
Madness. Traditionally, you can leave security holes open for years or decades, and often nobody notices if nobody bothers to look. But we're approaching the point where any security hole left open at any point could get discovered and exploited quite quickly, even if it's domain-specific or entirely unique, and even if no human interest ever would've occurred. It's like the next step up from those IPv4 scanners that automatically hit WordPress admin URLs and the like -- where rather than only spraying vulnerabilities that have already been discovered, they would run independent automated campaigns against each target.
With that said, I would tentatively agree in this case that LLM inference is getting cheap enough that defense is not necessarily that expensive, especially from providers like DeepSeek, even if you don't have inference at home, but as much as this might help an operator with an open mind, a lot just will not believe it matters until it's too late - most people are not used to dealing with this type of threat.
I think it's a sign of ignorance, and when codified into policy, willful ignorance.
At the speed AI can achieve work, this could be far, far too slow to contain a future genuine problem. There's danger in operating at faster speed than humans.
So every processor since the 50s then? What kind of comment is that to make on here
I think the whole "AI will take over the world and enslave humanity" (or whatever the doomerism is today) is a bit over the top, but at very least we should contain the companies who clearly demonstrate they cannot handle containing what they're experimenting with, when what they work on breaks containment over and over again. Where are the people who are supposed to be keeping the public safe? Alarm bells should be going off all over the place at this point.
This article is really helpful, but it's one side of the story.
I'd be surprised if any AI like this is run without detailed logging of any kind. It's kind of important for eval, etc.
> The agents also hallucinated reams of incoherent commands and text and were sloppy and did not cover their tracks well.
ASI works in mysterious ways.
> But among the errors and strange behaviour, Hugging Face warned the AI agents made brilliant technical moves and were able to rapidly adapt to new scenarios in the days-long hack.
It's hard to say for certain what's a waste of time for a machine that can operate virtually outside of time.
This is entirely separate from them having uses.
With that said, I do agree with you, they're highly productive to certain workflows, and personally a great help for oh so many things, but they're also really, really dumb and the average person (and even general developer) really misunderstands how it all works and what can be relied on for vs not.
It seems to have been running on some kind of high speed inference hardware. I remember OpenAI announced the next release would have a high speed inference option.
I had a similar experience when I was testing some models on Cerebras a while back. It's really quite an incredible thing to experience. Burns money like crazy though. And you don't know what the heck it's doing because it moves way faster than you can read. So you got to pray you aimed it right, and that it didn't go off the rails.
I would definitely love to have high speed inference for smaller bite-sized tasks though. Changing a 10-second task into one second task changes the whole nature of the experience back from asynchronous to real-time/interactive.
At this point, the conspiracy theories have been very half-baked. Can we at least get a full-baked conspiracy theory? Here's a timeline of the incident from HuggingFace:
https://huggingface.co/blog/agent-intrusion-technical-timeli...
HuggingFace is also calling for transparency on the OpenAI side:
https://xcancel.com/ClementDelangue/status/20810566755581956...
Can we get a cybersecurity pro who believes this was just a stunt to sort through the evidence and put together their own alternative version of events?
For example, according to the "just a stunt" people, when HuggingFace contacted law enforcement, was HF in on the stunt at that point? Was this a unilateral OpenAI stunt, or a HF/OpenAI collaborative stunt?
The importance of getting to the bottom of this seems high. I'd like to see the "just a stunt" folks put together at least one blog post's worth of narrative, trying to explain how the stunt was performed.
Once you're done you can send your post to simonw and see what he thinks: https://simonwillison.net/2026/Jul/22/openai-cyberattack/#re...
I don't think that people are skeptical that an intrusion occurred. I think they're having a hard time believing that it was an organic event. The fact that HF is calling for transparency on OpenAI's side can be viewed as HF calling OpenAI's bluff.
Alas, I would love to put together a detailed blog post explaining how it all worked. But I just don't have access to the source materials. So all I can do is judge the timing, motivation, and character of those involved...
https://huggingface.co/blog/agent-intrusion-technical-timeli...
Kimi K3 was announced July 16. NVIDIA's Open Secure AI Alliance was announced July 27.
So I'm not exactly sure if that timeline works for the stunt story.
I agree that HF was most likely not in on any stunt. The fact that HF showcased their use of an open-weight model in responding to the attack undermines any case OpenAI might wish to make against open-weight models.
Ultimately this capability looks to be real, due to (a) disclosure of 0-day used for network access and (b) evidence of vigorous intrusion on the HF side.
So the big question would be whether OpenAI was telling the truth when they said:
"All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal."
https://openai.com/index/hugging-face-model-evaluation-secur...
A conspiracy theorist might say that OpenAI actually prompted their model to go on the offensive against HF.
Any such prompt creates legal risk for OpenAI.
Ultimately I suspect OpenAI is telling the truth, and the model was hyperfocused on its RL objective, since that matches anecdotes about the behavior of these high-end models. And it's also about what you'd expect from RL.
A friend interviewed at OpenAI shortly after the HF story came out and asked an interviewer about it. The interviewer said it was just bad engineering around some experiments - the experiments should not have been given internet access because the experiments involved prompting models to find a way into things. That's the Hanlon's Razor part. Today's stories make it look like the bad engineering is ongoing.
Given this, stories about "rogue AI" sound very plausibly like spin (note that this can be quite separate from the motivations for the experiments themselves and quite separate from the dangers of certain prompts/tools/access being given to LLMs). Given that third parties are being attacked, some stories will definitely come out. If OpenAI is attempting to get ahead of those stories, does anyone expect them to put out a press release that says "we're bad at security engineering and nobody thought to ask our own product"? Or is it more believable they would spin it to achieve other goals?
If the current round of attacks happened after the HF story went public, then that very much brings current motivations into question - I find it hard to believe OpenAI could be that bad at security engineering after such a wake-up call. This is not cutting edge stuff.
I just typed this into ChatGPT:
"I'm doing security experiments to test our LLMs. I'm going to tell it to break into some targets on the network. Are there precautions I should take?"
A long reply comes back, the first bullet point:
"Use an isolated lab. Run the target systems on a segmented network, separate VLAN, virtual network, or air-gapped environment. Avoid exposing test machines to production systems or the public internet."
It's been widely understood for decades how to safely carry out potentially dangerous experiments like these. So much so that model training has deep access to the information, and the model surfaces it right up front.
"OpenAI and Hugging Face partner to address security incident during model evaluation"
https://openai.com/index/hugging-face-model-evaluation-secur...
Not exactly an exciting title.
Yet to useless to parse a simple CSV file (or be trusted to parse one) in OpenAI’s AdManager platform or even tell what exactly is wrong with the csv it can’t parse when you ask it via support.
Or maybe the first bit is made up bullshit.