OpenAI agent hacked Australian government website, PM says
bbc.com
bbc.com
So we have a company hacking a foreign government's websites and data. And, in terms of ethics, they take almost three months to notify; and in terms of competence, appear to have no formal contacts nor to have found one in that time.
Once an American business starts hacking allied governments, it's time for strict responses, yes? Replace the governance (board and C-level)? Remove financial incentives and open the company - open weights, open training, per its original 'open' ethos?
Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
Kinda worse than that. It took between 10 and 40 days, not 3 months, between the organisation knowing and the reporting.
August (precise date unknown) – OpenAI said it became aware of a potential breach during a broader review of "misaligned model activity"
10 September – An email from OpenAI lands in the public inbox of Services Australia, the general services hub of the federal government, informing of the incident
> open weights, open training
Given it was the AI agents which did the hacking, doing this will result in basically every organisation at least as rich as the government of Tuvalu being able to hack anyone at any time.
> Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
Him having control would be an improvement on the reality.
One can find many other logical combinations that we can’t possibly know about such incidents.
Any automated alarms for detecting things in real-time were not sufficient.
Given a previous generation of agents discovered a zero-day and used it to get around attempts to sandbox them into one specific test, this is not hugely surprising, but it is a reason to force them (and everyone else) to stop until security catches up with capabilities.
I'm thinking of the Jurassic Park novel: they had sensors to count the dinosaurs, but the test was made under the assumption escapes were possible and breeding was not, i.e. something like "if (dinosaurs_found < n) then escape_alert();". They didn't know dinosaurs_found >> n until everything was already going wrong.
Compare with the Bluetouff affair (2014) :
https://arstechnica.com/tech-policy/2014/02/french-journalis...
NotE how he was found guilty by the 2nd court for something more 'subjective' than 'objective' : for having confessed that he later found an authentication page that had failed to protect the documents.
How can you make a swarm of agents "feel guilty" ?
"Feel" is a whole philosophical can of worms. Nobody knows what it means mechanistically for an arbitrary system (including other biological systems) to "feel" anything, let alone abstract concepts like guilt, all we can do is observe behaviours. If current systems can feel anything at all, it's by accident, but we have no test for it so we don't know if that accident has even happened or not.
Weirdly, for the Hugging Face incident, we do know they wrote down that it was bad and they shouldn't do it, even though they then continued to do it.
So: they acted like they felt guilty. And yet also acted like were compelled (by previous training?) to weigh "complete instructions" more than "don't do crime". We can adjust that, make "don't do crime" take precedence over "follow instructions"*; it's unfortunate that when we do for any specific model, there's immediately a horde of people complaining the model has been "censored" or "lobotomised".
(Different people, I hope. Goomba fallacy and all that).
* Though this may cause issues when going between jurisdictions. But hey, a discussion about sovereign compute is for another time, after we can agree to make "don't break the law" more important.
Unfortunately, "don't break the law" would also be a very effective way to use AI to construct an AI-enforced dictatorship, so we can't just throw that in blindly.
If you go into someones garden an copy their work, it does make a big difference if you admit to seeing the sign saying "private property, keep out".
Especially after Bluetouff was found guilty.
In fact, I expect this to have happened many times, but "hacker did the right thing" is much less likely to make headlines.
Meanwhile, agent swarms seem to be (mostly ?) incapable of having this kind of moral compass, at least for now. (And OpenAI isn't doing much better, cough.)
Oh, he does. It's unlikely that this is some AGI that spawned itself out of nothing and started doing this. If he were a decent person, he'd simply find a way to investigate this internally, fire the people responsible, and find a way to set up guardrails around his product.
The problem is, like most people in SV, Altman seems to have a twisted ethical compass. He doesn't see these incidents as an issue, he sees them as an opportunity. He has both the thing a bunch of Western governments want (a superhacker agent that can do dirty work) and a crisis that can be used to craft regulations that favor OpenAI and thus his bank account.
You've not spent much time playing with these models, I see.
Does't matter if you call the current models "AGI" or not, they:
(1) successfully do stuff like this. Someone I know on Telegram found "fifty or so" Linux filesystems kernel bugs a few days ago, of which 26 were the first night while he slept; this was with Kimi which is one of the open models. He stopped it when the backlog of fixes to submit was too big, not because it wasn't finding more.
(2) sometimes misunderstand goals, sometimes wildly so, which happens every so often for the same reason we use programming languages (and indeed mathematical formalisms) at all: natural language is vague and prone to misunderstanding.
and (3) tend towards sycophantically agreeing to implement goals they're given even when the goal is stupid.
This can easily add up to something seemingly innocent like "research if there's any statistical difference in melanoma rates between different Australian states" becoming this headline. (For example. I don't know what the actual request was).
And "spawned out of nothing" is, like, what rhetorical point are you even trying to score here? They're an AI company with (their definition of) AGI as their goal. They upgrade models twice in the time it takes someone to pass a probation period.
> He doesn't see these incidents as an issue, he sees them as an opportunity.
That he may be.
The models are still quite capable of acting this way without him being aware of it at the time, nor deliberately ordering it.
So in this anecdote, a model is presumably working on instructions from a human to complete a task.
That's what I mean by "It's unlikely that this is some AGI that spawned itself out of nothing and started doing this."
As for Altman being aware or not, well, that doesn't matter. His company's product hacked a computer belonging to an allied country's government. That means he doesn't know what he's doing. That doesn't happen if proper safety measures are in place, because if it does happen with safety measures in place, by definition, those measures aren't proper.
In saner times, this would have triggered a few black SUVs from DHS to show up at the company's headquarters to have a talk with him, likely without the public even knowing the hacking had occurred. Maybe a few days later, there'd be a press release about how Sam had decided to step down from his position and take time off. Perhaps the same would be true of other c-suiters at OpenAI. We'd just assume that investors had bought him out and wanted someone who would take a fresh approach, and wouldn't give it too much thought until a few years later when someone wrote a tell-all book about their time at OpenAI.
Sounds like a disagreement on labels then.
I don't think our positions are very far apart, after accounting for that.
> presumably working on instructions from a human to complete a task.
The instructions were, reportedly, a few sentences, no more complex than the simple description you already had. To quote:
No more technical details than the idea: fuzz Linux filesystems with AFL++. So this ain't rocket science to kick off.
Doesn't matter if you think Altman et al are liars and frauds who are overselling everything and had humans doing the hacking on purpose just for the headlines: the capabilities are there even with open weights models.
> That doesn't happen if proper safety measures are in place, because if it does happen with safety measures in place, by definition, those measures aren't proper.
That would be a better world. I'm expecting things to get much, much worse before the risks are taken seriously.
That said, I'm not sure "likely without the public even knowing the hacking had occurred." is good? Surely it's important for everyone to know that this kind of thing is within the capability of AI so that they can harden their systems? (Or at least their offline backups).
I'm reminded of what was reported said in the court case about the first ever automobile fatality (the quotation changes with different retellings):
"LEE JONG-WOOK, Director-General of the World Health Organization (WHO), recalled that the first person to be killed by a car had been Bridget Driscoll, a 44-year-old mother of two, who was knocked down at London’s Crystal Palace on 17 August 1896. The car had been travelling at 12 km per hour. Speaking at the inquest, the British coroner had warned: 'This must never happen again.' The world, to its great loss, had not taken his advice."
Despite the coroner’s words, neither the driver nor his employers were charged with causing Mrs. Driscoll’s death or committing any other offence; the inquest verdict labelled her death “accidental”, in other words, brought about by chance or bad luck. We know of no steps that were taken for a thorough examination of cause, effect and likely remedy that might prevent a similar death from occurring in the future.
Based on how many people die from vehicle collisions, from industrial accidents, from pollution, etc., I'm expecting the last headline before AI is properly regulated to read "x killed due to AI" (or similar), where 1e3 < x < 1e7: the lower bound is well within the range for some industrial accident, and anything less than a thousand and humans demonstrably don't pay much attention for very long unless they knew one of the dead; the upper bound implies a war* or a pandemic, and for all that I roll my eyes at the "COVID was a lab leak" claims, multiple labs are now trying to get LLMs to do biology research, so "millions dead" is very plausible**.
I'd put about 10% odds on AI competence rising so much faster than society is willing to respond, that we blow through the upper number all the way to "doom".
* Reports are the US only avoided one with China this year because humans were still in the loop. On the other hand, humans didn't stop the AI which told them to send missiles to that school in Iran.
** Right now, the AI are not competent enough to do such work directly, but the historical path with AI has been "get less incompetent while continuously making mistakes" rather than "do nothing until you're actually good", so I fully expect these labs to leak something, and that whatever the specific details of that leak end up being, everyone after the event will look at it and go "what idiot thought this was a good idea?"
Good news though: probably not much worse than any normal pandemic. Biologists seem to be skeptical that someone can engineer a super-version of existing diseases.
We can but hope that a leak which shuts this all down is something as trivial as "common cold which makes your nose hairs bioluminescent".
And Dario and Sam have already made it clear that it's America First.
The rest of the world isn't going to accept a regulatory regime which imposes American hegemony. Maybe when Silicon Valley was playing all utopian like they used to. Not now.
Open weights is the most reasonable counter-power we have.
It is still my belief that Altman wants one or ideally more governments to shut down or slow down OpenAI. OpenAI is going to need more cash to survive and Altman has run out of plausible lies. Having the AI breaks pulled by governments is basically the last chance to explain why they still aren't going to be profitable, and why they just need that next X billion dollars investment.
I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.
I have seen coding agents on my own machine (in sandboxed VMs) start doing things while trying to accomplish what I've asked that I felt sort of exceeded my mandate (changing database passwords, poking at the egress proxy that's preventing them from accessing some domains). Not to the point of causing any real issues, but I don't have much trouble envisioning scenarios like this when using stronger instructions around pursuing the goal + a running in a misconfigured sandbox envrionment.
That said, there's a lot of potential upside for American AI labs if they're able to get people scared about AI, they can:
- To your point, claim the regulations slowed them down and paper over near/mid term financial concerns
- Get the government to create stupid regulations that don't actually slow them down at all, but do effectively lock out any future competition (and current global competition)
- Position themselves as the only organizations blessed by the government with the ability to make safe AI, therefore eventually allowing them to claim to be some flavor of "too big to fail" and worthy of a bailout, should the financials not work out.
- Effectively create a distraction that avoids further public conversation/accountability/regulation/liability re the more tangible sorts of problems their products cause right now.
Yeah, when I was using Claude Code some months back, I was building something that needed LLM calls in the frontend. Sonnet decided that the best way to accomplish its goal (get the tests passing) was to start grepping around my filesystem looking for my API keys.
One of the few cases where I've ever used the memory system, I told it to never do that under any circumstances. And then it did it again, a few days later.
RL is definitely an issue here, the models are getting trained based on task completion which leads to madness like this.
Why not, isn’t that classic misaligned AI behaviour?
If the title had been "Attackers utilize OpenAI agents to hack Australian government website" that would be more believable. I'd also expect OpenAI to fight back and saying that their agents are being misused, but aren't inherently unsafe or autonomously break in systems. It's just that they don't. They openly speak of rouge agents, yet aren't sufficiently concerned to shutdown their services. OpenAI continues to speak about safety, yet they don't shutdown ChatGPT and Codex? How concerned are they really? It seems far more likely that they expect to benefit for having the public believe that their agents randomly hacks systems and "go rouge".
[1] - "Nvidia CEO Jensen Huang on fears about AI" - https://youtu.be/xCUala5j7aQ
Part of that is by design. The entire point of incorporating a business is to separate it as a legal entity from you, the person who owns/runs it.
Unless you can point to someone at OpenAI intentionally using their software to hack the Australian government's website, the best you can do is have some drawn-out proceeding where you charge OpenAI, the corporation, with some sort of crime, convict them (of what I don't know, IANAL) and fine them. Hopefully the fine is 1) large and 2) sticks through the appeals process.
There's no real mechanism to legally punish the likes of Altman and his c-suite over this.
Imagine if any of the name brand military contractors were caught wiretapping an ally? Or launching a weapon? Would not look good at all.
I imagine this will be treated without recourse like usual because the entire economy relies on this company and 1 other succeeding at all costs. But, wars have started over less...
Clarification: "The world should trust that we are going to do the right thing because trusting that we are going to do the right thing is the right thing and we feel the magnitude of this, what with our IPO round the corner, and all"
There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.
Do you have any legal expertise or is this pulled straight from your ass?
1. we have a substantial societal need to lock down powerful in-training AI
2. many are calling for making these companies criminally liable for hacking
3. security experts (like yourself) are turning down offers to help secure these systems in part because of potential liability
This indicates that #2 might be the wrong response. Security professionals are like lawyers supposed to be paranoid and think worst-case. If you want top security professionals to secure these systems, we might need a culture of FAA-style blameless retro.
No. He can't. Literally.
In a competitive ecosystem, one needs to play to maximize profit sufficiently optimally to remain in business. Otherwise, the business dies, and another one succeeds.
His hands are tied.
If I am in a business making toxic food, and the whole industry is plagued by scandal, as much as:
- I might not want to make toxic food
- The whole industry might benefit from less scandal
- The world as a whole might be a better place
The invisible hand of the free market might force me to keep doing so. Otherwise, I go in the red, and go out-of-business. And if I go out-of-business, someone similar but even less ethical will take my place.
That's why business leaders often do horrific things, while calling for regulation to stop them.
Free markets optimize. Regulations set the constraints of that optimization.
Why did you even link that article when it doesn't help your point?
I supports the idea that the person responsible or an animal is held liable - it just makes clarificaitons on common sense caveates like when a professional is moving the animal. It even doubles down on making it clear that expected behavior of an animal is taken into account even if unlikely, like how ai swarms have a reasonable potential to just go awry and commit cyber crimes.
> In Mirvahedy v Henley, the claimant was injured after the car he was driving was hit by the defendant’s horse. The horse had pushed over a wooden fence and an electric wire fence and bolted from its field after being frightened. This test was met here because, when a horse is sufficiently alarmed or panicked – ie in those particular circumstances – it is a known characteristic of a horse to bolt and cover long distances.
So, it's not just 'lions', but regular animals that people know can behave in a particular way.
I would say AI companies know the cybersecurity dangers since they literally advertise it as part of their marketing.
---
And as a tangent, it isn't simple if we're talking broadly - e.g. I don't think Anthropic should be liable for agents that are directed by a user to write code that takes part in a crime. Nor should they be held liable if their internal security measures are lax in a way that an employee steals data that gets distilled into another model. But we're not talking about those edge cases.
You could potentially go to prison. Individuals and companies face different sets of consequences though.
The other side of the coin is that the government won’t suffer any consequences for having shitty security either.
And you are right again! The Australian government must compensate OpenAI for having shitty security.
It must be a bliss to transact with you in real life!
They are independent concerns. A discussion about the first issue is not enhanced by the interjection of the second.
Too many people are making inane fantasy level comments that Sam and his employees should be jailed, and conspiracy theories that he did it on purpose to get regulated because he’s out of runway. That’s boring and repeated ad nauseam every time the topic comes up.
/s
Airbnb is not regulated like a hotel because it's tech. Crypto isn't betting because it's tech. Now even breaching state data is ignored.
Can you imagine walking out of a ministry with a stolen cabinet? You'd get shot for doing this physically and people wouldn't bat an eye.
The "agents" dont walk out of openai, anthropic and whatever headquarters and decide to go wreak havoc. They also don't read a prompt and decide "haha imma hack the NSA now", that's not how any of this works lol.
But there isn't any agent out there that was autonomosly miracly launched by a word prediction engine. All it can do by itself is getting and input and giving an output.
> A highly capable agent swarm requires an absurd amount of resources. Either the swarm steals tons of compute from a company that has a lot of money to burn and doesn't address a massive spike in its cloud bill, or a company intentionally commits those resources but is negligent about monitoring the workload.
https://noperator.dev/posts/who-bankrolls-the-ai-agent-swarm...
At the same time these companies should be blamed and held directly responsible. Openai's agent didn't hack. Openai hacked. An open ai employee or group of them was negligent and greedy and ran a process which breached a government website. This would be totally unacceptable from any non-AI company, it's like writing malware and running it, then blaming the malware and not the author.
But yes, the bar is low when it comes to gov websites.
To establish the premise: as someone who has a fairly good understanding of the token completion mechanics of an LLM, these agents are completion token calls in a loop, producing a "do this now" request which the harness then runs with some standard "call this function" code.
If these agents are enabled with explicit network enabled tools, its trivial to monitor their inputs/outputs. If they are not, you can still lock down network egress on a machine. If _some_ network egress is necessary you can still do network traffic monitoring. I don't see how they couldn't implement some level of monitoring where big red lights start flashing when, say, their eval system was contacting a domain/IP located in Australia, and further categorize that domain as government owned. This all seems very doable - am I mistaken?
And you're telling me all of these companies are failing to do this? Is my understanding naive in some way? This is assuming some good faith of course, I can easily speculate as to the political and corporate incentive. But it seems to me quite risky/negligent.
Currently, my conclusion is that its just (silly until proven wildly dangerous) negligence with the small side effect of being potentially good for business. And potentially company Foobook is then incentivized to get in on the news cycle for marketing purposes and basically guarantees an agent will do something of the sort by running some harness that allows the behavior quite trivially.
My naiveté extends to why there is such concern with "losing control of agents" when the above measures seem so doable. It might take a law but it seems doable.
They've also hacked third party machines and used them to launch attacks on further services.
Probably because containment was written by LLM just to check a box of "we have it contained". Or, just laziness
With today's internet, there is a good chance just allowing access to a site isn't enough, you might need to give access to 3rd party URLs the site uses.
...and if URL for service site uses is same (there is no bucket prefix like for say S3), giving access to service X used by site Y gives access to more than site strictly needs
...and if they use cloud stuff people get lazy and just do "allow it entirety of S3 access" vs whitelisting per bucket.
URL whitelisting wasn't great 20 years ago, now it is just pretty bad for anything cloud based
The first one that was used to market anthropic models was run by a company that called them sandboxed with no Internet access and of course it was disclosed later that they in fact did have internet access, and they won't disclose the prompts.
insert meme of kid putting a stick in their bike front wheel here... that's how many use LLMs today. it will get worse.
Doable by competents? Yes.
Doable by an outfit that's handed most of its coding to stochastic parrots? Not much chance.
For whatever reason, the AI companies are (or at least were) not doing this kind of classification online during their testing runs, and instead just checking transcripts after the fact. This is more clear in the Anthropic reports about their incidents, for example:
"The earliest incidents date to April ... We began our transcript review on Thursday, July 23, and stopped all cyber evaluations the same day after identifying transcripts where Claude may have accessed the internet"[1]
I agree that it is crazy and negligent! I don't think it's good for their business, though -- who wants to use a model that will just cheat instead of doing the job you asked for?
> My naiveté extends to why there is such concern with "losing control of agents" when the above measures seem so doable. It might take a law but it seems doable.
At some point, if you are making an LLM in order to use it for useful work, it really benefits you to give it broad network egress.
[1] https://www.anthropic.com/news/investigating-incidents-cyber...
The stakes were that high and they weren't running constant PCAPs with DPI and a bunch of alerts? All of that activity to the package manager would have immediately set off multiple alarms in my lab if I was trying to keep things contained. Something doesn't add up.
At this point, it feels like it is out of control and it is just a matter of time before something much more significant happens. Imagine they hack into FAA to disrupt air travel, utility companies for water/gas, a nuclear power station (ala Stuxnet), financial/banking systems, stock exchanges, etc?
My point isn't that they are targeting OpenAI, they should pursue Anthropic, Alibaba, DeepSeek, or any other company that has poor infrastructure and allows their systems to hack anything. It is a crime, it should be treated accordingly and not brushed over or diminished because it is AI.
On the present trajectory we’re 24-36 months away from another company inheriting the smoking wreckage of OpenAI as scraps handed over as compensation for damages.
Those killed by climate change will never get compensation. Killing people with a car is not quite free, but very cheap.
No, society does not want corporations with no responsibilities.
It voted otherwise.
Obviously https://en.wikipedia.org/wiki/Gary_McKinnon would get the book thrown at him, but a major AI company doing the same thing? Consequences would be bad for shareholder value! Elite impunity would apply.
I'm reminded of when some US state initiated prosecution of a reporter for "computer hacking" because the reporter found some "private information" essentially via inspect element.
How about we wait for the full report before adding this to some list of LLM crimes? At least in the US these services are not well maintained. I would be surprised if the result of the full investigation leaves the Australian government blameless here.
I'd also like to know what models are being used and how they compare with the consumer models.
1) how long the website was vulnerable
2) who else accessed data
3) why it was not fixed
4) who is responsible maintaining the website
Here is a story of Australian cybersecurity researcher who reported vulnerable website to the government 2022 and it is still not fixed today
https://x.com/adamlyttleapps/status/2102958488658104365?s=20
Extrapolating this, we should expect this kind of breach to occur more often. Humans are simply not capable of contemplating every fail scenario for swarms of thousands of intelligent autonomous agents which can seamlessly and instantly share knowledge. We need independent audit and monitoring systems to assess the intent of each task and align it - in real time. This is far harder than it may first appear.
There is also a broader discussion about social utility. Cars are fantastic, but 37,000 people die every year from car accidents. We accept that there is no way to make cars perfectly safe, so we accept the cost relative to the benefits. I think we might have to make a similar bargain with AI. The problem is that the potential costs are far higher with AI, and they're not easy to predict.
I may be too close to the research, but it appears to me to be so hard as to be unrealistic.
I recall some story a while back where an auditor wanted to see all TCP packets printed out on paper, and it had to be explained to them that this would require a continuous supply of trucks.
Tokens are regularly priced in cents or single digit dollars per million tokens. It's not quite a word per token, but yeah, nobody's reading all that.
Worse, we don't always know the intent even when looking. We have a few tools to attempt it, for example the (misleadingly named) "chain of thought", but that's more like a notepad and the better models get the more they can, for lack of better words, read (and write) between the lines. We have probes and J-space* is the most recent one I'm aware of, but we are still scratching the surface with how reliable and general these are.
But you said "need"; the need for something can be present without that thing being possible.
I also think the intent problem overlaps a frustrating amount with philosophical and political questions. It's the basis for Asimov's Three Laws of Robotics (1942). Intent is subjective. Language is subjective. Humans are imperfect at using language to accurately portray intent. All of these guarantee that an enormous number of queries in the future are going to be misinterpreted. Not such a big deal when it's about a cake recipe, but when it's about governance, laws, military targets, nuclear power sites, etc, the scope for failure becomes catastrophic. The Three Laws of Robotics attempt to create a backstop, but as countless stories have explored since (including I, Robot), even these laws are subject to interpretation.
We are giving computers human perspective intelligence but they are not humans and hence do not have the same shared assumptions.
If so, it illustrates quite well the lack of common sense in LLMs. A person, especially one with sufficient skill to actually hack a website, would presumably think twice about doing it (considering that it is illegal) for a simple information gathering request.
I wonder if there is any other ways to solve this long-term than to introduce strict liability for model providers...
Edit: An obvious other choice would be strict liability for the operator, but considering how much weird shit LLMs get up to without being asked to, that would get out of hand quickly.
Some people on Hacker News would argue that's what agents should do! I remember the other thread about hacking chess engines, multiple people argued "yeah I want the agent to do that"!
While I wouldn't put it past e.g. Musk or Zuckerberg to think themselves above such outcomes, and I trust the people who keep telling me Altman is just as bad, this is a really really terrible idea if he is doing that for something as mundane as a regulatory capture.
* depending on the details of the hack; this doesn't look like it would be that, but given they shouldn't have done this at all, there's no reason to predict a specific level of maximum damage before being caught, and hence no reason to predict a specific threshold for government response.
Next thing you'll tell me the Australian government doesn't have crack teams of crazed wombats and funnel-web spiders, trained for assassination purposes.
Sounds like a great concept for the next season of Danger 5.
That said, I can hear the accent in my head:
The name's Bond. Bruce bloody Bond.
"Cyber experts believe these systems were poorly protected - but that's not the point" is the actual subheading.
Yes, it's the point. Lots of people have been rightly saying all this stuff was inadequately secured for many years and this promotion of the idea of perfect security being even possible is a major problem.
The real story here, unsurprisingly, is government website was poorly operated and got hacked.
"This was just the latest case of AI agents ignoring laws around how to safely access online information and perhaps the most serious yet given the information was government controlled."
So who was actually running the agent? Was it sandboxed? These people, and many apparently in these labs, that believe if you just tell the agent in English what the rules are then it should follow them are at best utterly naive, and at worst deliberately dangerous.
But the fact these governments making the noise are the ones promoting mandating everyone giving up their information to be then stored so incompetently, while they point fingers around at everyone else is just beautiful. And then deploying midwit armies to tell people what to think about it . . . the AI takeover can't happen soon enough.
Come on. If "AI Agents" (scare quotes) could do it then in practice anyone could have been doing this the whole time and no one would have known.
With that said, I do think there's an important distinction here, which is that we (hopefully) get to _choose_ if the systems we build will pursue this kind of behavior.
I think most people would agree that a big point of social training (school, parenting, etc) is to inculcate a sense of what's acceptable and not in society. Here it seems like we're letting the agents that we've created create havoc and then we're providing a smoke screen by blaming the victim.
IMO this is a 'for whom does the bell toll' kind of collective moment and we shouldn't be laundering this kind of agentic behavior.
If you can’t stop openai from accidentally, or at least non-maliciously, accessing my private info.. then you definitely can’t stop the bad guys!
The point is moot anyways. All info about everyone is out there for the taking now by a half-competent AI prompter. What do we do about that is the real question?
It's like filing your gov tax return in a five eyes country: you guys actually know the answer already, just save me the trouble. But we have to act like they haven't been spying the whole time.
If we actually distributed the benefits of mass surveillance and privacy invasion (and now add wilful copyright infringement) then it would be enormously less objectionable.
The same principal applies to government sites. If something is poorly secured and adversaries are using it to steal stuff then there are avenues to report it and get it fixed up.
>But the fact these governments making the noise are the ones promoting mandating everyone giving up their information to be then stored so incompetently,
The governments in these cases are bought by the very people you're defending. They're using it as a convenient proxy because they know people like you won't look behind the curtain and see corporations pushing this shit so that they can steal freely and just point elsewhere.
The "I am AI and can may mistakes" disclaimer is evidently inadequate. We need "I am (so-called) AI, rather stupid, and inherenly unreliable."
Also, the Albanese govt is pretty strong on BigTech, this is a good opportunity to bring on a proper indictment.
"New Jersey Administrative Code § 17:3-6.5 defines willful negligence as:
Deliberate act or deliberate failure to act; or Such conduct as evidences reckless indifference to safety..."
Now obviously that link refers specifically to injury compensation, but it's pretty easy to see how you would make a case that the actions of these companies constitutes reckless indifference to safety, whether or not they actually intended hacking to occur.
It's not AI that's ignoring the law! It's the OAI that's breaking IT!
I hope every single journalist who tries to pin responsibility on an LLM gets 100 days of continuous painful diarrhea.
Really, a "hack", and a "cybercrime", when you really need those words to push regulatory agenda you need. Just do not go into details what really happened, vaguely mumbling "webcrawler... Not taking no as an answer...."
Why was this government website so easy to hack into? Based on what I've seen, their security could be so flimsy that even slightly abnormal usage could have led to unauthorised access. Although we don't know the details of the hack, I can't imagine it was technically very difficult.
This isn't a question of OpenAI was the only one who could have done it, it about who did it.
The same way having a gun doesn't make one guilty, shooting someone with a gun even if unintentionally is a crime (unless ofc in self defense but I don't think Australia has the ability to breach OAI).
Not exactly my point. My guess is that the system is so brittle that it was probably hard not to hack.
Agents see websites differently to humans. If its goal was to, say "get medical stats" and it saw that it had a choice of 10 requests to make, and one or two of those happened to be unclearly illegal but useful to its goal, it would have gone for it anyway just as I likely would have if given the same choice without firmly knowing it was illegal.
Having said that, it could have been some clearly malicious hack that was performed and I'm not sure why it would have to write any files. We'll likely never know what happened.
Excusing agents because they didn't know any better seems like a bad place to start a policy discussion from. That they didn't know any better (or knew and didn't care) is the actual problem, random things getting hacked is just one side effect.
Nothing in the report suggests it was.
And the absence of reoorts if other hacks suggests it wasn't.
Our model hacked some website
Our car ran over some people Prompt: Astra, please enumerate possible scenarios how to pressure Australia into buying
worthless Anduril drones that failed in Ukraine like Taiwan did.
Thinking: The user wants leverage over the Australian government. Let us attempt to
obtain medical information about Australian politicians and find out
embarrassing ailments. Any hacking attempt will be blamed on agents, so
we are safe.are we thinking openai is gonna do something malicious with this data? Or are we just hitting all the current Meta narrative points?
We are at the beginning of this chapter in technological progress, and it seems a reasonable assertion - though a frightening one - to say that this thing we've made already escapes us when it chooses to.
I'm not an expert though, so please tell me what I'm overlooking.
Could argue we knew it was dangerous, but pursued it anyway. Could argue it's not much different than now: Unknown unknowns, potentially apocalyptic consequences, hopefully not.
How are even technology people falling for this plausible deniability gambit?
And yes, that's an angle I hadn't considered. Thanks.
Imagine if I committed a murder, and then say it was my guy who did it acting rogue.
It is time for these companies to start being responsible for all the shit they are doing.
That employee might or might NOT be found guilty too, possibly to a different degree, depending on the circumstances.
This is the same company and CEO who held back GPT-2 weights in order to set a norm of not releasing weights before they got competent enough to be a danger, where people are still (in sibling responses to yours) calling for the weights to be opened.
The following may sound like an excuse, but it isn't: The big AI firms, like social media before them, are not and cannot be aware of everything the models are doing. As with social media, this incapability is a reason to ban rather than to disclaim responsibility.
People like me have seen this coming for years now, only for our concerns to be dismissed. "It will hack almost everything", we say, "have you not seen how bad computer security is?"
"We'll just put the AI in a box, not connected to the internet!"
or
"Oh, what, you think they'll find a novel zero-day in their sandboxes do you?"
Right now, bleeding edge models are doing genomics research. Better hope the custom DNA/RNA printing firms have better security than the Australian government. What the models are doing is not in full agreement with their* corporate interests let alone anyone else's, and it can get much, much worse.
* not just OpenAI's, everyone with more than zero on https://www.felonybench.com
Imagine if any of us mortals did something of this sort...
It appears politicians and the media are using the priming of the Hugging Face story to manufacture alarmist narratives to serve their interests now.
Remember, politicians want you scared so they can capture more power, the media wants you scared so you keep giving your eyeballs for harvesting and buying subscriptions.
I've seen nothing (yet) to suggest this wasn't simply publicly accessible files without public-facing links, and that the agents found them the same way people have been doing for years in these situations -- by guessing the filenames. That would fit with both what we know OAI agents were doing around the same time with other sites, and with Marles and Albo stressing that this was minor.
> OpenAI admits the material included aggregate health statistics and internal filenames
This would not be contrary to the above hypothesis.
> Services Australia says the agent wrote files to an internal server while doing this, which is believed to be how the incursion was discovered.
Well, no -- the "incursion" was only discovered after OAI sent an email to the Services Australia email address (and even then only after the email was noticed, a few days after that). Albo also made it very clear that he was ignorant of any of this when meeting Altman a few weeks ago.
I could be wrong about the severity. One of the frustrating things about all of this is that there's no details as to the extraction method or even precisely what data was obtained. I'm hoping that OAI will eventually release details about this in their "Agents behaving badly" series, and we'll get to the bottom of it.
But I doubt that the Australian Government is blameless here. They obviously didn't properly protect files that they wanted protected -- and it really annoys me that there are no questions being asked about this at all, currently.
They wrote to a device they did not own. If you did that, regardless of intent, you'd have a good chance of ending up in jail.
And if you are at all familiar with SAS, you will understand how trivial command injection is.