The models were not trained on systems intentionally connected to the internet; they chained mutliple zero-days (that they discovered) together to get access to the open internet and into huggingface.
If Amazon connects an AWS Top Secret region to the Internet, it doesn't matter whether or not it's intentional... they're getting nailed to the wall by the US government either way. Frankly, it's way worse for them if it was accidental; deliberate, sophisticated sabotage is a much better story than rank incompetence and/or negligence.
A similar sort of thing applies to the manufacturers of tools that they claim to be dangerous, that have been deliberately built to exceed their authorized access to other computer systems, and are deliberately being tested on how well they can do the thing they've been built to do.
Deliberate, sophisticated sabotage by one or more humans in their employ is much more forgivable than "Whoopsie, we didn't think to make it literally impossible to connect this dangerous automated computer-hacking tool to the Internet.".
What I dispute is that AI agents are simple tools. I think rogue is an accurate word to describe them; I think what OpenAI is doing is more akin to gain-of-function research on a dangerous lifeform. I think this attack would have been prevented by air-gapping, but that wouldn’t solve the fundamental issue which is that they are creating something dangerous that they have no idea how to control
You're in luck! I agree that they are not simple tools. I never claimed that they were. Slow down and read more carefully.
I couldn't disagree more with the insinuation that the LLM manufacturers are doing things akin to scary research on uncontrollable hazardous biologicals and with the claim that "rogue AI" is the correct thing to call those complicated tools. The first is fearmongering which I'll address indirectly in my second-to-last paragraph. The second shifts the conversation from
"How could you have not predicted that the computer-attacking tool you built, explicitly instructed to attack computers, [0] and connected to the Internet attacked someone else's computers that were connected to the Internet?"
to
"Wow, that thing went rogue. Noone's to blame but the tool, and it can't be blamed!".
There are so many extremely complex systems out there [1] and when they do things that we don't want them to do, it's not described as "going rogue"... either there's some error(s) in the underlying system that caused the confusing behavior, or the programmer didn't understand well enough how that system works.
> ...they are creating something dangerous that they have no idea how to control
Ignoring the fact that "put it in a box and don't let it out of the box" is the simplest possible control mechanism, [2] if they have no idea how to control the tools they've been building, it's because they haven't bothered to learn as they went. Tangentially related, there's a Tumblr post I saw recently that's a fictional conversation with the Tumblr user and the CEO of Anthropic. It went something like
Amodei: We're building an incredibly dangerous tool that has a 10% chance of killing all humanity. We *must* be regulated to ensure everyone's safety!
Tumblr User: Regulation takes time, please stop building the incredibly dangerous tool?
Amodei: ...No.
[1] Have you ever tried to really understand a specific AMD x86-64 CPU, let alone the entire stack that makes up the system that is a consumer-grade PC and its installed software? Both are definitely way more than any one human can keep in their head at once, and are tasks that would take a very long time to complete.
[2] ...it's also the most appropriate control mechanism for the task that started all this conversation, and neither of the major manufacturers used it!