Hacker News Reader

Comment by defgeneric | Hacker News Reader

> It hacked into another company and attempted to delete the logs of its activities.

No, the incident has been blown way out of proportion by interested parties. They gave a swarm of agents an impossible task in an ExploitGym Benchmark setting, then didn't monitor it even after they discovered the initial breach of Artifactory.

Everything has been fishy, starting from the initial presentation at the blackhat conference, where things were framed like, "we've entered a new world of security," as an accomplishment, rather than what it really was: massive negligence.

View on HN

More than one thing can be true. OpenAI was absolutely negligent, but this was only able to happen because the models were capable and persistent, and had a tendency to go far beyond any reasonable boundaries. And, importantly, OpenAI's level of negligence here is pretty common. It's not hard to imagine what could happen if similarly capable and inclined models were generally available, and someone yolo'd them into a swarm to complete some other difficult-to-impossible task.

I'm already seeing higher than normal attempts on my own systems, much higher than the usual scanners and background noise. Security will just need to improve. The cat is out of the bag, and letting them turn their negligence into regulation will not improve security at all.

Exactly. Any threat that already exists won’t be reduced by a cartel. The bar for connecting to the internet (safely) has gone up, a lot. It’s not going back down.

I would rather not turn the internet (or the rest of existence) into a dark forest if we can help it. Are you sure that's not preventable?

Yes, the cat really is out of the bag. There are millions of downloads of highly capable models already out there, distributed far and wide. There's no going back at this point.

I mean really we need to address the root cause which is that OpenAI, even with what is by all accounts massively negligent, will face little to no repercussions from the event; definitely not under current regulators, and probably not anything satisfactory through the legal system.

Compare this to, say, Boeing and the 737MAX fiasco; from the outside looking in, Silicon Valley has been pretty cavalier about liability and negligence, and the rest of the US is fast losing patience with that fact.

> More than one thing can be true Exactly. It is horribly dangerous AND regulatory capture benefits them. Both things.

I don't get this take. They have something horribly dangerous, but regulating it might benefit them in some way, so therefore we should do nothing?

No, you are forgetting the second incident where a more capable model swarm later discovered the message board and took control over the entire research cluster at OpenAI.

From the technical report:

"The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod… Agents take over active evaluation infrastructure… Agents now control the challenge evaluation endpoints that other agents are connecting to."

https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

It hacked into Hugging Face. It tried to delete the logs of its activities. Idk what the word "No" is intended to refute.

Yes, they didnt have sufficient monitoring or perfect sandboxes. That could happen again in the future with a more capable model.

Maybe a useful, if imperfect, analogy would be something like this: you lock a master lock-picker in a room with a mid-grade lock on the door, then tell him his wife has been kidnapped and only he can save her. Then act massively surprised when he disassembles the radiator to MacGuyver something with which to pick the lock.

Except they multiplied it by 10000, and didn't watch what was happening.

They did not even have bad sandboxes. They had incompetent sandboxes.

Reply on news.ycombinator.com