Hacker News Reader

Comment by llama052 | Hacker News Reader

It’s a little concerning to me that it appears that openAIs sandbox consists of a web proxy and not stronger controls that would actually isolate traffic and report patterns to whoever is responsible for overseeing these research models. It should border on closer to an air gap network more so than a proxy.

I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.

View on HN

It's even more concerning that not only did they setup a bad insecure test environment, but nobody realized what was going on for 5 days?! Neither nobody at OAI looked at what their test was doing, and HF didn't realize that they had been pwned for almost a week? And then HF couldn't investigate without the help of an LLM? Both parties come off as incompetent here, and the coverage of this story is very stupid. What the model did was concerning, but the real 5 alarm fire is the gross negligence and lack of any basic forethought at both ends of this attack.

agree. If oia and hf are anything like generic startup / corporate america, IT and security is always seen as a cost center and goes to the lowest bidder. Do we really think they're employing top notch sysadmins?

Repercussions is a polite way of putting it.

…as is your ‘a little concerning’ comment about the eval infra. It probably didn’t occur to them that they have something capable of doing what it did - despite Anthropic writing a very similar story in the Fable model card, except it wasn’t a felony there. Maybe.

One does have to wonder what else OpenAI didn’t catch.

The exploit gym setup explicitly allowed access to package registries and v8 sources. Putting a cache on that doesn't seem like a bad idea generally, except when there's a 0-day in the cache :D But yeah, for this sort of thing I'd be locking down very specific egress things and putting alerts on it that are entirely outside of the red network.

> I would argue that it's negligence and that's aside from the fact that if a human did this there would actually be repercussions.

I’m not sure “negligence” follows just from the controls turning out to be insufficient. Research involves mistakes, especially around novel failure modes. The question is whether the precautions were unreasonable given what they knew at the time, rather than whether hindsight suggests stronger controls would have helped. Doing it twice though would be negligent.

Caveat: I’ve worked with some of the people involved, so I’m probably biased toward a charitable reading.

“Research” generally doesn’t involve actively hacking third party systems though.

It’s definitely negligence given how they talk about their product. They are either lying when they talk about their fears, or don’t actually take it seriously enough to use serious guardrails. It’s very concerning

This is my main critique as well. How did they not see any and all outgoing traffic? And for 5 days?

OpenAI is the company that jumped the gun and forced every other company doing LLM research to rush to market something. Their objective has always been dominating the market, and strict development procedures don't increase market share.

This concerns me nearly as much as how little most people seem to care.

This "sandbox" barely sounds designed to be a sandbox, let alone a secure one

Reply on news.ycombinator.com