Hacker News Reader

Comment by no-name-here | Hacker News Reader

> can only create a sandbox that a half skilled human operator could have broken out of easily

The exploit:

> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]

Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

[1] https://openai.com/index/hugging-face-model-evaluation-secur...

View on HN

The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.

Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.

This is all documented by OAI, with a timeline, here:

https://openai.com/index/hugging-face-incident-and-the-road-...

To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn't a great idea.

The more you look into the details of this thing, the more it does your head in.

Probably wasn’t a human operator at all that detected this. Just someone’s agent saying “I noticed there’s messages on the artifactory and the agents are communicating, here’s the fix I propose” from someone’s codex, skim read and someone just going “yeah sure go ahead”

> The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.

This annoys me so much. Everyone is acting as if the model went rogue, when it really did exactly what it's been trained for. This story is so fucking engineered.

Those PR stunts are the paperclips we cannot stop.

That website makes it look like they're so proud of what happened. I don't think it was 100% deliberate, but they really were not concerned about their models doing something shady

They LOVE the PR and the “danger” angle bc it makes them feel powerful. Dont think too much about the psychology…

It's almost as if they don't even realise just how bad it looks.

But in many ways they've gotten away with it. I don't feel that the full scale of their incompetence/malice/apathy in this event has been adequately reported on, and they seem pretty chuffed with how it all played out.

It's irresponsible to a degree that it should have some very serious consequences but I highly doubt anything at all will come of it.

> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

Yes. It’s a fairly simple SSRF attack as far as I can tell. One of the first things I’d try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.

Calling it zero day makes it sound elusive. It’s a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, don’t get much scrutiny and are often riddled with issues.

> Calling it zero day makes it sound elusive ... has like 40 CVEs this year alone.

1. I clicked into ~20 of the recent CVEs, and it looks like about all of them came from openai or anthropic research?

2. In fact, every CVE in the last 2 years came from the last few months -- i.e. other than these recent CVEs, you'd have go back more than 2 years ago to find a single other CVE.

So like I said: unscrutinised and riddled with issues.

>>>> a sandbox that a half skilled human operator could have broken out of easily

>>> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox

>> Calling it zero day makes it sound elusive ... like 40 CVEs this year alone.

> [The CVEs] came from openai or anthropic research? In fact, every CVE in the last 2 years came from the last few months

I still don't understand how you can specifically point to zero CVEs being discovered by humans for year(s), then ~40 being discovered once OpenAI/Anthropic start looking at it, as the only(?) piece of data you provided to support the claim "a half skilled human" could have "easily" found/done this?

Well, yes. You don't find something if you don't look. The LLMs did the looking before the humans did, so they found the vulnerabilities.

It's definitely worthwhile to point out that LLMs have changed the economics of looking; we can now look in far more places far more quickly than we could before. But the above points were that the vulnerability itself happened to be an easy one that didn't intrinsically require an LLM.

Isn't the issue here that they used something like Artifactory for this purpose in the first place? It is not intended as a robust and secure packet manager for sandboxes. Why would any human look for vulnerabilities in Artifactory at all? So the scenario never arose where a human was incentivizied to scrutinize Artifactory, something that was never needed because it was never intended for the purpose of being a packet manager in a secure sandbox.

Means the problem lies elsewhere: deciding to use it at all. That points again towards the lack of basic SysAdmin skills.

> the lack of basic SysAdmin skills

If we exclude those who lack "basic" level skills, what solution would most typically be used by someone who has "basic" level skills for this use case? And what solution for someone with "expert" (or medium) level skills?

Don’t let software inside the sandbox access the internet on request. Have a package repository with approved software the agents might want, and push to it over a channel that is ingress only. I can imagine ways of breaking out of this, and come up with a lot more to mitigate, but this would be fairly basic stuff that’d be vastly superior.

> ingress only

The issue we're discussing happened while running ExploitGym. Doesn't the ExploitGym arXiv test explicitly have external "outbound" network requests as allowed for package installation and supporting toolchain installation?

Part of the toolchain had full internet access. Agents had access to Artifactory, Artifactory could do whatever it wanted. So even locking that down to certain external sites would have stopped this particular attack.

Well the most obvious thing would be to gate access to artifactory behind tools. E.g. apt get install is a tool and only the tool gets access to artifactory.

Right, that’s basically what I’m saying. There’s not zero use for an attack like this, but it’s not a likely situation.

Said another way: if Artifactory was somehow a common gatekeeper between grounded techy teenagers and their access to internet porn, this would have been found ages ago.

Why would it be getting attention? There’s an enormous amount of garbage software out there, and not an endless supply of researchers. Especially tooling like this, used internally where it’s assumed that security is the job of something else. It’s far from the first time serious but simple vulnerabilities have gone unnoticed for a long time. And the use case of having this be your way to the internet is probably rare, so nobody has tried very hard (or tried but never told anyone).

I’ve looked at the CVEs a bit more and it’s just very clearly a pattern of systemic issues with validation, be it URLs or tokens.

It just shouldn’t be that hard to believe that OpenAI just didn’t care very much and thus did a crap job. The whole model of the sandbox is terrible, so why would they bother thinking about the implementation much?

The amount of garbage has also dramatically increased lately, spending a lot of that attention in places other than a normally-not-attacked-at-all entirely internal system.

It really isn't a high profile target, lots of fairly obvious issues is to be expected. Which is not at all a good thing, but very obviously the status quo for large software.

What's the point in hacking the internal corporate artifactory instance? You'll get fired if you mess around with it.

Not to mention the only thing the artifactory instance got you was indirect internet access which is irrelevant if the artifactory instance is publicly exposed.

Also it might not be obvious but artifactory is proprietary software that used to be open source, so nobody is looking at the code anymore.

The key is motivation. It's a lame package proxy. No human was motivated to use it to access the Internet.

Nobody sane was relying on artifactory for this kind of adversarial workload

In fact, the narrower the constraint the simpler it becomes to focus effort towards the remaining opening. To any problem solver which these agents are it is like being given a hint. Real defense often employs misdirection, message black holes, honeypots and other boobytraps, not a jail with one obvious door.

> the commercial package registry cache proxy

Any closed source program is insane liability. Trusting in competence of one company is the easiest way to get burnt.

Part of my day job is to set up testing of our product in air-gapped environment. It's not difficult. There's a straightforward way to ensure no connection to Internet (we use KVM, so, I just edit the VM description and remove the NIC from it). It's not any kind of rocket science. The tests then communicate over serial console.

The reason we have to test it isn't because our product would randomly break into someone else's system, but because it's meant to be sometimes deployed in systems disconnected from the Internet and we need to make sure the image provided contains all the necessary parts to create and operate such a system.

The whole setup where they "tried" to isolate the test but failed is laughable. It's like if an adult tried but failed to tie their shoelaces.

Hey, even Joey can hack a Gibson with enough determination.

Reply on news.ycombinator.com