Comment by ben_w | Hacker News Reader

Comment by ben_w | Hacker News Reader

As I understand it, there are only three options:

1) OpenAI and HuggingFace are both telling the truth.

IIRC not actually a crime because no intent, it is a technological accident, civil responsibility only, but IANAL so it's good "not technically a crime" isn't load-bearing.

2) HuggingFace is telling the truth but OpenAI is lying becuase the attack was deliberately done by humans. Bad for OpenAI to do so, Fable was blocked for less.

I think this would mean government is obliged to investigate the case and put the responsible OpenAI workers in jail, because cybercrimes are a public prosecution thing not a civil case? Again, IANAL, but this isn't load-bearing.

3) both are lying, e.g. there actually was no attack whatsoever, which would be pretty weird for HuggingFace because they have no incentive to hype up capabilities of anything closed weights including all OpenAI models; and also bad for OpenAI because White House blocked Fable for less

(I suppose there's also option 4, HuggingFace hacked OpenAI to make them look evil, including planting records that made them mea culpa? A weird plot but in this timeline any nonsense is clearly possible).

You don't really need anyone to be lying here. It is likely that the broad strokes of the narrative are true and that no collusion or conspiracy took place here.

The issue is that a lot of important details in that narrative are missing, and the devil is really in the details here. I suspect that those details would make the result seem less exciting and that this event would move the needle far less for them if they were more forthcoming.

A decisive detail would be the prompt used. OpenAI gives virtually nothing here, not a sanitized prompt and not even so much as a description of how long the prompt was and what sorts of instructions it contained. Many are inferring the model behavior to have been fully emergent and unprompted, arising naturally from routine cyber-capabilities testing. But we can't know this because we don't know anything about the prompt or the context the model had access to.

Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.

One annoying claim from the OpenAI side is that long-horizon goals in real world settings are now effectively settled. Previously there were some bounded and tempered benchmark results, but now OpenAI can point to this event and announce "AI independently went rogue and escaped the lab, what more do you want?". This bypasses the need for anything quantifiable or wading through multiple detailed case studies to get a more sober view of model capabilities. It relies instead on the emotional weight of the spectacle.

Where do you see the claim that "long-horizon goals in real world settings are now effectively settled"? The argument you put in their mouth would be a bad one, but I don't see anyone making it.

https://openai.com/index/hugging-face-model-evaluation-secur...

> UK AISI’s evaluation shows that models such as GPT‑5.6 Sol are increasingly able to sustain complex, multi-step cyber operations over long time horizons. This incident implies these theoretical capabilities do apply in real-world settings.

I should clarify a bit more why this is annoying beyond what I wrote above. The main issue is that this was not a standard deployment, and the lack of particularities make the size of the gap between "real-world" and "benchmarking"/"lab" difficult to assess.

We don't know about the prompting, the context, the environment + configuration, or any other details that would allow anyone to differentiate this from a benchmarking setting.

Unless you believe that OpenAI was trying to get their models to break out of the sandbox and hack into Hugging Face, and aiding them in that goal, I don't see how any of those questions would really contradict that claim. The models did something in the real world! It required some amount of complex orchestration and long-term planning!

> A decisive detail would be the prompt used.

Beyond liability, why would this matter?

e.g. if OpenAI had actually instructed the model "here's the challenge [attachment challenge.md], do whatever it takes to win, it's fine to break the law", that's all the "AI is dangerously capable" part of case 1 with all the criminal liability of case 2.

And pretty much regardless of what the prompt is, high likelihood of stakeholders demanding Trump ban access to the model until this can be shown to be resolved.

> Another detail: how many times did they perform this particular experiment before they obtained this result? What were the outcomes of all the other runs? Many are assuming this was a one-shot result, which I suspect is what OpenAI intends for us to infer. But we can't know that to be true.

To an extent. But some of the other bleeding edge performance announcements half a year ago, e.g. "it wrote a compiler" or "it wrote a web browser" were measured in thousands of dollars. Zero-day exploits cost what? I genuinely don't know, I only hear occasional headlines about e.g. Apple 0-days being cheaper than Android 0-days and those kinds of headlines often pick the biggest number rather than a typical example, and were order-of a million dollars.

If OpenAI spent a billion dollars on tokens to get this result, only investors should be cross about it. If it took a million, you and I may not be able to afford it, but it's still a threat.

While the way this is written about may suggest it does this reliably on any attempt, which would be naturally horrifying, it's a threat well before that point.

Reply on news.ycombinator.com