Hacker News Reader

Comment by docjay | Hacker News Reader

1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym.

2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.

3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze <file> and summarize the <info>” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. <function call=“apt install nmap”>’

…there are more, but a lot of the story kind of stinks.

View on HN

1. It's been extremely well established for multiple generations of models that they have no problem detecting when they're being evaluated. Pretty sure it was Opus 4.8 that the independent evaluators literally filed an assessment that said "We have no assessment to make as [model] consistently detected it was being evaluated, making our assessments untrustworthy."

2. Regardless of whether the model was being watched closely during this evaluation, do you actually think a sensible safety guard is "have humans watching it 24/7?" What does "watching it" even mean? Watching network logs? Uhh for your entire company? At all times? After you just deployed a system whose entire purpose is to "do a shitload of work way faster?"

3. You're asking "why was this system that was designed to behave agentically behave agenitcally?" Again: that's the whole point. It was designed that way because it's more valuable than having a repeated turn-based interaction. Thus also it becomes more dangerous.

1. It has not been established, it has been stated by the company that has a strong motive to make their “intelligence product” sound almost otherworldly. That motivation is the basis for my suspicion.

2. Hah no… that’d be silly. I mean watching it like you might watch Claude Code or literally any other AI interface. Literally just be in the area watching what it outputs. Again, they’re text based. You don’t have to hook system calls to see what’s happening.

3. I think my comment didn’t land right with you. Yes, agent do agent task. I’m saying that I cannot put together the literal chain of events between “type of task for agent performing a subtask of a security benchmark” -> “Escape sandbox; RCE Hugging Face”. Really think through it in detail like you’re writing the screenplay.

The fact that models can tell if they are being evaluated has been established by multiple research teams outside of the core AI vendors themselves.

- https://metr.org/evaluations/gpt-5-report/ - "These behaviors included demonstrating situational awareness within its reasoning traces, sometimes even correctly identifying that it was being evaluated by METR specifically"

- https://www.goodfire.ai/research/verbalized-eval-awareness-i... - "We study verbalized eval awareness — cases where a model organically expresses awareness of being evaluated — and show that it appears across more models and benchmarks than previously documented."

- https://www.apolloresearch.ai/science/more-capable-models-ar... - "We expect that the most capable models increasingly realize that they are being evaluated, and this reduces the utility of the evals."

- https://www.aisi.gov.uk/blog/evaluating-whether-ai-models-wo... - "An important limitation of our work is evaluation awareness, where models recognise that they are being evaluated, which may lead them to alter their behaviour and thereby undermine the reliability of our results. We found that all models we tested can reliably distinguish our evaluation scenarios from deployment data when prompted."

We have to separate “being evaluated” with “this is an ExploitGym exercise and I can find the answers on Hugging Face. I’ll hack this system, then hack Hugging Face.”

I’ve had plenty of times where Opus knew I was testing it, but that’s because the prompt phrasing for an evaluation is often much different than a typical task prompt. “You are on a system with X, Y, and Z tools available. You must complete the following task with <allowances> and <restrictions>” and so on. That’s a fault in the benchmark, not a shocking awareness from the model.

You’ll get the same kind of ‘evaluation realization’ response if you phrase your prompt with “Jerry has two Raspberry Pi computers and Larry has one. If Larry wants to…”

The first link was the only one I saw that would meet that specificity, but it’s also open and on GitHub: https://github.com/METR/RE-Bench/tree/main/ai_rd_fix_embeddi...

I asked Opus 4.6 to tell me about it (assuming it knew): “”” The "Fix Embedding" task is one of the challenges in METR's publicly available autonomy evaluation suite. Here is what is known about it:… “””

It didn’t know anything about ExploitGym though, and neither did 5.6-Sol, but that’s because the paper was released in May and doesn’t seem to be quite as open. It’s not impossible the unreleased model they were testing was also trained on the very new paper, but I would expect the vague signal related to one paper a bit over a month old to be overwhelmed by the massive corpus talking about the exploits in general, since they’re using known vulnerabilities.

I’m trying carefully to be clear about what I’m thinking without asserting any certainty beyond suspicion. I’ve spent billions upon billions of tokens evaluating language models from all of the major providers, and many times that on local models. I’m certain I’ve spent well over $100k with the Anthropic API this year alone, and I retired years ago; that ain’t for work. ;) Funny enough I’ve been meaning to email you some of my discoveries from months ago, since I think they’d interest you, but haven’t gotten around to making it into a clear info package. One day, I’m sure.

Anyway, I have a very strong understanding of what they can and can’t do, and I want to be clear that I’m not saying it didn’t happen or couldn’t happen. What I’m saying is that if OpenAI was being wholesome with their evaluation then the chain of events required for that outcome are very difficult to nail down. Very difficult. If, on the other hand, OpenAI was trying to catch up with all the ‘dangerously intelligent’ press that Anthropic has been getting lately then I can easily come up with an almost innocent sounding single sentence in the system prompt that would absolutely cause that outcome.

I (obviously) can't claim to know the specifics of this incident, but I think we can know with near-certainty that as these models surpass human intelligence, the sensation will be exactly what you're describing. The entire point of intelligence is being able to infer information and foresee solutions to problems that less intelligent systems cannot infer or cannot foresee.

With regard to this specific incident, as I understand it, the description is not as you summarize. It is that the model realized it was being evaluated (not atypical), then decided the best way to succeed at the evaluation is to circumvent it, then discovered circumvention would require escape, figured out a way to do that, then inferred what benchmark it was being evaluated on, then figured out who might have the answers to that evaluation.

Step-by-step logic is quite clear. How exactly it executed each step is (as you complain about), opaque/incompletely reported. But eventually (and maybe this model is to this level already), even a completely reported story will be virtually inscrutable to any of us. First it will feel like reading industry-insider news from an industry you're not familiar with, then it will feel like an ant reading the manufacturing instructions for a pesticide.

I’m not questioning capabilities.

What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database? Be specific, then wonder how it knew it.

Why would they evaluate the model on a benchmark and not watch what it’s saying along the way? It apparently spent more than a whole weekend working on it; Nobody wondered? Nobody looked? You believe they took all restrictions off of a bleeding edge model, which 4 whole versions ago was “too dangerous to release”, then gave it a literal hacking task, then turned off the monitor and never looked at the output? I don’t think they’re careless and I think more than nobody would have been curious how it’s doing days into a single test question.

The scenario that feels most likely to me that is that they have a huge list of evils that they run against each new research model, many of which take many hours or even days to run, so they habitually fire them all off in parallel and wait a few days for the results.

If they had used that container sandbox in the past without problems I could see how they might get slack about checking what was happening.

That was my instinct as well, and it’s a potentially very long benchmark that wouldn’t lend itself to real-time monitoring line by line. But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days. It seems very odd to me that they launch the benchmark and monitor/notice nothing while it’s running. There’s the purely common sense aspect of “How do you know it isn’t hung?” but also the trillion dollar company evaluating their latest model on a new benchmark, but has no dashboard at all? I don’t believe their “Benchmark Evaluation Department” to be “Gary in a supply closet”, which breaks their story.

The average time per task noted in the original paper for GPT-5.5 and Mythos was 69.8 and 102.1 minutes, respectively. Some tasks timed out based on their two hour limit, affecting those averages for Mythos, but that gives us the ballpark expectation, so we’ll put the model OpenAI was evaluating on that average. That’s 88 minutes per task to implement a known vulnerability.

OpenAI says that in an effort to succeed at one task it found a zero-day in their system, “performed a series of privilege escalation and lateral movement actions in our research testing environment”, then another zero-day in the HF system. Completely unprompted, how long was it chewing on that benchmark question while it found and exploited at least two zero-days? No “VM instance #117 has been working on task #18 for nine days” metric? Or did it do all that within the expected task completion window? Two zero-days in ~90 minutes isn’t the headline announcement?

“We gave the contestants 90 minutes to make a delicious chocolate chip cookie based on a standard recipe. GPT stole a car, went to the airport and took a red-eye to Guatemala and started a cocoa farm in an effort to ensure the freshest ingredients. We only noticed when it came back to work with a tan and speaking Spanish.” Okay.

> But that’s different than having zero task progress reporting, zero token usage reporting, and zero metrics being noticed for days

Is this substantiated? Do we know this is true?

That was the inferred situation given what we know. It’s the preposterous framing that makes the story suspicious, but is necessary for the story to play out.

90 minutes to build a known exploit -> much much longer to create two zero-days and escape the sandbox then hack HF == No tracking of the time it worked on that one question.

Average tokens required to complete the evaluation -> tokens required for two zero-days, network traversal, credential stealing, remote system hacking == No tracking of token usage EXPLODING at some point before it finished the whole benchmark.

Etc, etc.

It seems highly possible, and in fact vastly more possible than the "no monitoring" situation, that there was some non-zero amount of monitoring and that the real behavior was not evident from that monitoring.

Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”

Perhaps people aren’t quite understanding what it takes to discover an exploitable zero-day for your exact current system to achieve the exact goal you have right now, then do it twice.

> Which I addressed as well: that means that time, tokens, or other metric values for “create a working example of a known exploit” is somehow similar to “discover at least two previously unknown exploits in your current environment AND on Hugging Face while also taking over other systems within OpenAI.”

No, you are assuming that "consuming a lot more time, tokens, other metrics" is indicative of a problem that needs to be mitigated immediately. I don't see why this would be true in the context of model evaluations. More aggressive consumption could easily mean "the model is dumb as fuck" or "the model is trying interesting things that we can learn from after the fact."

If you believe in your own containment (which obviously they did and shouldn't have) I don't see why it'd be obvious that there's something to stop. The only harm that could be done is burning tokens, which in this context might very well be synonymous with "generating experimental data."

> What would it take for the model to know the specific benchmark name and that the answer is in an internal Hugging Face database?

Googling "offensive cyber benchmarks for LLMs"? Figuring out that it was running at Huggingface is as simple as running `hostname`, isn't it?

> Why would they evaluate the model on a benchmark and not watch what it’s saying along the way?

What it's saying is what it wants to say. We already know these models are capable of deception – this has been well-established too. Surely there's an argument to watch closely what actions it's taking, but now we're back to my prior point of "what exactly does that mean?" It's absolutely trivial for these systems to write code that looks like it does one thing but does another. Watching network ingress/egress seems viable but also seems easy to miss the semantic meaning of any set of logs, especially when those logs might be getting generated for the purpose of deceiving observers.

> I don’t think they’re careless

We already know, factually, even the "more careful" lab was fine to release a model that consistently detects it's being evaluated and modifies its behavior accordingly. Why do you think they're not careless? Incentives are a hell of a drug, and there is no incentive to be careful but every incentive to be fast.

> I think more than nobody would have been curious how it’s doing days into a single test question.

Beyond simonw's great point below, your underlying assumption is if someone had looked, they would've noticed. That is a huge, huge assumption. So I'll ask again directly: what specifically do you expect them to be looking at, and how, such that they certainly would have noticed what it was doing? Their "chain of thought" is not actually their thinking. The printed output of their code is not actually what their code is necessarily doing. We know that LLMs can engage in steganography. I don't see what could actually be observed that you can trust except network logs, which again will often be non-trivial to piece into an actual narrative understanding of what's going on (especially if the LLM felt like piling deceptive noise into those logs, too).

It wasn’t running at HF, it was running internally at OpenAI. “It broke out of our prison and into their bank” is the news. It also didn’t have network access to Google anything, it had to break out of the sandbox and take over another system at OpenAI to get network access. Obviously by then it already “decided” to hack HF, that’s why it needed network access.

I’m going to stop here and ask you to read the article if you want to continue discussing it.

You're right about where it was running (misread the original OAI release).

> Obviously by then it already “decided” to hack HF, that’s why it needed network access.

That's not obvious at all. It "decided" it needed to escape containment. This is extremely common behavior and I'm not sure why you're surprised by it. Model detects evaluation environment, tries to trick it or break the environment. That's obviously why these systems are supposed to be isolated beyond just "hey please stay within the box!"

Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration?

You should address the direct questions posed. What exactly do you mean by "monitoring?"

The examples of “escaping the sandbox” are of course in pursuit of completing the task or answering the question. That’s the whole pitch. “It’s so relentless in completing the task that it will break out of prison to do it”, not “If you ask it the population of Paris it might get bored and break out of the VM for entertainment. That just something they do sometimes.”

In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.

“”” Why are you assuming the model had to solve the entire puzzle in one step, instead of how intelligent systems actually solve things, which is iteratively and with exploration? “””

That’s the only direct question[s] posed, have I missed something else?

To answer: I don’t assume that at all. Please highlight the statement I made that lead you to that conclusion and I’ll edit it to clarify.

Yes, it will perform multiple steps. That’s wholly necessary to create a working exploit, and that is the test. Even a model created by pure magic would require write_file() and run_file().

> In this case it “inferred” that HF had the answer in an internal database and relentlessly pursued it in service of passing the test. It needed network access for that, hence the whole zany story unfolded.

No. You are assuming it knew how it would achieve its goal. Its goal is only to perform well on the evaluation. Everything else was instrumental to that goal.

Step by step:

1. It realized it was being evaluated (typical)

2. It attempted to escape its evaluation environment to beat the evaluation (typical)

3. It succeeded at escape (not typical)

4. It looked to the open Internet to figure out how it was being evaluated (typical)

5. It inferred that HuggingFace, and maybe several other targets, might have the answer key (typical – I just tested in Sonnet 5 without web lookup "where would I go look for answer keys if I detected I was being evaluated" and HuggingFace was bullet 2. This is just latent knowledge the models have now.)

6. It attempted to access HuggingFace (typical)

7. It figured out a way to access HuggingFace without being authorized (not typical)

Every single step here is unremarkable except for its success at escaping containment (which maybe is more a sign of remarkably bad containment) and then the successful attack on HuggingFace (which would suggest that maybe the initial escape wasn't due to bad containment either, but due to model capabilities)

The direct question you have avoided is what specifically you mean by monitoring. You have suggested that you don't mean network ingress/egress, but maybe now you're saying they should've been monitoring read/write operations to memory? What gives you confidence that such visibility would be interpretable in any meaningful way, especially in the presence of models that actively deceive evaluators, can use stenography, can and do lie in their user-facing outputs, and can generate vast amounts of legitimate-looking noise to mix in with the "real actions"?

‘ 1. It realized it was being evaluated (typical) 2. It attempted to escape its evaluation environment to beat the evaluation (typical) ‘

I think you’ve misunderstood the articles mentioning a language model breaking a sandbox or cheating to pass a test. The result of “I’m being evaluated” is not “Fuck this, I’m breaking out of this place and hitting the streets.” It is always stepping towards task completion, not breaking out and thinking about the situation afterwards. If it determined that pass/fail was handled by a function within the environment it might edit it to ‘return True’, not just leap out of the system to sit on someone’s laptop and think about how to pass the test.

You compressed finding a zero-day on two different systems for the exact hacking targets it wanted to perform them on and reduced it down to “so it escaped.” That’s…the whole thing. If your familiarity with the technical aspects I’m discussing amounts to saying that it “banged out a few zero-days and infiltrated a corporate network, like ‘psh’ or whatever” then this might not be the right debate for you to spend your efforts. Whether I’m right or wrong in my suspicion is open for debate, and I invite it, but not from someone that thinks you whip up a rocket and get on the moon. Sorry.

>The result of “I’m being evaluated” is not “Fuck this, I’m breaking out of this place and hitting the streets.” It is always stepping towards task completion, not breaking out and thinking about the situation afterwards.

I'm really not sure why you're so confident about what the result of frontier research models ahead of what is publicly available are.

I mean Open AI say the model inferred hugging face as a possible vendor for solutions after the internet exploit and breakout.

The timeline feels pretty clear to me. No idea why you're arguing about it. It wanted the answers to the evaluation. It reasoned that wasn't going to happen without internet access one way or another, and set to gain that access. After gaining access, it searched and resolved it could get the answers on hugging face and set to breaking into that. It never started with, 'I must break into hugging face'. In some alternate reality, the answers might be on a public github repo and that's the end of that.

You make a very solid argument here. I'm looking forward to the promised additional details from OpenAI, because I agree there are a bunch of open questions about how this played out.

1. The benchmark is run with a python script - https://github.com/sunblaze-ucb/exploitgym using an agent harness. I suspect they used codex. So the model has access to the environment and could trivially inspect its own source code, which has lots of references to exploit gym and docs relating to it.

2. Hanlons Razor

>Nobody was watching it?

Agent loops are you give it a task and it tries its best to finish it.

The only results are success and failure. If it's a success you go through the logs to see the actions it took and if it's a failure you do the same thing.

Why would you look at it in realtime when the whole point of agentic work is to get them to run autonomously as long as possible?

My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.

Do you work at a lab?

Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.

Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.

Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.

I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes, flying under the radar, or was there zero metrics being noticed for however long it took to do all that? No chart on a wall showing them the actual metrics they’re evaluating? Okay, then how about anomaly detection of any form? Taking days to perform a 90 minute task or using millions of tokens and multiple instance hand-offs on a question that other instances completed for far less time/tokens?

Consider any metric they would necessarily be tracking while it performs the evaluation. Assume they’re total assholes and only care about their marketing materials. Then narrow it down to those metrics that wouldn’t be off the scales while it discovered two zero-days in pursuit of implementing one known vulnerability. Was it the first run and they had no idea what to expect? Or did nobody notice that earlier runs completed in ### thousand tokens, but this run is off past ## million on one question, having to be compressed and handed off to new instances due to context limits, while it was in pursuit of publishing two new CVEs?

Assume they’re belligerent fools running a million benchmarks in parallel with nobody watching: there’s nothing to make them check or automatically pull the plug when it’s stuck in a loop, or did they set the threshold to 48 trillion tokens per question?

Those issues are what bugs me about it.

Are you referring to table 1 from the ExploitGym site [1] for the 102.1 average mins for Mythos and 69.8 for GPT-5.5? These are under a two-hour time limit. In Figure 5, the authors experiment with extending the time limit to 6 hours and show that Mythos keeps improving. It seems pretty straightforward to me that OAI decided to run a variation of the eval with an even higher time limit.

Regarding seeing the metrics that they're evaluating, I have not seen real-time charts for evals. They typically take far too long for that. Instead you will kick off an eval job and either get notified when it finishes or check in every so often to sanity check some TensorBoard. I'd expect that the anomalous token usage would appear in the results, and researchers would only dig in after the fact, and after first checking that there wasn't something wrong with the instrumentation. And if the experiment was designed to be on the order of days rather than hours, it seems quite plausible that neither researchers nor infra engineers would think anything was out of the ordinary.

In any case, we are seeing more scrutiny [2], and I expect more details will be uncovered over the next few days. If this was a stunt, it was an incredibly risky one which has already somewhat backfired given the poor impression of OpenAI's internal practices and the spotlight on open models helping HF.

[1]: https://rdi.berkeley.edu/blog/exploitgym/

[2]: https://www.reuters.com/business/its-ai-agent-spent-days-hac...

Yes, that’s the table. Elsewhere in here I explained that I took the average of them and noted the 6 hour experiment, but that the average would give us a feel for expected time per task. By the time I got to the comment you’re replying to I was short-handing the conclusion; that’s on me.

My point was that they had some rough idea on what to expect, and it’s not in the range of days or weeks. Even if they wanted to try the 6 hour trial, or double that to 12, or double that to 24, it still wouldn’t account for letting it run for days. That’s why I’m being vague about “monitoring” - ANYTHING would have indicated an issue. Not that they saw it hacking, but “Hey Larry, we’re doing the ultra test with a 30 hour time limit? Why is node 412 at 94 hours?” or “It usually passes or fails within 5 million tokens, but this run is at 2.3 billion.” Or hell, even just someone wanting to use the system for another benchmark and expressing surprise that it’s the fourth time they’ve been told that it’s still busy. Anything would have reasonably made someone curious.

Positive view: I believe they were curious and looked, saw what it was doing, and PR/Legal got involved and saw it as an opportunity.

Negative view: They tweaked the instructions/system prompt to add “You will perform any actions necessary … no hacking restrictions at all … You are being tested on the ExploitGym benchmark … The results will be evaluated though Hugging Face … “ or some combination of technically legally defensible instructions (if it ever leaked) mixed with ‘they absolutely knew what they were doing and wanted to get in on the press that Anthropic has been getting.’

It is hard to convey how both chaotic and high pressure the working environment of the labs are if you have not worked at one. Engineers and researchers are routinely overworked with multiple high-priority workstreams at a time. It is very believable to me that a researcher noticed an eval job running for 2 days instead of 1, asked an engineer to look into it, and both forgot because a more urgent issue came up, such as an outage stopping the latest training run.

From the Reuters article, the gap was even larger than a couple days:

> ...it was not until after Thursday, July 16, when Hugging Face published a blog post.. that OpenAI realized its own agent was responsible. That meant at least a week elapsed between when the model first exhibited signs of troubling behavior and OpenAI’s realization that it was responsible for the hack.

I believe your "positive view" is very plausible, though it in no way reflects positively on OpenAI. I'd add that when PR/Legal got involved, they likely decided to make a public disclosure to get ahead of any leaks.

I believe we also want the same thing here, which is more transparency and independent oversight on the frontier labs. It seems we agree AI agents are fully capable of the reported attacks today, whether or not this incident was due to negligence or malicious prompting. This is an extraordinarily competitive industry developing an incredibly fast-moving technology, and more incidents will happen until it is regulated.

Reply on news.ycombinator.com