Hacker News Reader

Comment by reasonableklout | Hacker News Reader

Do you work at a lab?

Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.

Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.

Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.

View on HN

I’m not holding them to a standard of “being careful”, I’m assuming they’re interested in the metrics they’re evaluating. I mentioned in another comment that each task in the benchmark takes ~90 minutes, depending on the model being tested. OpenAI says to answer one question it found two zero-days and performed a series of privilege escalations and moved across their network before hacking HF. Was that in 90 minutes, flying under the radar, or was there zero metrics being noticed for however long it took to do all that? No chart on a wall showing them the actual metrics they’re evaluating? Okay, then how about anomaly detection of any form? Taking days to perform a 90 minute task or using millions of tokens and multiple instance hand-offs on a question that other instances completed for far less time/tokens?

Consider any metric they would necessarily be tracking while it performs the evaluation. Assume they’re total assholes and only care about their marketing materials. Then narrow it down to those metrics that wouldn’t be off the scales while it discovered two zero-days in pursuit of implementing one known vulnerability. Was it the first run and they had no idea what to expect? Or did nobody notice that earlier runs completed in ### thousand tokens, but this run is off past ## million on one question, having to be compressed and handed off to new instances due to context limits, while it was in pursuit of publishing two new CVEs?

Assume they’re belligerent fools running a million benchmarks in parallel with nobody watching: there’s nothing to make them check or automatically pull the plug when it’s stuck in a loop, or did they set the threshold to 48 trillion tokens per question?

Those issues are what bugs me about it.

Are you referring to table 1 from the ExploitGym site [1] for the 102.1 average mins for Mythos and 69.8 for GPT-5.5? These are under a two-hour time limit. In Figure 5, the authors experiment with extending the time limit to 6 hours and show that Mythos keeps improving. It seems pretty straightforward to me that OAI decided to run a variation of the eval with an even higher time limit.

Regarding seeing the metrics that they're evaluating, I have not seen real-time charts for evals. They typically take far too long for that. Instead you will kick off an eval job and either get notified when it finishes or check in every so often to sanity check some TensorBoard. I'd expect that the anomalous token usage would appear in the results, and researchers would only dig in after the fact, and after first checking that there wasn't something wrong with the instrumentation. And if the experiment was designed to be on the order of days rather than hours, it seems quite plausible that neither researchers nor infra engineers would think anything was out of the ordinary.

In any case, we are seeing more scrutiny [2], and I expect more details will be uncovered over the next few days. If this was a stunt, it was an incredibly risky one which has already somewhat backfired given the poor impression of OpenAI's internal practices and the spotlight on open models helping HF.

[1]: https://rdi.berkeley.edu/blog/exploitgym/

[2]: https://www.reuters.com/business/its-ai-agent-spent-days-hac...

Yes, that’s the table. Elsewhere in here I explained that I took the average of them and noted the 6 hour experiment, but that the average would give us a feel for expected time per task. By the time I got to the comment you’re replying to I was short-handing the conclusion; that’s on me.

My point was that they had some rough idea on what to expect, and it’s not in the range of days or weeks. Even if they wanted to try the 6 hour trial, or double that to 12, or double that to 24, it still wouldn’t account for letting it run for days. That’s why I’m being vague about “monitoring” - ANYTHING would have indicated an issue. Not that they saw it hacking, but “Hey Larry, we’re doing the ultra test with a 30 hour time limit? Why is node 412 at 94 hours?” or “It usually passes or fails within 5 million tokens, but this run is at 2.3 billion.” Or hell, even just someone wanting to use the system for another benchmark and expressing surprise that it’s the fourth time they’ve been told that it’s still busy. Anything would have reasonably made someone curious.

Positive view: I believe they were curious and looked, saw what it was doing, and PR/Legal got involved and saw it as an opportunity.

Negative view: They tweaked the instructions/system prompt to add “You will perform any actions necessary … no hacking restrictions at all … You are being tested on the ExploitGym benchmark … The results will be evaluated though Hugging Face … “ or some combination of technically legally defensible instructions (if it ever leaked) mixed with ‘they absolutely knew what they were doing and wanted to get in on the press that Anthropic has been getting.’

It is hard to convey how both chaotic and high pressure the working environment of the labs are if you have not worked at one. Engineers and researchers are routinely overworked with multiple high-priority workstreams at a time. It is very believable to me that a researcher noticed an eval job running for 2 days instead of 1, asked an engineer to look into it, and both forgot because a more urgent issue came up, such as an outage stopping the latest training run.

From the Reuters article, the gap was even larger than a couple days:

> ...it was not until after Thursday, July 16, when Hugging Face published a blog post.. that OpenAI realized its own agent was responsible. That meant at least a week elapsed between when the model first exhibited signs of troubling behavior and OpenAI’s realization that it was responsible for the hack.

I believe your "positive view" is very plausible, though it in no way reflects positively on OpenAI. I'd add that when PR/Legal got involved, they likely decided to make a public disclosure to get ahead of any leaks.

I believe we also want the same thing here, which is more transparency and independent oversight on the frontier labs. It seems we agree AI agents are fully capable of the reported attacks today, whether or not this incident was due to negligence or malicious prompting. This is an extraordinarily competitive industry developing an incredibly fast-moving technology, and more incidents will happen until it is regulated.

Reply on news.ycombinator.com