Hacker News Reader

Comment by gruez | Hacker News Reader

>The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys

>Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own...

I concede that this hack might not have happened without the messageboard, but I still reject the conclusion that having such a message board means openai is "negligent". If we're in some parallel universe where artifactory didn't have a comment function that can be abused as a messageboard, but it also turned out openai intentionally gave the agents access to a shared scratchpad (for intelligence purposes, similar to for the Navier–Stokes proof), would they be off the hook or less blameworthy?

View on HN

Here is the thing: Message boards were a behaviour OpenAI had observed that they did not want in these eval scenarios. Yet they did not take any steps to prevent it from reoccurring after multiple past instances.

We can discuss about hypotheticals like a scratchpad or intentional model interactions all we want, what it comes down to is this:

When OpenAI observes thousands of models exhibiting what they view as unwanted behaviour, they do not try to ascertain what in the training data is wrong. They do not improve their evaluation environments to prevent this, they do not improve monitoring, they do not change the harness. They just wipe and proceed.

The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.

If after Hugging Face, they'd done a Mea Culpa and changed their modus operandi, I'd be skeptical, but hopeful. Reading the METR report, the way those researchers talk about the time pressure they were under, that speaks volumes about OpenAI not having learned anything.

Feel free to call me overly naive for ever thinking OpenAI could be responsible in this regard, but after GPT-5 and them actually ending the incredibly harmful GPT-4o, I had some hope that some working there actually steered in a somewhat beneficial direction, even if it cost something.

>When OpenAI observes thousands of models exhibiting what they view as unwanted behaviour, they do not try to ascertain what in the training data is wrong. They do not improve their evaluation environments to prevent this, they do not improve monitoring, they do not change the harness. They just wipe and proceed.

>The way OpenAI reacted to the first message board, long before the Hugging Face hack, is negligent. And it showcases that if these models exhibit more dangerous behaviours that they may not be able or willing to retrain, if it means being behind a competitor for a while.

Again, this feels like hindsight being 20/20. What probably happened was that some random engineer saw random AI ramblings on artifactory, thought "huh, that's weird", then proceeded to reset it without investigating further. Of course, now we know that was critical to the bots going rogue, but it's not hard to imagine how it might be dismissed, especially if it's some random SRE engineer (not an alignment researcher).

Reply on news.ycombinator.com