OpenAI disbanded the team that assessed catastrophic model risks | Hacker News Reader

OpenAI disbanded the team that assessed catastrophic model risks

thenextweb.com

32 pointsnyku14 comments

The labs have been continually widely criticized for releasing anything that points out possible AI dangers, with critics claiming the labs are only trying to spread fear or alarmism about AI risks, so it should be no surprise for them to the disband a catastrophic model risk team. If oil, nuclear, or cigarette industries were getting started today, I wonder if they’d similarly be accused of spreading fear/alarmism - Exxon claims their industry’s products will be so incredibly popular they’ll literally change the planet’s temperature and causes natural disasters? ‘They just want to scare us that their product is incredibly powerful, plus want regulation to prevent upstarts.’

I often wonder if the alarmism / marketing ploy accusations are fake / astroturfed.

Not even necessarily in the organized sense, just people wanting cool toys, realizing they need to posture like such if they want to get them, and then just proceeding to disregard the consequences.

Certainly what it's been feeling like at least, not that I'd put too many eggs into this basket though. Just an impression.

It feels like there's an air of this amongst the inability to communicate effectively. Too much communication, too early and historically now viewed as FUD doesn't help honest analysis from being dismissed early. Dario seems to be the poster child of this, and so you start to wonder what his actual motivations are.

Beyond this I don't trust much of what any frontier provider is saying when they are claiming doom and gloom of the models and then actively choosing to run those models on data centers illegally polluting. Seems like a very hypocritical approach to the "trust me bro" attitude by these companies.

Finally, these are not scrappy startups. They are well funded (hundreds of billions of USD) enterprise organizations with more than enough resources to pay entire legal, marketing, research teams to do this right. Yet, look at how they message much of what they say out loud.

They are very real and not because we want "cool toys". If OpenAI, Antropic disappeared from earth, I would not mind. They can take Thiel, Karp, Musk and the rest of that class with them, as far as I am concerned. They are the biggest threat to democracy and well being of the rest of us, not ai.

And yes, their argument is that they need to amass as much money as possible right now, to proceed as fast as possible with creation of the doom machine and somehow we should trust them. Meanwhile, as they talk about immanent permanent underclass and 80% unemployment and blame AI for their own negligence ... they dont care one bit about harms related to their data centers, corruption they facilitate in government, surveillance they facilitate.

There was no positive intention in "warning about all artists being obsolete and movies being made by Sora only". It was pure bullshit of someone who dont care about artists, but wants to sell slop machine right now. (or seemingly hate them along with all creators for some reasons)

Like, common. It was never good faith. And if it was, their idea of harm is "AIG will take over the world" which completely ignores actual harm to people, social structures and environments they themselves are cause in the process of trying to create AIG capable to take over the world.

> OpenAI shut down its preparedness team at the end of July, the Financial Times reports. The team existed to work out whether OpenAI’s models posed catastrophic risks. It also designed the ways to contain them.

> The work has not disappeared. Responsibility now sits with senior staff inside existing teams, split by subject.

Seems pretty reasonable. Given everything that happened with Huggingface and similar incidents it’s clear that the team wasn’t doing shit.

Or it came to it's natural conclusion: "Yes, there are catastrophic risks"

But, you know, it's a business, so now it knows it has catastrophic risks. What, are you going to just stop building the torment nexus?

Or maybe they weren't allowed to do anything effective. I don't know why you give OpenAI such a big benefit of the doubt.

Is it "clear" from a car accident fatality that seatbelts and airbags aren't doing shit?

They just had their most important learning experience and you get rid of them? That doesn't sound reasonable to me. It's like firing your SREs after an incident

Read the article. No one is getting fired. They are being embedded within individual teams instead of having a separate org.

My bad, I didn’t read before reacting :)

Brought to you by the for-profit-non-profit established to build proprietary open source models by the philanthropist CEO who is building bunkers.

not a bad assessment in the end though it doesn't entirely read as it ends. basically: openAI is growing up moving towards IPO and cancelling stuff they were tring out with and things that don't have an obvious benefit for enterprise customers.

on the flipside i do think if there is an entire team to assess critical risk, that the model during testing hacking some company is either: 1) really terrible assessments 2) team was 100% ignored already and basically just for show

most likely its 2, not the fault of the team members, but mismanagement which has now been resolved. Made the team but didn't want to produce the outputs, maybe because people were fearmongering and they hired a few poor souls to counter-market that they were doing good on the safety front.

A bit cruel to hire people for such things but i suppose its the AI-flavor of a diversity hire -_-

Another fata point for all those who sided with OpenAI and Anthrophics claims that they are somehow more responsible and careful and that others, especially those pursuing pesky open weights models are a danger.

Reply on news.ycombinator.com