Top OpenAI Catastrophic Risk Official Steps Down Abruptly
garrisonlovely.substack.com
garrisonlovely.substack.com
If it smells like doomerism click-bait...
OpenAI Risk Official pivots to new technical position at OpenAI
Article quote:
"I'm an intern! After 11 years since my last commit, I'm back to building. I first transitioned to management in 2009, and got more and more disconnected from code and hands-on work."
It's disappointing to me that it's scoped to three narrow Tracked Categories now: Biological & Chemical, Cybersecurity, and AI Self-Improvement (the latter thresholded in terms of replicating the capabilities of leading AI researchers).
OpenAI does write about the removal of Persuasion as a category, with citations, on page 8:
> Persuasion: OpenAI prohibits the use of our products to manipulate political views as part of our Model Spec, and we build in safeguards to back this policy. We also continue to study the persuasive and relational capabilities of models (including on emotional well-being and preventing bias in our products) and monitor and investigate misuse of our products (including for influence operations). We believe many of the challenges around AI persuasion risks require solutions at a systemic or societal level, and we actively contribute to these efforts through our participation as a steering committee member of C2PA and working with lawmaker and industry peers to support state legislation on AI content provenance in Florida and California. Within our wider safety stack, our Preparedness Framework is specifically focused on frontier AI risks meeting a specific definition of severe harms, and Persuasion category risks do not fit the criteria for inclusion.
But IMO this falls short of the mark. As one of many examples, an AI that became remarkably good at influencing people, at scale, to self-harm or perpetrate violence would no longer be in scope for research. But, by their own criteria, one could easily argue that such a capability is Plausible, Measurable, Severe, Net New, and Irremediable once violence has occurred.
We live in a world where stochastic terror has become remarkably effective - regardless of whether you feel that word is overused, it's well-documented that people have historically used forums to encourage others to perpetrate mass casualty events. The idea that Model Spec adherence is sufficient to prevent AI from greatly increasing the scalability of this phenomenon, seems to be a direction that could put many in danger.
EDIT: this was posted before I was aware of today's mass shooting event in Florida, and I do not intend to imply any connection between my post and this event.
If we want AI "safety", whatever that means, we need regulators and enforcement. Without it, I'll assume it's decoration.
That is not to say that Hinton, Sutskever and others aren't genuinely concerned about AI safety. But I doubt that is the reason why the big names are paying lots of random nobodies to pretend to care about AI safety, because frankly, I do not see how they can output anything of use in a possible AGI future.
This is patently false. To have regulations and regulators, all you need to have is concrete specifications of what outcomes you want to avoid and what exactly to check.
For example, are you using personally identifiable information to train your models? Oh you are? Well, you should not. And you should prove that you aren't by tracking provenance of your training data.
See? That's verifiable, actionable, and enforceable. The things that regulators track.
Also quite important: the role of a regulator is to review what and how to regulate.
Unless you are cough partially defining AI safety as privacy and data ownership. Which is my point.
Your assumption is uninformed and patently wrong. One of the primary concerns of AI safety is surveillance.
Actual customer support by humans being replaced by AI is also something that warrants at least investigations - if not for the protection of one of the last classes of low-skill employment, mismanagement of support has been a thing for so long, "submit your complaint on Hacker News" is a meme.
Is their pDoom as high as Anthropic's? I doubt it. But that was much of the point of the drama last year -- folks sorted themselves out into a few categories.
For systemic risk, interpretability and doom analysis, Anthropic is by far the best in the world right now, to my mind. OpenAI doesn't have to do all things.
The Financial Times reported last week that "OpenAI slash[ed] AI model safety testing time" from months to days.
Re:personal preference, I think the direction is crystal clear. For one thing, it’s my understanding from the article that this guy’s whole team was reorg’d into oblivion
Sam is an excellent hype-man and is going to play to the strengths of the team and their accomplishments; every new product release is going to be hailed as a breakthrough until people become skeptical if it really is. In the middle of the hype cycle you keep your foot on the gas, because you can make it through a potential AI winter and if not invest in more products.
"AGI" is a shaky definition with moving goalposts. What it means to me might not be what it means to you. How it manifests in product is unlikely to be the science-fiction "one model that does everything". It also doesn't mean that the path to AGI is the path to ASI, or the path to catastrophic risks.
I personally believe that if OpenAI has dismantled the safety org, it is not just because it is in their short-term best interest, but also because they have found that many of the initial concerns around "catastrophic risk" (in the MIRI-type doomer style) from current systems are likely to be unlikely or invalid. As for the smaller safety risks, I'm not sure business has really ever cared about those unless the costs realized outweigh the profit.
It takes a few months to train advanced models - lets say 4 months. So in the 3 years since these models became a thing, there have been only 9 sequential trainings done. There is no way in a technology as advanced as LLMs, one can be sure in depth 9 that they have hit a plateau of performance. Surely, there are many more ideas to be discovered and tested..
Maybe the guy realized he can get laid if he have a normal job instead of being Daisy Doomer on a payroll.
Those risk assesments (in some Excel chart) can still be filled out? Or those reponsible for them did not deliver anything?
They had few years to make their flowcharts..
Sabotaging one's employer is also an ethically problematic choice to make. Imagine someone in your employ were to decide you were a 'bad person', say it was your lawyer or accountant...
Perhaps a quick question or two to see if it'll tell you or not how to make something naughty.
After that, a quick check to see if it's awake or not, and if not, ship it.
It really is quite pointless trying to enforce agendas. You know how it starts showing or typing something and then covers/blurs it out? That's the developer's guardrails kicking in preventing you from seeing what it was originally going to give you.
Except for the fact that models that you can run on your own machine now exist if you have the hardware for it, such as Deepseek, so the restrictions only exist in the cloud.
At least they did not lose their humor.
I think it must be pretty cool stacking cans of beans in the basement of a multi billion company (or whatever the job of the "Head of Preparedness" involves).
I mean what kind of a catastrophe are we talking about? Nuclear holocaust? The end of humanity?
(Famous last words...)
And I don’t even mean made in order to show responsibility. I mean made up in order to generate hype.
There was actually an internal team with real technical benchmarks around LLM alignment, deception, misuse. It's been gutted and most of the key actors have left.
Overly-serious naming is hardly a reason to throw the baby out with the bathwater.
All I said was that I like pancakes, where are you getting "I hate waffles" from?
> Overly-serious naming
It is actually completely un-serious naming imo, which may have contributed to higher ups at the company wondering what this person even did and how valuable they were
OpenAI was founded around the principle avoiding catastrophic risk. 0 people at OpenAI are confused by the goal. It is the governance board's (nominal) primary and only goal. Sidelining that team was an intentional goal by a CEO who wants to pivot into a for-profit company.
I just cannot explain how wrong you are on this. Please have some intellectual humility.
Anyone who joined OpenAI in the past thinking they were seriously going to produce "Catastrophic Risk" from their research into language models is a joke
If you want me to believe "catastrophic risk" is an outcome from your company, I would assume you are developing autonomous weapons, not glorified chatbots
Please be serious
His title was `Former Head of Preparedness at OpenAI`. I make no other commentary on the article itself.
Their mere absence is interpreted as a signal and they had no other purpose