My Rebuttal to the Anthropic Extinction Event Hype Train
You may have seen a post like this yesterday from an Anthropic researcher named Jacob Coxon who quit the company in protest over fears that AI may be an extinction level event. These are a few of the quotes I saw surrounding it, and I want to share my own takes on them:
1. "The more senior the people, the more scared they are"
This says more about Silicon Valley culture and incentives than anything else. The mythology around AI has taken on a life of its own in the Valley, and the financial and regulatory incentives support it growing.
2. "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."
Any evidence? Any sort of even implied math behind the 10%? I haven't seen any, only vague illusions to ASI.
3. "Coxon left Anthropic, leaving so much money on the table in spite of the upcoming IPO because he's earnest in his beliefs"
First off, earnest doesn't mean correct. And pre-IPO options at Anthropic's current valuation even now aren't exactly a sacrifice, they're still worth an extraordinary amount. I don't believe the martyrdom framing.
4. "Self-improving AI is coming and following the exponential improvement curve"
In my opinion, this is a kernel of truth hidden under a mountain of narrative spin. Yes, AI has undoubtedly gotten much better. However, behind every headline such as the Navier-Stokes solution, there is a much more grounded, nuanced, and controversial story than the headline might suggest.
5. "We are approaching the singularity"
...of OpenAI and Anthropic's IPOs
The really unfortunate part of these extreme narratives is I believe they are actually counterproductive. It obscures the specific, tangible problems that we can solve in extreme language collapsing nuanced positions into two sides. For example, bioweapons, autonomous weapons systems, AI in critical infrastructure, and large-scale cyberattacks are real risks that deserve serious technical and policy attention right now.
When everything is a potential apocalypse, nothing gets fixed.