Personally, I think we should take the doom narratives very seriously, and act on them, but make sure those actions are not in the interests of Anthropic and OpenAI investors.
The problem with that hope and approach is regulatory capture. Big money is now the only thing that apparently matters in shaping policy, and changing that will require a lot of very sweeping vote and policy changes that may not ever have the momentum to succeed.
I guess not literally, due to the dates? But they have substantive technical details.
https://garymarcus.substack.com/p/the-ai-2027-scenario-how-r...
But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?
None of this makes any logical sense whatsoever.
Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.
However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.
It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.
https://en.wikipedia.org/wiki/Maximum_Overdrive
And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.
To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.
Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.
And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.
Or, we could avoid all that by making sure AI labs are properly regulated.
But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.
Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.
It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.
And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.
But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.
[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).
Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.
I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.
But upfront, it's a detailed path with specific breakthroughs and a concrete plan to achieve or it's just more fanfic from doomer fanbois. Gary Marcus wrote a fantastic critique of AI 2027. Start there.
I work with Codex and Claude daily. They're fantastic for script coding, config issues, and simple projects. They lose the plot on bigger things to this day. Just last night, Astra decided to cheat to present the illusion of progress until I called its BS. And I was so looking forward to finally working with the AGI. Humans remain the muse, the common sense, and the manager of making these things productive. And as much as I agree with Carmack's suggestion to not become the out of touch Kung Fu Master:
https://x.com/ID_AA_Carmack/status/2098443262214230095
working with them on projects daily gives me a pretty tactical read on what they can and cannot do IMO.
That's how I reconcile it.
But let's say it had. That's just 1 of 20 events that have to happen in sequence to hit the jackpot here.
Just like AlphaFold 2 genuinely increased the weak sequence but strong structural homology detection of proteins, but it did far less for out of sample domains and it provides no intel into the kinetics of folding. Details matter. And nobody likes to hear about them.
I see similar conceptual gaps in coding agents when I work with them, even Astra and Fable, so maybe this a is decent proxy to describe the gap between hype and reality.
I thought the Bengio blog post posted here the other day answered that question quite well (ignoring your timeline).
The argument is that in training for the capability to achieve certain goals, you may inadvertently also train for secondary instrumental goals you did not intend, such as aggressive behavior or deception.
I'm actually no sure I buy that this is particularly likely, or necessary, but I found that take reasonable and worth considering.
*TBF before we dismantled America's clinical trials infrastructure, the monkeys were making insane progress here: https://www.nytimes.com/2026/09/04/opinion/clinical-trials-d...
Why? Those aren't the things I'm actually worried about, and those aren't the only "doomer" ideas, just the most extreme ones (thus easy to straw-man).
My main worries are about economic disruption, social damage due to the pace of technological change outpacing human ability to manage it, further concentration of power into small elites (along existing lines), and the reallocation of labor towards less personally-fulfilling pursuits.
I also totally buy into the idea that "AI leaders" are playing up certain risks as a marketing strategy. After all these are smart tech people, and those types have a tendency to get really arrogant and be irresponsible.
So the main thing is to slow all this down, so there's a chance of getting it under control, and snatch the payday away from such greedy and irresponsible people.
Tangent: what happened to these staffless supermarkets of 10 years ago? I have never seen them again after .. covid.. weird!
Did you see what it did to people in your neighborhood?
That’s the path.
But today, we have ecoflows and solar panels. Back then, we didn't.
That to me is the greater threat than annihilation due to autonomous weapons, etc.
What might not affect you, might affect millions across the ocean.
A society no longer creating bullshit jobs because there is no reason for it anymore. You have some AI.
Right now a whole industry is transforming (again) but now filling the automation gap we were unable to do because of a shortage of labor or expertise.
Our society already has peak employmet reached, now AI takes away all the random digital jobs too. Someone building a website? Takes 5 minutes. Someone doing odd jobs on fiverr with blender or any other tool? Takes 5 minutes now.
Regression can happen very fast. The Romens were a high level culture and an empire. Gone.
Today you have more people on the planet then ever, everyone who might want to eat and who needs to work something for it.
AI already replaced people, already makes a lot of tasks A LOT cheaper and faster and human independent.
We never had a proper discussion as a society. Most countries have a social system but look at countries like the united states of america: if you are poor, you are fucked.
And AI is not the only thing which is now peaking: Physical AI / Robots are progressing fast, its clear today already that in 50 years these robots will be better than humans (50 years is very conservative).
If we have 10 more golden years due to restructuring and rebuilding things and preparing everything for agents and robots etc. what happens after these years?
If we have 10 or 20 more of these years, what do you think will happen to all the new kids (yours perhaps too?) and jobs and everything?
Riots? Famine? Mass migration? Mass death? Slums?
And don't respond with "enough people will survive this" bullshit. No one is saying that earth as a planet is not surviving. Its the same thing with climate change...
But the end of scarcity and disease is a fantastic thing if it happens. If our lizard brains do not bridge that transition and instead attempt to maintain the billionaire status quo, I agree, we're going to go through some things once people start starving, two meals from a revolution and all that. But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars. Bread and Circuses kept Rome alive for 500 years if you need an example.
And Zuckerberg bought so much land and bunkers, he is prepared for shit hitting the fan. After all he doesn't care what his platform did to humanity anyway.
No 'we' didn't. A whole shitload of much smaller isolated communities did. The world added 7+ billion more people since then. Imagine chlorine doesn't show up tomorrow? How many people die in two weeks? How far do you travel when no gas comes out of the pumps. How much will you stand out at night if you're the only ones with power? How fast do working antibiotics run out?
The billionaires you're talking about are mostly ignorant fucking idiots with a god complex. The moment any complex system starts falling apart they are saying as dumb of shit as everyone else (during covid for example).
Most of the pasts anti-fragility is gone. We survive by means of very complex systems and if you lose too many key points in those systems they can totally collapse.
1B+ individuals worldwide currently live in squalor. They will barely notice. The real crime is that in the midst of the west's carbon footprint buffet, we won't bother to uplift their lifestyles with the most basic conveniences. And while I am no fan of the billionaires, they absolutely have public and private personas or their business ventures would implode. They absolutely have god complexes, but don't mistake that for genuine idiocy.
Lack of technology alone will not wipe out the entirety of humanity, I agree. But we're not talking about just a lack of something. We're rapidly sprinting to give the fewest people the most power over the planet ever. I think it's likely to happen in the 21st century. What happens when the worst case scenario isn't just being left with nothing, but facing an entity that is so rich and powerful that it can use active measures to kill as many as possible? This is more power than any warlord or emperor could've dreamed of in the past.
> But as much as I think people like Elon Musk and Mark Zuckerberg are unhinged lunatics, I also think they're smart enough to keep people fed and entertained for pennies on their accumulating dollars.
So far, every time the billionaire class has been asked about how much money should go to them in society, their answer has always been a resounding "ALL OF IT". You don't get to that level of wealth while retaining a state of mind that allows you to toss a penny to the beggar. Reality is what's holding them back right now, but what would happen if their dreams of total automation eventually come true? How long from then until one of them snaps and decides to use state of the art automated military technology and weapons of mass destruction to wage war against the peasants (armed with sticks and stones) to solve the 'problem' once and for all?
To be clear, I'm not saying any of this is imminent or remotely likely to happen soon. But our society is moving in a direction where something like this could eventually be possible. Giving a human truly absolute power over the entire world has never been tried before.
So with that, we can rule out extinction at the hands of the machines 100% leaving only machines somehow accessories to near ELEs humans could trigger today and could have triggered for decades. And at that point, Scary Potter and his friends need to STFU.
So scaling down to an AI takeover in the next decade, paradoxically, the idiotic pronouncements by Sam Altman and Dario Amodei that AI is coming for your jobs and that they are afraid of their own creations is not the sort of thing that leads to an AI takeover, it leads to regulations and bans. What would would make such an AI takeover viable in the next decade would be a magnificent seemingly benevolent AGI that raised living standards and spread happiness as it enchanted and delighted its way to everyone happily ceding control to the greatest thing ever. Does that remotely sound like the near future to you?
In the words of Ted Chiang "most fears about AI are best understood as fears about capitalism" and that aligns with what's going on right now. So let's skip the intermediary surrogate and attack the worst bits of late stage capitalism, mmkay?
> So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?
Your comment has a false premise. If "the end of the world can be stopped," there's no way that the only way would be one that "helps Anthropic/OpenAI['s investors]." That can be shown by a counterexample: nationalize them and leave the investors holding the bag, then take whatever action you're talking about.
And unfortunately my tautology is actually closer to being reality than alternatives. Stopping progress even in Chinese models (a deal with Xi for example) but that specifically helps OpenAI/Anthropic.
There are very few scenarios that would not help these two companies by adding regulation. Your counterexample is 100% less likely because it's not legal for the US to nationalize them outright without taking other action first.