Give me your best thoughts on this most recent call for AI slowdown.
That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.
So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.
Beyond that, we already know AI systems have caused harm in various contexts where they have been used, like facial recognition getting people arrested without justification, loan applications being denied wrongly, jail sentencing being mishandled, and so on. All of these things have actually happened in the real world already. So it doesn't take much imagination to extrapolate a little and see the potential for greater degrees of harm. And consider the standard we're comparing against here: "probability greater than zero". That's not a very high bar, so I feel pretty well justified in holding this position.
On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.
So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.
- They have internal models that still show there's gas in the tank, in terms of improvements
- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.
- The true demand for high-intelligence tasks is lower than what they need to justify their spending
- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again
- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture
I'm personally skeptical of LLMs getting there without at least one new invention. But it's also empirically clear that the pace continues to increase, even if we ignore that exponential growth is the default in economics in general.
1) That level of demand is likely true, but that doesn't mean that the total spending demand is to match to that level.
2) Most of said office work can be automated via normal software, and need not use LLMs. If they were to be automated, the amount of work left available for Anthropic to routinely work on would be (lower than before the automation took place).
- destabilize the world just by what they do: making previously very hard things (generating fake everything, manipulating people, hacking, bio-weapons research) easy enough fast enough that our civilization doesn't cope with it and/or
- accelerate AI research to the point where they make the dangerous ASI thing appear years earlier than would otherwise have happened
Based on our past inability to handle things well, it could be extremely devastating to a large population.
I would like to think we will navigate into a new way of life to make up for the job losses and other things, but the impacts are just too devastating and we’re just getting started.
I think that the genAI companies themselves present a greater danger to us (arguably not an existential danger, though) than genAI itself.
genAI is a tool without any intention. It is amoral.
I'm not saying that genAI is harmless and the dangers can be ignored. I'm just saying that the dangers of the tool don't rise to the level of human extinction. That notion is something that genAI companies have been espousing from the very start, and it's part of their whole approach of manipulating people to achieve their goals.
For the record, I actually think that genAI (at least in the form it exists in now) is likely to cause more harm than good.
As for the technology itself - it isn't like anything we've ever seen before and, when embedded in human societies, it won't behave in a way that has any comparable precedent. I worry that basically nobody of good will who has any political power understands it.
And no, its not a marketing ploy.
If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.
https://www.pymnts.com/healthcare/2026/modernas-ai-built-can...
https://news.unchealthcare.org/2026/09/unc-lineberger-secure...
If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.
https://www.anthropic.com/threat-intelligence-report-septemb...
> Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous—or create entirely new ones. Without the correct safeguards, such capabilities could have catastrophic consequences.
Results from evaluations of older models (for example Claude Opus 4 and Claude Sonnet 4.5, from 2025) clearly showed that these models were well below the threshold where they could meaningfully assist a sophisticated user in carrying out dangerous biological research. As a result, safeguards on these models were less stringent, directed mostly at preventing access to content that might uplift novices in recreating known bioweapons. But for today’s models—which are capable of assisting in a range of complex scientific research tasks—the evidence is no longer certain, and we cannot make that same assurance. For this reason, and out of an abundance of caution, we have launched recent models (most notably Claude Fable 5) with stronger safeguards that restrict access to a wide range of dual-use biological research queries.
...
Here, we present five case studies of actors using our models in ways that could support biological weapons development. These examples are illustrative of the kinds of tasks to which our models are put, and the often-difficult judgements we have to make when assessing whether or not a given biological use is dangerous. They also convey that we encounter what would otherwise be non-public insight into the risks associated with biological misuse from AI
This is not AI waking up and deciding to merc us, which is what normies think we're talking about.
This isn't really a fear of technology, it's a fear of one person being able to harness an entire country of geniuses towards some bad end.
Maybe in your imagined future.
AI will definitely be used to kill people (already used in targeting). But the doom / panic is about an existential threat to our species. It's sci-fi and unwarranted for LLMs.
It’s a classical regulatory capture attempt by the major players, rather than a marketing ploy. They use safety as an excuse.
AI agents can generate and run arbitrary code. They are very capable hackers and there is virtually no way to really control what they do. They increasingly become black box systems, both to their creators and users. They know everything about everything. They gather lots of private data. And they are able to manipulate people. They have too many evil use cases.
Listen to real AI pioneers like Hinton et al. They tell you the truth.
This prediction has proven quite accurate so far.
Don't think anyone should panic, but people should be aware.
If it happens that AI is able to replace say 25% of the office workforce (including programmers of course). The result could be catastrophic, if governments do not step in and do something. It could definitely lead to a depression and another world war, given that we already have some regional wars in the Middle East and Eastern Europe.
You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.
It's weights. Spun up billions of times over.
If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.
But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?
Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.
Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.
Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).
The venture capitalist world is full of people who will deliberately ruin a company based on the extremely unlikely but possible scenario where the companies goes to the moon in value. I think there's pretty much 100% certainty that multiple venture capitalists are going to try exactly what you're suggesting on the off chance that it makes them the new world controller.
The fact is that AI is incapable of living up to the hype and AI companies can't keep promising that any day now it's going to cure cancer and become self aware and be as smart as humans and let employers fire most of their staff. They know that isn't going to happen. They want to be able to blame the Slowdown for AIs continued failure to deliver while also claiming the Slowdown is needed because of how successful AI is.
That said, AI isn't harmless. It could be harmful to the internet since the flood of bots and slop could drown out the voices of real people. AI will be used and misused by companies and governments to screw over people and hide discrimination. People's air and water will become increasingly polluted while their already difficult to afford energy prices will go up. AI will enable new levels of surveillance and control at a scale that was never possible before and with a lot of false positives and negatives.
We don't need an "AI slowdown" so much as we need to rethink what we're really gaining in exchange for what it's costing us and what steps need to be taken to protect the public and make sure they get prioritized over private profits.
- work became horrible
- internet content became horrible
- even restaurant menus became horrible
Where do we go from here?
Sure AI sucks, but not to that degree (yet)
What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.
The damage AI is causing to society is real and happening now. Potential doomsday scenarios are a distraction from the real harm AI companies are doing at this very moment. They have amplified nearly every asymmetry and turning into mulch whatever was left in trust we had in society. Disinformation is now more convincing than ever. Computer security threatens to become a fiasco, only available to those who can pay the AI mob (the very same people who pioneered the tools attackers use). The entire intellectual property of humanity has been plagiarized without compensation to build these things. Every day more and more is asked of workers for the same compensation. Academic work is sniped at best and outright stolen at worst. Education has become a disaster with students unable to pass the same competency walls as students even just five years ago. Not to mention the environmental damage.
I'm not saying that the technology isn't impressive. I'm just saying its suspiciously convenient that we're suddenly focussed on some potential future big bad done by a vague uncontrolled AI when the harm is here now and under the purview of humans.
For a while the hottest (stupid but still the only hot one) topic was crypto. We had this for quite a while. I believe this was because we hit walls left and right on how to progress further.
Then GPT-3 came out and made it very clear for a lot of capital and actors were it is going.
Machine Learning and types of it like LLMs need massive compute and massive amount of data.
So suddenly we discovered a way forward for optimizing and automating things no one thought would be possible: Software development and computer use and basically everhting digital. In parallel this Machine Learning push became relevant for robotics too. But LLMs delivered something fundamental to robotics: Language and a tremendesly good interface for humans.
Now we have a massive amount of capital in form of VC who wants to disrupt the next market. China is doing this with manufacturing, so what is left? Everything which was not touchable.
Now it is.
And it doesn't just allow for alot of automatistaion, it hits us were it hurts: Software bugs, security, fraud, scams etc.
It isn't just automatisation and getting more money, its also a weapon (CVEs, Statesecrets, propaganda, etc.)
So we don't have just a handful of VCs investing in this, we have the richest countries on the planet investing in this. We also have the fastest moving companies investing in this. The best connected ones, with a massive amount of very smart people.
Google has massive compute, its own network around the globe and massivly overpayed smart people. Microsoft, Amazon and even Meta has crazy amount of money too.
Then we have state actors like USA vs. China vs. hust Europe.
I also have never seen any technology progressing that fast. Image generation was unthinkable, now we can generate whatever we want. Video? Can now be done in real time. Text? No brainer.
GPT Astra can use Blender very very well and that only after 6 years of progress.
These 'stupid' LLMs reduced already the price of making Websites, basic 3D models, basic research etc. basically free in comparision.
So now we ned to find back to your question about eliminiating humanity: We have seen unrest in the last few years for a lot of basic human reasons. We as a plent are rich enough to take care of everyone on our planet. we do not do that thats why unrest can happen very fast.
In the USA alone A LOT of people have weapons. Around the globe people will unrest due to fuel shortage, disruption of supply chains etc.
We have state actors like Isreal who tried to attack Iran with Stuxnet, we have Russia which manipulated the elections of Europe and USA. We have high inflation, oil crisis.
And now we have a very fast moving and progressing technology which will add fuel to it and might eliminate quite a lot of white collar jobs. This will lead to more pressure on blue collar jobs.
We as a society are very bad in acting fast and the industrial revolution was, at that time, very bad for a lot of people.
There is only one russia necessary running GPU clusters of uncensored LLM model to start pushing boundaries left and right. One Isreal doing something fucked up or one Donald Trump.
Software is very shitty in a lot of places and the most complex thing we humans manage. A lot of shit wasn't found or solved because of the time and effort and people needed to find it.
Now the last Ubuntu LTS had over 900 CVEs, my own systems were attackable 3x this year already.
I don't want to say it will happen, but I do think we should be very careful of how we progress and for a lot of people the next 5-30 years will be very very disruptive. It is already disruptive to 'others' we might not count but let it be clear: all of these people who lost their jobs to AI already, it counted for them.