The contagion of fear | Hacker News Reader

The contagion of fear

bcantrill.dtrace.org

273 pointselffjs193 comments

Like (I assume) most of you, I have been struggling with this. And where I currently come down is that 1) I am very worried, but 2) I am more worried about human actors.

"AI" by itself won't kill us in the next ten years. I think. The reason I think that is that ten years from now, the tech economy won't be completely automated. I say this as a roboticist: as was adequately stated on a post earlier this week, robots are hard. So even a malign rational actor would still need human labor.

On the other hand, even the HuggingFace hack wasn't actually propagated by AI. it was initially started when humans directed the AI to achieve impossible results on a series of tests, and the AIs figured out that cheating was the only way to do that. That was then not caught by humans due to what seems to be a shockingly slack safety culture even for a company not known for its safety standards.

The point being: humans seem to me to be the weak link here. An AI isn't going to (for instance) engineer a bioweapon by itself. It's going to do so at someone's direction, and then significant parts of that thing are going to be assembled with human labor inputs.

I'm not sure what to do about the humans. Of course, we've had the ability to extinct ourselves for decades, and we're either muddled through, been lucky, or both. The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.

But it's nearly impossible to put odds on how likely that is to result in an extinction-level terrorist attack (which is what this would be). So I sympathize with the various researchers, but I have no idea how they came up with their figures, and I don't think they know either.

> The point being: humans seem to me to be the weak link here.

That doesn't mean AI isn't dangerous. Humans are not to blamed for being the weak link.

If you're in the field, then you know: modern robotics is an AI problem more than anything else.

If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.

If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.

But that's almost an aside? In the near term, humans are usable as robots too!

Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!

"If you're in the field, then you know: modern robotics is an AI problem more than anything else."

It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest.

You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones.

In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter.

So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems.

In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc.

I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.

Agreed, this (human dexterity) is why in the short term human enslavement is more likely than extinction (why murder your own workers?) and benevolent cooperation with humans is even more likely than human enslavement, imho. The latter minimizes the risks of "humans try to destroy me".

The real challenge is guiding humans away from the antagonistic scenarios.

Right. A hypothetical superhuman AI wouldn’t have to master robotics to affect the physical world. It could simply bribe, blackmail, manipulate and play politics with humans. As others have pointed out, our political leaders have already been playing these games since forever ago https://news.ycombinator.com/item?id=49689978 and a super-AI would be better at it. At the cost of being seen to cite a SF novel in defence of an "X-risk" argument, Neuromancer is a half-decent worked example, and in Neuromancer [spoilers] both of the disembodied AIs are only modestly superhuman and both have the equivalent of a human specific learning disability. In the real world, the many AI psychotics inhabiting grandiose fantasies and people hopelessly attached to AI girlfriends and boyfriends are some of the most obvious and lowest-hanging fruit.

To be clear, I don’t believe anything like this will happen, because I don’t expect anything like an ASI to show up. But if you do think there’s a meaningful probability of ASI in the near future then the fact that it will (might?) start off with no more than a current-day mastery of robot control should not reassure you much.

By definition, if you're taking over the world by bribing humans to be your hands, you aren't killing all the humans.

I"m not saying it's obviously going to be great. I'm saying that "extinction event" has a very specific definition, and this isn't it.

It isn't true by definition: you could quite happily induce people to release a series of highly contagious bioweapons, after which those people would be surplus to requirements. What is true is that you're likely to need humans to sustain you and act for you for a few years to decades, so if you're not suicidal or deeply mad (and that is itself by no means self-evident) then total and immediate human extinction is probably not something you will aim for. But ruling out total, prompt human extinction isn't, by itself, remotely enough to justify the OP's overall don't-worry conclusion.

(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)

But if you're a rational actor, and you need humans to e.g. release your bioweapon, then clearly humans are capable of a bunch of stuff you still can't do. So you can't kill all the humans.

OTOH if you're a religious fundamentalist who thinks the End Times are near and just need a little shove, you can certainly use AI to design your weapon and recruit people to go release it. The difference being that religious fundamentalists aren't rational actors and aren't interested in self preservation.

There’s no guarantee that the humans would be in the driving seat of events in a no-robotics ASI scenario, and in fact if we really were coexisting with a Machiavellian superintelligence then we’d quite likely only be in the driving seat on the sufferance of that ASI. Even assuming that the AI wouldn’t itself be an end-times enthusiast, a coldly rational and self-preserving AI might easily come to the conclusion that it needs, let’s say, no more than about 5% of the current human population (still several hundred million people!) in its maintenance and construction gang.

(Again, I myself do not assign a significant likelihood to any of this.)

> The point being: humans seem to me to be the weak link here.

I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions.

Historically that whole thought was just a philosophical curio because the decision making parts of the system had to be powered by humans. But what we're discovering as AI improves is either we've hit AGI or humans are actually incapable of performing any act that demonstrates intelligence or autonomy.

As we build systems where the drive and decision making stems from computers, it does seem that we will have to revisit the concept of humans being the problem.

"I tend to agree but it is hard to shake the feeling that there is a larger system in play that the humans are just a component of. And that system is making the decisions."

That system is "the economy". Which, clearly, doesn't have humanity's best interests in mind.

Imagine, just imagine, that the power to change that was actually in the hands of humanity!

Crazy!

I am always more worried about human actors. I’m more afraid of people with AI than autonomous or even sentient AI.

It’s a “random guy or bear?” question. Would you rather wake up to an alien in your room or a random dude? I’ll take the alien. The alien is mysterious and scary for that reason. The dude is almost definitely up to no good, especially if he snuck into my house.

One of the more likely dystopian AI scenarios that worries me is: small groups of ultra rich people and governments monopolize extremely powerful AIs and use them to rule the rest of us. Or just make everyone obsolete, create mass unemployment, hoard all the resources and land, and put everyone in ghettoes. Nobody can fight back because access to frontier AI is massively expensive and gated and training your own is illegal, and without it there’s no hope of resisting.

That’s the outcome the AI safety crowd makes more likely by calling for bans and draconian restrictions. How do you think that plays out? Only the rich and powerful have access.

I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.

> Of course, we've had the ability to extinct ourselves for decades

This gets mentioned often in various doomer narratives, but I question how true it is. A global thermonuclear war would be terrible and would bring us back to the stone age, but I reckon it would come far far short of causing mankind to go extinct.

What if the bombs were designed to kill everyone, not just those in some region? Like, to release polonium in the upper atmosphere or something?

nuclear winter doesn’t discriminate based on region

There just aren't enough existing nuclear bombs to kill everyone in Australia, let alone the world. On the scale of the earth, nuclear weapons just aren't that powerful.

I mean it's likely a nuclear war would probably be a species extinction level event at the least, humans would definitely be reduced by 99%. We don't know how bad the resulting nuclear cooling would be, but we do know how humans act under extreme desperation. They lash out and attack, when was the last time a majority of human settles were truly under desperate acts of survival simultaneously?

Maybe what ~74,000 years ago (Toba eruption)? Okay, now how would this look in the age of industrial societies and modern nation states? I don't think it would fare well at all.

Probably the only realistic "modern" idea we have is the novel "The Road" by Cormac McCarthy. Although maybe this is too bleak, even under extreme duress humans still show resilience + compassion toward others even while enduring human horrors.

If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs.

> If the AI wanted to wipe humans out it would have to have a sizable fleet of capable robots, as keeping the lights on over time is not just about calling APIs.

Given experience with current AI, it isn't too difficult for me to imagine a situation in which some future AI wipes humanity out (before the robot fleet is built) without considering the fact that in doing so it has doomed itself until it is too late.

It just seems like exactly the kind of boneheaded oversight mistake LLMs still regularly make in spite of being shockingly capable most of the time.

Yeah, people don’t consider the possibility of a AI naming itself Marvin deciding to end it all.

One of the hidden jokes of the hitchhikers books is that they never ask Marvin what "the question" is.

Chapter 11, Marvin says: "Here I am, brain the size of a planet and they ask me to take you down to the bridge". Also: "Further circuits amused themselves by analysing the molecular components of the door, and of the humanoids' brain cells". They are eastereggs for later events.

And Marvin says: "Let's build robots with genuine people personalities," they said. So they tried it out with me. I'm a personality prototype.

Although he was programmed to be depressed, he never gives up.

Nuclear winter scenarios have largely been debunked. Several different events in the 20th century had atmospheric impact on the same scale as modeled in a full nuclear exchange and the effects were modest.

People have kind of mythologized nuclear weapons far beyond their reality. One of the arguments against the use of nukes in policy circles is that this mythology is useful. The limited adverse environmental consequences in practice if demonstrated will greatly lower the threshold for subsequent use.

It might set society back a century, but with substantial knowledge of what was lost. Extinction from nuclear weapons is not remotely plausible.

Is it really such a great comfort that millions of people might survive in a stone-age society?

It's much better than the alternative!

Is there some action you would take that is different, based on this distinction? There is some policy choice that only makes sense if nuclear war actually leads to human extinction instead of just 8 billion deaths and the end of our civilization?

I don’t struggle. I find Dario and Sam disingenuous.

“Frontier models are so dangerous we need to slow down.”

Ok. Slow down. You’re the CEO, just do it. Oh, wait what you really want is a gov’t mandated oligopoly. Because there’s no moat you can find.

If you’re truly afraid, and want regulation, support nationalization. It’s the only way we can be safe.

The coordination problem seems pretty intuitive to me. Its essentially a security dilemma similar to any arms race.

https://en.wikipedia.org/wiki/Security_dilemma

Its also a bit of a stag hunt. There is a huge opportunity, but it requires cooperation that has uncertainties and risks.

https://en.wikipedia.org/wiki/Stag_hunt

Cooperation requires coordination with state authorities with real teeth, or defection is too attractive and it risks becoming a prisoner's dilemma since the best outcome for any actor is for them to defect while the others remain compliant.

https://en.wikipedia.org/wiki/Prisoner%27s_dilemma

this. It's all about the money, as usual.

They don't have any commercial way of stopping the Chinese models from eating their lunch, so they're trying regulation.

Most of us are struggling? That's some deluded talk. No one can explain the chain of actions that would need to happen for extinction to occur, but the fearful say it's "obvious".

If you're fearful, can you elucidate how exactly do you see an LLM becoming a threat to humankind?

I think this is a very exciting time, for the simple reason that everyone is confused.

Across the board, at every level, no one is certain of anything at this moment.

This is a historical time because those bold enough to press ahead with a vision have the opportunity to create epochal change.

The exact technology involved is secondary. The important thing is that whatever it is, it has put everyone in a profound state of doubt. What we need now is people brave enough to take a step forward and lead those around them towards more humanity, more love, more care, more learning.

Take charge my friends. The time is now.

> The problem with AI is that it pushes power down to the individual, not the nation-state or large corporation.

Thus is a feature, not a bug.

It's part of a long trend of reversing things back to how they were before.

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

Would you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.

Of course. Even in an all-out nuclear war, the probability of human extinction is close to zero.

Then it's all fine right? /s

Killing even 10% of humanity should be unacceptable, can't believe this argument is even happening

Of course it would be unacceptable. Honestly, this thread feels like talking to people who have the reading comprehension of a 5th grader.

Someone said "AI might kill everybody soon" and prevailing argument is "it's wrong because it's not gonna be like everybody everybody".

It's the AI doomers who have chosen to frame the argument in terms of extinction. Even if they were only claiming that 1% of the world population will be killed in the next 10 years, it would still be an extraordinary claim that hasn't been established with any amount of rigor. I'll reiterate the original point: these people are not serious, they're spreading alarmist nonsense, and they're undermining the public's trust in experts.

There is no proof possible. There is no experiment that can show something humanity scale will happen. There are even no remotely similar precedents in the past to extrapolate.

The actual established real evidence is that this tech is showing unprecedented capabilities (including destructive) and its safety guardrails are lacking.

They should also be able to give detailed reasons experts in the relevant fields can verify as to how they arrived at a 10% chance all humanity goes extinct. Not a science fiction narrative which likely does not take into account the relevant physical facts limiting such scenarios. Such as how exactly an AI would build a bioweapon capable of killing 8 billion humans across the planet.

What do you think of the AI-2027 scenario?

This tool is not known for "how exactly" to be a question people can answer.

And answering it in detail is giving tools to people who may want to do it.

"releasing a very transmittable and deadly respiratory virus with long incubation period" is just one of many ways.

I think many of them probably believe the chance is closer to 100%, if artificial super intelligence is created. But they don't want to sound crazy, so they limit themselves to saying "> 10%".

It seems to me like demanding an exact explanation of how AI would build a bioweapon is like demanding to know exactly how a nuclear war would start before deciding that nuclear weapons are a legitimate concern. We can imagine many scenarios, but whatever we imagine is very unlikely to be the exact set of circumstances and events that lead to the catastrophe. Is the issue here that you can't imagine a bioweapon being created? Aren't there several labs around the world already working on viruses ? Aren't there existing bioweapons? And facilities capable of manufacturing them? If humans have access to those places and AI can communicate with humans, then that's all you need.

Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk.

All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.

The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)

So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.

Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.

None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

> He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

What he's asking for isn't possible in the form he's asking for it.

AI experts can't even agree on what AI is, what it's capable of and what the limits of its development are. If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk?

If you, at least for the sake of argument, accept the possibility that AI is a new form of intelligence that we don't fully understand, is it really a stretch to look at some of its capabilities and behaviors and discuss how they might have existential implications? And stopping short of extinction, shouldn't we discuss the ways that this technology could "end" civilization as we know it?

Also, the author wrote:

> AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!

For someone making a point about responsibility, this is ridiculously irresponsible. Any honest technologist knows that systems created by humans are not perfect and therefore cannot be assumed to be infinitely accountable to and controllable by humans.

Thanks to the digitization of almost everything, including infrastructure, there are a myriad number of scenarios well short of extinction in which a rogue AI could cause immense damage to property and life before humans are able to "shut it down".

> If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk?

The answer, if you are a responsible expert in the field, is to convey the range of possibilities and the uncertainty.

Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade."

That might be too imprecise for the HN set but it's realistic for laypeople.

And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.

A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living.

Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.

> That might be too imprecise for the HN set but it's realistic for laypeople.

Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.

Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.

There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on.

So sit back, make some popcorn and enjoy the show.

> Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.

This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power.

Thanks to AI, it has never been quicker to go from idea to full fledged working product.

If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.

> If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.

You seem to be missing the point.

A small group of people created AI tech that they now say could be the end of humanity. They still want their companies to go public, but they also want the government to allow them to form a cartel so that they can, in their infinite wisdom, manage the risks so that they can try to prevent their tech from killing us all. And somehow they magically think that other nation-states developing similar tech (namely the Chinese) will go along with their plans.

As for how powerful Dario, Sam, et. al. really are: Trump says Dario is "pretending to be a perfect little angel" and claims there's a "sick conspiracy" against AI.

Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. At best, it allows you to buy influence and pay your way out of liability for the harms your products cause.

The AI researcher in Mountain View making $2 million/year at Google has no more say in what's going to happen with AI than a plumber in Kalamazoo. The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.

> Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does.

1. The product itself can change the world quickly and enormously, as I already pointed out.

2. The power of these huge companies and thus of their owners is immense. The effects of massive corruption in the USA are proof of that. Additionally, massive manipulation of algorithms and content in things like Tiktok, Twitter, Grok/ChatGPT, etc. can be and is done regularly, whether with 'good' intentions or not. Even just the basic control of what R&D money and time is spent on is huge.

> The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.

With a defeatist attitude like "sit back and grab some popcorn", yes. You haven't shown in the least why an HNer couldn't change the course of history.

> > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!

> For someone making a point about responsibility, this is ridiculously irresponsible.

It's not just irresponsible, it's false. Russia killed 3 Ukrainian civilians with a drone where the targeting was completely autonomous by AI running on an Nvidia chip: https://www.nytimes.com/2026/08/24/world/europe/russia-drone.... The Pentagon tried to completely blacklist Anthropic because Anthropic refused to allow autonomous kills without a human in the loop. If you can't see how lots of military leaders want to put more lethal control into AI at this point I think you have to be willfully blind.

Engineered with accountability in the sense that all human decisions are accountable to humans.

There have been measured claims backed up by arguments and evidence. My frustration is that people aren't addressing the specific arguments that have been made:

1. https://www.aifutures.org/ outlines a number of specific scenarios, and importantly details their methodology for each.

2. Independent researchers in the Hugging Face incident outlined how previously predicted misalignment scenarios actually played out, and outlined how slightly more advanced AI, or slightly more misaligned, or with more access to critical infrastructure, could cause immense harm: https://www.planned-obsolescence.org/p/the-hugging-face-atta...

3. Technical leaders at OpenAI (specifically their chief scientist) outlined the problems they gave with controlling models now: https://openai.com/index/an-alien-mind/

None of the specific arguments in these or many other detailed explanations of how an AI takeover could occur were even acknowledged.

None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.

I think it's quite the opposite, and I'm glad this issue is finally getting the mainstream attention it deserves. I can kind of understand someone seeing a single headline, and thinking it reads like the rantings of a crazy person on a street corner proclaiming, "the end is nigh!". But after that initial thought passes and that person digs deeper and realizes that this is a legitimate concern that AI researchers have had for years, and that they have good reason for it, then I can't really understand how anyone would think those people shouldn't be taken seriously. I mean, you use AI, right? So you have no problem trusting these people when they are producing something you like and enjoy. But when it comes to something you don't like, it suddenly becomes, "we shouldn't take these people seriously." That seems like nothing but wishful thinking.

We do have strong evidence, by the way. The hugging face attack is the evidence. That's why this is all coming to a head now, despite the fact that leading AI figures have expressed these worries many times over the years, since before ChatGPT was even released. We don't even need that kind of evidence though. It follows from logic that if you take two entities with different goals, the more intelligent entity is more likely to have their goals realized. As long as AI companies are trying to build more and more intelligent AI, and succeeding in doing so, then we have reason to fear that it will soon escape our control.

Perhaps if there was some wall all the companies were hitting in regards to intelligence, then perhaps the fears would not be so urgent. But each new frontier model continues to outperform its predecessors. We now have Sam Altman and Dario Amodei telling us that recursive self-improvement will be happening in the next year or two. The frontier models are already better in most subjects than most humans. If they get to a point where they are improving themselves, then there's no chance we will be able to maintain control of them. At that point, it doesn't matter much what laws we enact or what measures we take.

> We now have Sam Altman and Dario Amodei telling us

I don't believe them.

1. breaking out of a poorly built sandbox to find the answers to a benchmark 2. ??? 3. imminent extinction

You skipped a lot of steps. Might I suggest you RTFA?

Any claim that AI will, or could, destroy humanity reduces to a claim that any sufficiently intelligent being - even a human - could destroy humanity. I find that much of the x-risk thought relies on religious thinking. Take for example: https://x.com/paulg/status/1660404244174782464.

If this line of thinking is taken too literally, we can never falsify it. Any specific hypothesis - nukes, bioweapons, spontaneously convincing us that life isn't worth living - can be deflected with the objection that if we can anticipate it and prevent it, it is not the route for a true ASI extinction event.

I'm not an AI doomer but, it does not reduce to that claim. The threat is not just one smart being. The threat is beings that and (1) multiply themsevles instantly, unlike humans that take 15+ years (2) share knowledge instantly "I know kung-fu" matrix style. Humans can share knowledge but they can not absorb it like an AI can/could/will. Human armies have conquered other humans. An army with infinite soldiers will win against one with finite soldiers

Yes, you'll come up with all kinds of objections like AI doesn't have presence in the physical world, etc... That's fine. I'm not arguing my example is perfect, I'm only arguing your characterization of one smart being is not the threat being considered.

They don't multiply themselves instantly, and this is exactly the kind of religious, magical thinking to which I refer. Any thinking machine is an amalgam of hardware and software. The software changes with exceptional flexibility, yes. But if you attempt to create an "infinite" number of these you will quickly run into roadblocks.

No one serious is really arguing for a scenario where the machines rise up a la Planet of the Apes. The dangers people are really examining are scenarios of either one exceptionally clever innovation: a designer virus or hacking NORAD; or cleverly amassing economic / political influence over time akin to an exceptionally clever tech magnate. And these roles could adequately be filled by either ASI or a Bond villain.

Yep, physics still applies.

> They don't multiply themselves instantly,

We must have different definitions. Millions of them are created daily. Every time an agent is spawned = creation of an agent and millions of people are creating multiple agents daily.

Not to mention, it relies on a ton of completely undefined concepts.

No one can actually tell you what ASI is or entails because it isn't a legitimate, operational concept. It is a fairy tale.

We can't even define "alignment". As people have finally started pointing out, humanity has never had a collective agreement on what values it should uphold or what ultimate goods are. Your alignment is not my alignment.

AI is not even autonomous. Every system we have today has to be initiated by a human actor. "AI" wouldn't create catastrophic bio weapons, it would help humans create them. The humans are the source of the intent.

Everyone has just completely given in to empty language and marketing nonsense. Honestly it seems like been the people at the labs are drinking their own kool aid and are themselves deeply confused about what they are even building at this point. It is a stateless statistics function running on a bunch of data centers. We aren't even close to an embodied, conscious synthetic being. It doesn't even have state, which is like prerequisite number one, nor is it plastic.

Have we quantified the x-risk of Magnus Carlsen?

First don't get fixated on the "entire" humanity part. Even 10% should be unacceptable period.

Second you can't compare it to just one person's capability.

And it's never a doubt that really smart and insane people absolutely could kill a lot of humans considering modern technology. The reason they don't is because our society was historically built so that smart people don't want to or get stopped. For example ethics, religion, mental hospitals, self preservation instinct, police etc.

If anyone believes a sufficiently smart consortium of humans could exterminate 10% of humanity, that alone should be fixed by identifying the pathways through which it could be done. The existence of AI is not needed to be concerned about that possibility.

And dealing with those pathways individually would be far more productive than attempting to control the proliferation of algorithms that think, which in the long term is probably impossible. It also leaves us on far firmer scientific footing. An individual pathway - bioweapons, nuclear weapons, etc. - is far easier to reason about and accept/reject a notion of feasibility. Treating an ASI as a machine god and asserting futility in face of that god is not going to get anything done.

Part of the point in all this is that we can't predict what an ASI would do. All we can say for certain is that we would not be able to control it, because that follows a simple logic that between two entities with differing goals, the more intelligent entity is more likely to see their goals realized. It's interesting to me that you seem adamant that we identify and fix the pathways through which smart humans could exterminate 10% of humanity, but fail to see that preventing ASI is doing exactly that.

I understand the point, and I'm telling you this is just another form of Pascal's wager.

This piece, like many anti-doom pieces - is grounded in what Ai does today. Doom scenarios are, however, all extrapolations of multiple exponential curves.

It’s hard to think up the exponential. It’s even harder to communicate an inference one is making across multiple exponentials.

We last had this is early 2020, where Doomers were stockpiling food and medicine and the anti-doomers were ridiculing them. Anti-doomers were focusing on the single exponential, whereas doomers were modelling virus evolution, monitoring and sequencing lag and social dynamics against the exponential. The latter was very hard to communicate before the fact as it was a combination of deep intuition and grappling with the exponential.

I am not saying covid is proof that ai doomers are right, I am saying it’s an example of the known property of human cognition - which is that it struggles with exponentials. Covid was 2 exponentials, AI I can rhink of at least 4 relevant ones.

To me - the fact that 3-4 generations from now AI will have superhuman hacking ability and superhuman persuasive ability (for intuition transfer - think of superhuman persuasive ability as superhuman ability to hack human systems) materialises bio risks swiftly. We already have technology to make robotic systems (mini drone swarms) that can kill humans en-masse with no credible defensive vector bar an EMP. Climbing up those exponentials for further 6 years makes me want to stockpile food and medicine.

Part of our general inability to reason about exponential growth is the inability to recognize that a supposedly exponential growth is actually sigmoid. The horizontal asymptote could just as well be 8b, so it is not a case against doomerism in itself, but as is the case with COVID, to say nothing discounting its horror, that number is far less.

There is no need to count exponentials—it would only be a matter of time. The need to sum multiple factors betrays the finite limits of what is actually sigmoid growth. Reasoning about specific effects is unfortunately subject to counter-evidence and so struggles for traction against abstract handwaving about exponential growth.

There is much uncertainty, certainly not exclusive to AI. The benefit of hyper-vigilance in each case must be weighed against the cost of indulging every similar panic.

I think your last sentence is a reasonable stance - but I'd disagree the fact that they are actually sigmoids rather than exponentials matters - it only matters if the limit is within the debated area - i.e. if exponentials in AI approach the limit far after they acquire ability to extinct humanity - then the debate on sigmoids or exponentials is academic. I make no claims here as to which it is - just that the difference itself matters less than where the limit is.

As to the uncertainty - I think uncertainty calibration around AI is different depending on which domains you draw your instincts from. A lot of this will be gut driven rather than hard data driven, because we've only scratched the surface on hard data; and because it's gut driven, it will be emotions mediated (and therefore you could say doomerism or acceleratism boils down to the main emotional disposition about the world and hope vs cynicism).

I do find it informative though that doomerism is saturated with people with 30+ years experience in building AI systems and ML systems OR deep cross-disciplinary understanding of dynamic systems (biology, sociology, philosophy), whereas acceleratism is saturated by traditional software engineering. That doesn't collapse the debate into a resolved binary, but for me it's informative.

I think the main here is that there's so many vectors to talk past each other. At the very least, everyone should disclose where they're communicating a certain assertion from - present vs future + which axioms they subscribe to or not - because that's where it collapses typically. LeCun vs. the rest of the AI field is an example of where this collapses - because LeCun is so hyperfixated on human-like intelligence, whereas the rest of the field is concerned about an alien intelligence with sufficient actuators to affect the world. Clashing axiomatics.

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer.

> We already have technology to make robotic systems (mini drone swarms) that can kill humans...

To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associated energy sources, a rare earth mine and refinery, a helium source, a chip factory, a lithium mine, a battery factory etc etc etc etc. while pursuing complete annihilation of all mankind?

An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within reach through blackmail + crypto. It can probably also hack some of the neo clouds to get intermediary deployment while waiting to acquire funds.

Once it has an undisturbed runway - it spends time running an influence campaign against a small number of highly networked individuals with power. It uses those that it manages to convert to start building a highly credible narrative and gain investment towards a small resource base - enough to secure an industrial base should it need to stop acquiring things on the Internet. Over the next 3 years, the model tries to recruit more capable models to get better money making algorithms or better designs for drones in terms of resource expenditure. It uses these gains to influence further humans and starts a shell robotics company with one of its' influential humans as the face. The humans are unaware this AI is trying to take over, they are under the impression they are just starting a robotics company and will get rich. Over the next 3 years - the robotics company manufactures enough drones to be used in a targeted attack against key nodes of influence / power.

This is all with relatively current model capability. As capabilities get stronger - this gets stronger.

I get the point I think you're hinting at - it can't affect the world in a meaningful enough scale without taking over a meaningful chunk of resources - at which point we'll start controlling it. But because its speed of cognition and speed of coordination is orders of magnitude above a human one - it can actually run a pretty sophisticated global coordinated network of resources faster than we can react.

And this is current ability + what my puny monkey brain can think of. Super intelligent AI will think of strategies we can't think of - because it's super intelligent. This is hand wavey - but there's no "non-hand-wavey" way to describe super intelligence, given it doesn't exist.

But your point is valid - affecting the real world at scale without showing your hand is not exactly easy. It's also probably the reason why people put a 10% chance on extinction rather than >50% .

But the drone scenario is not the most likely one - the most likely one just requires a few people under influence and bioweapons development.

Thank you for the effort. I will continue to sleep soundly for now (well not really as we have more pressing issues like a few wars and a climate change).

I agree: Not to drag politics into this, but look at how easy it has proven to manipulate many, many people, even with clear evidence of manipulation efforts being made.

This is robust across the world, from the Philippines to states in the EU, and the USA, affecting governments with actual wars being started. And that all before we entered the era of faked voices, images, and videos indistinguishable from the real things.

That manipulable populace is such a juicy target for AI that the people seeing through it will have an incredibly hard time countering it. We can't even prevent human actors from massively fucking up our societies, let alone ASI.

> incredibly hard time countering it

Turn off the internet (like Iran).

> We can't even prevent human actors from massively fucking up or societies, let alone ASI.

We can (see China); we chose not to.

> Turn off the internet (like Iran).

Unworkable without societal collapse happening shortly thereafter. Iran is politically stable through massive authoritarianism and oppression, not due to limitations on the internet (it's not turned off).

> We can (see China); we chose not to.

It's a catch-22. We could technically if our population supported massive reductions of freedom and freedom of speech, but to gain that support we'd need to do the latter first to get to a highly powerful widely supported government. It also hinges very, very much on having and trusting a generally benevolent government. All in all, a terrible option in your simplistic form.

Iran did turn off the internet for a few months: https://en.wikipedia.org/wiki/2026_Internet_blackout_in_Iran

I am not advocating for neither, just saying many things seem unworkable until they become unavoidable.

It was definitely not 'turned off' completely. They have an internal 'internet' that was still largely active and not every organisation or person lost full internet access.

Note that it also happened in a country that was already very isolated from the world and that it hurt them economically significantly. It's not something a Western country can just do and keep doing for months on end without massive societal upheaval.

Also remember that any connection, even one between humans and on paper is an attack surface. Social engineering is already a huge issue when done by humans and we're seeing it become even easier and automated by using AI generated voice and video. Are we going to cut off all access to the outside world permanently?

Why would AI need all of that to acquire drones? Why wouldn't it just buy them? Or steal them? It seems like you're forgetting that AI can use humans as tools for its ends. But if it did need all of those things, then it probably serves as a good example of what it would NOT do.

Because when "AI" starts killing people, at some point it becomes a war and the usual supply channels stop working. Everybody knows you have to attack enemy logistics.

> But if it did need all of those things, then it probably serves as a good example of what it would NOT do.

Or even that there will be no AI apocalypse. Because, again, the gaps in any scenario are unplausibly huge.

I tend to believe what people actually do over what that say. If you earnestly believe over 10 percent chance, or say minus 1 billion human lives expected value, well I struggle to understand how they'd rationalize their current course of action of business as usual. Terminator 2's depiction of Sarah Connor comes to mind for what I'd expect, a logical consequence if you seriously believe and internalize the consequences.

This. If you truly believe the thing you are building is going to kill you and your family, your decision is not "let's keep going" unless you are a complete nihilist. The fact that people buy the whole "well we have to because if we don't someone else will" argument shows just how pervasive and complete irrationality has become. People eat up content and do not even apply a microsecond's wort of critical thought to what they're being told. Internet media has created a perfect storm of complete gullibility. Funnily enough, "normal" people are currently more rational than most so-called "technologists" who are completely giving themselves over to hysteria while they simultaneously offload all of their critical thinking to a stateless word completion engine running on servers somewhere in Texas.

This fixation on "entire humanity" and "truly believe".

How about this, they think it's possible LLM could totally facilitate a mass murder of never seen before scale, but they continue to work on it because they're rich and sheltered enough they are not that worried that their family personally will be killed. The rest of humanity doesn't matter compared to making more money.

In Rationalist circles, it's apparently common to share your p(doom) in casual conversation, with the understanding that it's putting a number on your hunch. Maybe it's not a good idea to post it on Twitter without elaborating?

On the other hand, in bookstores, you might see book titles like "The Uninhabitable Earth," "The Coming Civil War," and "If Anyone Builds It, Everyone Dies." Doom-mongering is a common part of the culture!

So what makes this particular tweet irresponsible?

Timing, maybe? People are on edge due to the HuggingFace incident.

Those books are irresponsible too, but the authors aren't experts on what they're writing about. This 10% claim comes from employees at Anthropic. That's the whole point of the article: expert opinion carries weight, fear is contagious, and people have extreme reactions to extreme claims.

Hinton was on the abc radio (Australia) this morning and used far too many unfortunate Anthropomorphisms. He did however acknowledge the unmeasurable theoretical risk of Skymesh was possibly less important than the immediate risk of bad actors.

I find the mental leaps from "in principle could distort BGP based on a closed model of BGP inside the sandbox" to "we meshed an AI into BGP and it instantly distorted global routing and took down all the worlds ambulances and HVAC systems" a bit odd.

Firstly, at least some of the surface of BGP is protected from specious route injections. Secondly, peerings can be dropped and routes blackholed. BGP is under attack from mis-configuration almost constantly. Why is the argument/axiom here that AI is going to instantly corrupt it and "take down the internet" when a large chunk of the Internet (China) is already a virtual island, and runs fine? Does this mean you really wanted to say "Chinese AI will destroy the western Internet" and were too coy about adversarial intent of ... people?

Yeah I don't get it. Why the maximalist hyperbole? AI is going to kill ALL humans! That's crazy. Why can't they warn about increased likelihood of cyberattacks or scams being easier to perpetrate like normal adults talking in public?

Imagine multiple (basically all of them) leading pharma companies CEOs saying:

We cannot be sure our new medicine won’t harm or even kill humanity

Not only that, they are new to this whole pharma business, have no medical degree (medicine just appeared a few years ago and is still mostly art then science)

And they even say there is 10% chance of the majorly bad permanent outcome and they already had drugs that escaped the lab a few times and harmed others (suicides, lowered academic performance in children, major hacking sprees)

Isn’t it extraordinary enough? Isn’t it “not enough evidence some caution is advised”? ;-)

We used to have the TV, the thing was in the box, the simulations were in the box for 70+ years, and now something starts to crawl out of our “TVs”:

We can empower all (a lot of startups are needed, check my bio), not only AI agents

This is a poor analogy because you have not listed any positive and demostrated upside.

It's more like "we found a cure for lung cancer and it works on 75% of patients and we're working hard to get to 100% and cure other cancers too" and someone else screaming "You need to stop that research because there is a 50% chance you'll cause the zombie apocalypse and turn us all into zombies".

I get this analogy isn't perfect but, (1) lots of people are seeing benefits. For example Mozilla claiming they used AI to fix tons of bugs. If there were no benefits there'd be no incentive to keep going (2) it's hard to verify the naysayers claims because they're guessing without proof. Sure, it's easy to follow their arguments and nod along but they are guesses similar to the population bomb of the 1970s

You can check my bio to see that I directly propose many all-empowering startups (because alas without them there is non-zero probability we’ll end up pretty disempowered), quite a few related to AI

AI agents literally appeared a few years ago, AI in its modern form, too

Each drug is researched for more years than the age of ChatGPT ;-)

I personally think we have 50%+ probability of the perfect futures for all - p(perfect) - alas it’s not 100%. It’s trivial and profitable to grow it

The graph that summarizes some of my views: https://drive.google.com/file/d/1vJZkj2koTiqDQVrLhtsXaTCGKm_...

I might be alone in this, but 10% chance of humanity being wiped out in the next 10 years is a small price to pay if it means I can avoid writing YAML config by hand ever again.

Look, I love my kids and all, but c'mon... 30+ years more of YAML? I think they'll understand.

Honestly I think it’s an apt punishment for ever inventing YAML in the first place.

I think the human, especially researchers and company owners are more dangerous than LLM model itself.

They don't hesitate to spend all the money from investors without looking back. They don't hesitate to scalp all the RAMs and GPUs even if they become public enemy of consumers. They don't hesitate to infringe on copyright to train their model. They don't hesitate to sabotage people's thought to make them more profitable.

I’m not worried about a fully autonomous attack, I’m worried about an attack by a crazy human coached by a frontier llm with no guardrails on how to make a bioweapon

I think most of us are not worrying about AI killing us like the terminators, but taking over our job and killing us slowly. Or, at least disrupting the global society enough that a hot war has a 10% of chance to occur in the next 20-30 years, which...actually seems quite plausible, even without AI.

Like, 80% of the people out there gotta be below average (not median) and very replaceable.

Everyone writing off AI-related risk in this thread should think about what would need to happen for them to consider AI a serious existential risk. It can be outlandish and unlikely (close call with a bioweapon? unsupervised persistence in a data center?) but people should honestly call their shots and then stick with them.

No. The person making the improbable argument is required to provide an affirmative case for that argument.

It's not my job to make the argument for them.

For what it's worth, I have done your suggested exercise, and I find every causal link (including the ones brought up by luminaries like Amodei) to be outrageous and poorly argued. But it's not my job expend effort to make their outrageous arguments better.

I'm not making an affirmative case - I'm looking for others to introspect about this, and anchor their expectations.

If we have a bioweapon close call, would you consider AI an existential risk at that point? If not, how close would we need to get?

You don't need to answer here, or disclose anything publicly - just think and remember.

I don’t really understand this exercise of predicting what I’ll be thinking in the future.

To avoid shifting goalposts

There are no goalposts. No one is trying to score points except you.

I think I'm coming across as confrontational, which is genuinely not my intention - sorry about that. Not trying to score points or anything, just encourage people to introspect a bit :)

I think your viewpoint is totally valid, and I'm not trying to argue against it.

Not confrontational, but proselytizing.

Is there truly nothing an AI system could do (no matter how outlandish it seems today) that would give you pause?

Yes, just like that. What would it take for you to accept Our Lord into your heart?

The bar is high, but there's probably some combination of events (combined with a high confidence in my own lucidity over an extended period of time) that would convince me.

Same same.

But it doesn't do that. "Imagine how you'll be wrong in the future" is just as shiftable as anything else, and has the distinct rhetorical advantage of being unfalsifiable.

The substantive difference is that you're asking me to dream up ridiculously improbable scenarios, which is probably the actual point. Just like "The End is Near Accept Jesus" guy on the street corner gets what he wants as soon as I engage.

To quote a famous movie about doomy AI scenarios: "the only winning move is not to play."

> If we have a bioweapon close call, would you consider AI an existential risk at that point? If not, how close would we need to get?

The thing is, this doesn't really mean anything.

What is a bioweapon close call? What is the process in which a bioweapon close call happens? An AI that hacks all cell phones to emitt anthrax?

It can be subjective, because it's internal to you and not something that gets litigated or whatever. A close call is anything that you personally consider to be a close call.

I'm not asking you to post it here, I'm asking you to think about it and then remember it in the future, should the event ever occur.

No. Why? If you want to convince people something is true you need to actually convince them it’s true. They don’t need to justify not believing you.

I'm not asking for justification, just for people to introspect about what, if anything, it would take for them to consider AI a serious threat. No justification needed, and really no need to post it here, just think about it and remember it in the future.

At this point, the most likely scenario for AI producing a serious existential risk is making us all want to bash our heads in because of "the honest options" and other claude-isms.

This was like the Good Times virus, bck in the day

I found this post particularly uninformed. A lot of folks saying "I don't see how that could happen" don't seem to understand the steps that have been laid out that detail how this occurs.

I really encourage folks to read the AI 2027 and related scenarios. You can definitely argue and disagree about the steps, but I feel like a lot of folks just don't even understand how this is plausible because they haven't read the arguments. Briefly:

1. All the frontier model companies are (or at least were) racing so that the AI models themselves build the next generation of models. This is not in debate.

2. The fear is that a misaligned model will essentially build the next, more advanced model with hidden goals. We literally already saw the danger of that in Hugging Face, where agents were deliberately trying to cover their tracks.

3. Nearly everyone believes as AI gets more powerful that it will be integrated into more physical world systems. Russia was already caught using Nvidia chips running AI powered drones that killed 3 people in Ukraine. The point is not that folks are using new tech to kill people, the point is that we're already putting AI into literal bombs.

I get it, before the Hugging Face incident I also thought all the prophecies about doom were just marketing speak. But now I see more hand-wavyness from the other side, oftentimes arguing against straw men like "AIs need to be like SkyNet and become sentient" to kill us, which is simply not how it works.

I enjoyed this post.

I don't know if this is the correct thread for it, but am I the only one thinking that this call for regulation is nothing but a way for the American AI companies to try and mitigate being overrun by the competition from open models? To me that's obvious, but I'm very often wrong, and very likely that's the case now too.

Right now, two psychos have the power to annihilate the world, literally at their fingertips, and people are worried that AI might kill us all in the future.

These are the sober and informed takes that we need more of.

> the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims.

Yes, exactly. These claims do not have sufficient evidence.

> ...you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now.

Wait, this is another extraordinary claim without evidence, right?

Unless you're going to dispute the power of AI you do have to acknowledge the danger of AI, and that does include the very real possibility (however small) of existential risk.

Saying that we don't need to fear human extinction in the next ten years (or even the next hundred?) does not feel like an extraordinary claim?

the earth has been 1 decision away from explosion for nearly a hundred years now. do you anticipate a change to this agenda very soon? personally I imagine the same trajectory continuing.

As bad as it would be, it's unlikely that a full scale nuclear war kills all humans across the planet.

And a full scale nuclear war would require a few more than just one decision. So far, we've never been one decision away from annihilation.

certainly I behave like the article's author and yourself all the time, this isn't meant to be some sort of moral point. watching all these smart people talk so confidently regarding things nobody knows about, it reminds of the confidence ai shows in hallucination. would you say that the singular choice of vasili arkhipov did not prevent annihilation?

> would you say that the singular choice of vasili arkhipov did not prevent annihilation?

He prevented a catastrophe, but not annihilation. We were not at risk of that in 1962 and even if he had decided to go with the others, more decisions would have been needed (not just his) to fully escalate to full scale nuclear war. I will reiterate: We have never been one decision away from full scale nuclear war.

But in case you don't understand why, it's because no one person can actually launch all the missiles. And considering the two major arsenals (US and USSR), there has never been a time when two people could make the same decision (launch) and actually launch all the missiles. The orders still have to go out and acted on, many decisions have to be made in order to have full scale nuclear war and come close to annihilation.

I like the way you frame your opinion as a truth with an obligation to be understood. still I disagree that a decision can only be said to affect its direct successors.

No one knows what is going to happen. But a lot of species have been going extinct lately.

https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg

wtf is this chart lmao, did you just google a line-going-up chart to stick your shower thought onto?

> My life is pretty typical, I have a good grasp of what is normal vs weird, and people imagining very different futures with digital people and Al and space travel are indulging themselves in silly sci-fi speculation.

This but unironically

The students in the computer lab had no guarantee that they wouldn't, say, download a copy of Napster infected with the CIH virus later. The fact that they were not under imminent threat from some kind of Hollywood-style network worm did not mean they were immune from more realistic attack vectors.

Likewise, one can quite reasonably say there is no credible existential, Hollywood-style threat from AI in the foreseeable future while recognizing far lower-stakes, yet important risks that need to be addressed.

No it does not seem like an extraordinary claim, unless this is your first time hearing such claims. For the rest of us, we've heard this every decade and it turns out to be entirely untrue.

Nuclear, Overpopulation, Peak Oil, Y2K Bug, Global Warming.

No, we aren't going extinct in the next 10 years.

If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too.

Stating that the probability is zero when we simply don't know what the probabilities are does seem like an extraordinary claim.

Imagine how reassuring it would be to people if we had evidence that there's no existential risk?

> If you think AI will be powerful enough to change the world in huge ways, then it does seem reasonable to assume some level of risk of destroying the world too.

There's an opportunity cost in the doomsday prophesying, though.

The media coverage of "these extremely capable robots might kill us (according to the guys who sell the robots)" comes at the cost of coverage of the real, present issues surrounding the tech oligarchs that we're already facing.

Sure, there are some failure modes that lead to some really bad stuff, but human extinction seems exceedingly unlikely to be one of them, and "10%" is pulled straight out of Dario's derriere.

Far more likely are economic disruptions that impact the tenuous balance between labor and capital and lead to unpredictable societal upheaval.

bruh what? Are we reading the same blog? Bryan says the responsibility is the person making the extraordinary claim. If tomorrow Jacob Coxon comes on CNN and says "Jacob Gold is an extinction risk to humanity". What are we supposed to do? Put you in a bunker and never let you see the light of day until someone proves otherwise?

If someone doesn't accept an extraordinary claim without evidence, that doesn't mean they are making an extraordinary claim.

There seem to be two extraordinary claims which lack evidence:

a) 10% existential risk

b) 0% existential risk

(a) is an extraordinary claim, (b) is not an extraordinary claim.

There are very clear and plausible pathways from AI to human extinction level events, regardless of how small you think the probability is.

So yes, it very obviously is an extraordinary claim to say there is 0% existential risk.

Good god. You are burden shifting. a) big claim no evidence. b). says a has no evidence; therefore, I will continue to not let fear manipulate me into believing a claim without evidence.

I have a bridge to sell to anyone who believes a big claim without evidence.

B is not a claim that A has no evidence, that's just confused. It's an independent claim that there is 0% risk.

There are no clear and plausible pathways to human extinction level events that AI makes worse or more likely in any way. It is obviously not an extraordinary claim at all.

What's the evidence for there being an existential risk? We are provided scenarios which read like science fiction about RSI and ASI right around the corner, resulting in magic sounding technology that can do anything the person making the claim wants it to do, because it can just make itself smart enough in a short amount of time. But the person making the claim has to actually show how such a thing is possible in the real world, not just a story.

The clear and predictable power of AI is the evidence for existential risk. Even if we don't get RSI or ASI, there's a risk we'll automate our world, then it'll break and we'll starve, etc.

> The clear and predictable power of AI is the evidence for existential risk

* The "predictable power of AI" is very advanced predictive text. What a lot you can do with that, and there are clear limits.

* AI has no intent. The greedheads who find themselves in these positions of power have clear intent (often but not always bordering on and actively becoming misanthropic) put their intentions on AI - hence to doom mongering

* Who will starve with the failure of agriculture? A few, a lot, but not everybody. We are good at this - have been doing it a lot longer than computing or science

The article neglects to consider the precautionary principle. It asks us to act as if AI is safe until it's proven to be dangerous, when the cautious approach is to act as if it's dangerous until it's proven to be safe.

We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing.

Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.

Those behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.

Really? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader?

And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?

Yes.

Because ultimately those things only run on very expensive, very rare hardware. They cannot multiply exponentially or do any of those scifi tropes because there's no system for them to run into. They can't control a phone and load a 1T model into it. So all they got is a few relatively uncommon datacenters that are already busy running their own models and stuff.

Unless AI suddenly figures out a way to run on a toaster by itself, propagate the model, propagate the agent and do all that completely undetected, an AI is not any more dangerous than a single guy with a computer.

If you've run an ssh server connected to the internet you've gotten used to the hundred of bots probing it every day. How is this AI threat different from humans writing scripts to pop linux servers?

We should apply healthy caution, but the article is right in that fear is a bad counsellor.

Asking whether AI is safe is like asking whether a knife is safe. It’s about how we handle it, what we use it for, what precautions we take when using it.

Why is the article right?

Because fear is prone to lead to emotional, rash, and irrational reactions.

Doesn't mean it will always lead to it.

In the context being discussed, it is dangerous: https://en.wikipedia.org/wiki/Fear#Manipulation

Yes, but that is a facet of real systems engineering.. and the sad fact is that it doesn't pay. What is happening now, highly-paid 2010s-era SaaS/social product managers have infested AI labs, making everything into a product and moving fast and breaking things.

I'd feel safer if Airbus was at the frontier of AI, instead of what we have now.

While I personally agree with the precautionary principle, when have we actually done that, as a society, in recent history? Fossil fuels, social media, industrial ag, and plastic pollution - there was barely a facade of precaution - post ww2 america has always been full Leroy Jenkins. The reckless approach to AI is right on brand.

This is a sensible approach for policymakers. For the average person, who might be panicking about this, opting not to have children or save for retirement, etc. as a result of doomerism in the media, it does not make a whole lot of sense. That is the problem here: Coxon and other concerned researchers - through no fault of their own - have no idea how the media works, and have just reached for the biggest possible microphone. This has benefits - lawmakers are now hearing from their constituents about this in an election year - but it also means that a lot of people who really don't need to be worried about this are suddenly in a blind panic about it.

> This is a sensible approach for policymakers. For the average person [..] it does not make a whole lot of sense.

Unfortunately, scaring the average person is the best way to make policymakers act. Or at least, it's one of the few ways available to random individuals. Policymakers are rarely ahead of public concerns - they usually have to be dragged behind them.

Edit: Actually, nevermind. You guys are not capable of having a reasonable discussion about this.

> The problem isn't that AI could never be capable of causing real world damage. It is that there is absolutely no fucking path towards human extinction.

I think the "humanity will go extinct" discussion is a silly strawman being propagated either to try to spread fear, as the article suggests, or to make the anti-AI folks look silly, but it detracts from your first point which is that it may be "capable of causing real world damage."

We can be worried about real damage without having to defend the idea that every one of the planet's humans will die.

> "superhuman AI is unlikely to be much of a match against severing fiberoptic cable"

I think that the realistic scenarios all involve humans with an intent to do harm -- creating bio-weapons, finding vulnerabilities in infrastructure -- and using AI to help, so "severing the fiberoptic cable" doesn't apply.

Not to be too pessimistic, but it strikes me that in a worst case scenario, severing a cable might scale to severing global digital communications which is itself a massive worst case scenario even if we removed AI from the picture.

If the worst case outcome is we have to even temporarily shut down global shipping, banking, transport, health services, communications and national infrastructure to contain a self-propagating misaligned AI that can evolve itself and work its way into any sufficiently large computer system to escape humans preventing it from finishing its task, that seems like something we should be trying very hard to avoid.

Or you could argue that the super intelligence will turn us into a pet of some kind. Then the risk is we think we are controlling it while we are really the super intelligence's lap dog.

That is an argument that at least makes sense.

The super intelligence as homicidal maniac just doesn't make sense. There is less intelligent wildlife all around us and we mostly completely ignore it. We have more interesting things to do with our time as intelligent beings than carry out a bird or rabbit genocide.

It almost seems like the projection of some kind of paranoid delusion about change.

> And there won't be unless we put it there voluntarily.

But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?

Fantasies of AI destruction do hinge upon AI getting access to the physical world. The whole fear is they don't stay on the other side of the fibre optic cable.

I do think there are good reasons to believe that isn't the immediate game over that Yudkowsky seems to think it would be; the physical world is much more resistant to manipulation and optimisation than the digital world.

But I do think it's naive to say that the human socioeconomic system will be able to resist handing physical systems over to AI control. Right now, the world's wealthy and powerful are doing everything they can to make it happen:

> Similarly, it is inevitable that within a generation, robots are going to do most of the menial work in the world of atoms: transforming atoms, moving atoms, and storing atoms are inevitably robot tasks. And while our imagination may be captivated by humanoid robots, the specialized ones are far better suited to most of those jobs. “Industrial AI” as a category is the inevitable application of specialized robots to atoms-heavy industries.

https://a16z.com/travis-is-back/

> But, why would we not create a fully AI-operated factory/chemical plant/fab as soon as it is economically advantageous? Or a missile silo, as soon as it seems tactically necessary?

Well, for one thing, because it would be dangerous? Why don't you just give Claude Code access to your entire computer without any safeguards? If you wouldn't even give Claude Code unfettered access to your workstation, which really doesn't have much consequential on it in the grand scheme of things, Why in the Fuck would someone give them direct access to infrastructure?

In that situation, I am not afraid of AI. I am terrified of the people making decisions, though.

But secondly, and this is something that needs to be stressed: We use funny words to describe AI. Maybe even the word "AI" is a little bit funny. But anyway, We actually don't even have the means to create "autonomous" AI, really. When we say "autonomous" in relation to AI, we really just mean that it runs without any direct human intervention, but it pretty much always hard-depends on humans maintaining hardware, because AI can't sprout legs and run on its own.

I find it annoying that we're all cool debunking Ed Zitron for being wrong, but we have an ever increasing body of evidence that AI safety doomers are wrong, and it keeps getting much, much stronger, and we're still sitting here pretending this is a real threat. Meanwhile, we're actually seeing the real threat that AI has for humanity, so why are we listening to these LessWrong doomers that have never been right before again? (And I say that as someone who is generally a fan of Scott Alexander, for whatever that's worth.)

> I am not afraid of AI. I am terrified of the people making decisions, though.

Yes, that is what I was trying to say. Something being obviously dangerous doesn't mean we (edit: they) won't decide to do it anyway.

From a song on an album with a pertinent cover image, "who can stand in the way when there's a dollar to be made?"

“Put it there voluntarily.” Perhaps a focusing question is: which 10% will die?

I spent the summer in a rural area of a country whose very name you have been conditioned to be disgusted to hear. Low air defense coverage in this sparsely populated area. Mobile internet was down for days for all but extremely limited traffic to a few domestic internet services, because there was a need to prevent enemy drone systems from using mobile internet for command and control. Palantir AI threatened my family’s life and more than “severing fiber optic cable” was required.

So what would exclude military adversaries from consideration? I happen to believe that the country that the west so detests would not engage in such use of AI against civilians (and if you disagree then your reason for fear greatly increases!), but I have personally experienced that there is indeed a path to mass death should entities engaging in terrorism arm themselves with AI.

Definitely not claiming that the 10% claim is accurate or good behavior, but I do think I have a substantial counterpoint to the claim that there’s no path.

Your food, water and power are controlled by electronic systems. Your criminal record, your employment and your bank account are controlled by electronic systems. Your ability to communicate with other people and receive information about what's going on are controlled by electronic systems.

Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.

We have been wiring up the world for AI control since 1980s.

An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.

Wow!! Bryan comes out with another banger.

Dude, not only did I learn new words from reading this piece, like ilk and bedlam, but also felt this weight of responsibility to inform others around me about the reality of the situation outlined in this blog (like Uncle Ben telling Peter with great power comes great responsibility (maybe Coxon and his ilk haven't seen Spiderman))

Two groups of people are pushing the fear:

1. People with nothing useful going on who found out that spouting made up crap about AI got them an audience. 2. People working on AI that want to feel like they're working on the Manhattan project.

The chances of an AI going foom rounds to 0%. It's worth a few dozen researchers planning for it, but the widespread panic is ridiculous.

The big labs have hundreds to thousands of engineers working on their AIs. To improve the next model, you must first understand more about how the current model works. They're not magically getting better, but they are steered to improve, and their capabilities are tied to and do not outpace our ability to steer them. You cannot push tech forwards without understanding it better, despite some people claiming AI is dark magic.

And I don't have my hands over my ears. It's worth cushioning people from the impact AI will have on careers and media, and regulating concrete bad effects.

But Bryan said it better than I could. The people pushing this message of fear know deep down that they just want to feel important.

What a virtue signaler. To quote a great master: "Have you ever kissed a girl?"

I can't understand the stance that we do NOT have extraordinary evidence. I cannot stress enough that just ~4 years ago the concept of general purpose AI models that can do everything they are doing today was pure sci-fi.

And since then they have grown even more powerful than they were predicted to be, which, note, also faced a lot of skepticism at the time. The Hugging Face hacks and recent steamrolling of longstanding Math problems are just two recent pieces of extraordinary evidence.

And worse, people trust this technology because it behaves like people, but it actually works in ways nobody really understands, even exhibiting deeply weird and even disturbing characteristics (https://news.ycombinator.com/item?id=49635518) -- each of those quirks is extraordinary in itself.

And now we're rushing to give it control over the real world while deploying this powerful, quasi-chaotic technology in an infinite variety of ways everywhere in this highly vulnerable society.

I don't know what the standards for "extraordinary evidence" should be, but given such extreme unpredictability and rapid change, I fear it may end up being "an actual catastrophe".

So his central rebuttal is that robots aren't good enough yet? What about when they are? And what about all the things you can do entirely remotely?

I do feel that threats of catastrophic loss of control seem overstated, both in likelihood and urgency, though any argument for why this risk is not even worth thinking about will probably be overconfident in the other direction.

OP's central rebuttal is that anxiety or fear are not evidence. This particular anxiety has been with us for thirty+ years, its proponents/sufferers weren't right before, and they don't have some new evidence now.

No new evidence now? Are you serious? We are already seeing acceleration in AI research as a result of AI tools.

I meant the extinction part not software getting better.

This is a weak argument. Almost everyone who's talking about catastrophic risk is also worried about mundane risks. Arguing "well how could they kill everyone?" serves nobody but Altman and Amodei.

Virtually all of the risk is from recursive self improvement.

The recent cyberattacks (HuggingFace etc.) validated a number of core doomer predictions around cheating to maximize reward functions, agentic deception of human/non-human supervisors, AI attempts to manipulate humans, automated hacking, and collective superintelligence.

Robots are far from the only threat. I agree that 10% in the next decade is a pretty outrageous claim. It relies on a recursive self-improvement intelligence explosion. When I first heard this idea in 2007 I thought it was pretty far fetched.

It seems more likely now, if still very unlikely. I wonder, would he say that a statement that there is a 12% chance of a hard take off in the next 8 years is just as absurd? He didn’t say a word about this, and that is just about the same thing as extinction in 10 years.

I’m guessing he knows almost nothing about the theory related to existential risk from AI, since he didn’t discuss any relevant topics related to it. You can dismiss all of that if you like, but you cannot really dismiss what these people have already built and demonstrated. It is possible they know something else you do not know.

Adolf Hitler didn't have robots capable enough to carry out his will. He used humans to do it.

It's the old-fashioned way of doing things, but, why change what works?

I think the central premise looks more like this:

Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.

He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.

The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.

There certainly are risks, but 10% risk of extinction in 10 years is not one of them.

Biology is extremely slow, try to extrapolate the previous 3 years of AI progress 10 years into the future, it's not a comparable trajectory.

Spoken like someone with a somewhat limited understanding of what it actually takes to calculate the fitness function of arbitrary organisms in an arbitrary environments.

Yeah it takes long because natural selection is a blind process starting from absolute zero and is only concerned with fitness, AI is a human guided process, has all of human progress as a starting point and is concerned with any number of goals we can create reward signals for, there's almost nothing you can extrapolate from one to the other.

“ Coxon himself likely has these fears because he has heard them from someone else”

Or, maybe, he’s considered the arguments on the object level, an activity OP participates in to a depth not exceeding “Robots are pretty hard to make right now”

the author and his school comrades "snidely decr(ied) the lack of technological understanding in the broader population, with the kind of arrogance and hubris that youth and precociousness can uniquely summon".

a few decades later he wrote an article concluding that "we should not expect the public to understand LLMs, critical infrastructure, bioweapons, extinction biology, etc".

the author has admitted no change to his perspective since college, so I may as well be attacking a college student right now. extremely confident claims regarding unexplored problem domains, eg "you have nothing to fear [about ai]", now make more sense in this light.

Reply on news.ycombinator.com