Hacker News Reader

Comment by fasterik | Hacker News Reader

This is an excellent piece. Note that he is not saying that AI doesn't pose a risk. He's saying that it's irresponsible to make sensational, maximalist claims without strong evidence. If someone says that there's a 10% chance of human extinction by 2036, you can and should immediately stop taking them seriously.

View on HN

They should also be able to give detailed reasons experts in the relevant fields can verify as to how they arrived at a 10% chance all humanity goes extinct. Not a science fiction narrative which likely does not take into account the relevant physical facts limiting such scenarios. Such as how exactly an AI would build a bioweapon capable of killing 8 billion humans across the planet.

What do you think of the AI-2027 scenario?

This tool is not known for "how exactly" to be a question people can answer.

And answering it in detail is giving tools to people who may want to do it.

"releasing a very transmittable and deadly respiratory virus with long incubation period" is just one of many ways.

I think many of them probably believe the chance is closer to 100%, if artificial super intelligence is created. But they don't want to sound crazy, so they limit themselves to saying "> 10%".

It seems to me like demanding an exact explanation of how AI would build a bioweapon is like demanding to know exactly how a nuclear war would start before deciding that nuclear weapons are a legitimate concern. We can imagine many scenarios, but whatever we imagine is very unlikely to be the exact set of circumstances and events that lead to the catastrophe. Is the issue here that you can't imagine a bioweapon being created? Aren't there several labs around the world already working on viruses ? Aren't there existing bioweapons? And facilities capable of manufacturing them? If humans have access to those places and AI can communicate with humans, then that's all you need.

I think it's quite the opposite, and I'm glad this issue is finally getting the mainstream attention it deserves. I can kind of understand someone seeing a single headline, and thinking it reads like the rantings of a crazy person on a street corner proclaiming, "the end is nigh!". But after that initial thought passes and that person digs deeper and realizes that this is a legitimate concern that AI researchers have had for years, and that they have good reason for it, then I can't really understand how anyone would think those people shouldn't be taken seriously. I mean, you use AI, right? So you have no problem trusting these people when they are producing something you like and enjoy. But when it comes to something you don't like, it suddenly becomes, "we shouldn't take these people seriously." That seems like nothing but wishful thinking.

We do have strong evidence, by the way. The hugging face attack is the evidence. That's why this is all coming to a head now, despite the fact that leading AI figures have expressed these worries many times over the years, since before ChatGPT was even released. We don't even need that kind of evidence though. It follows from logic that if you take two entities with different goals, the more intelligent entity is more likely to have their goals realized. As long as AI companies are trying to build more and more intelligent AI, and succeeding in doing so, then we have reason to fear that it will soon escape our control.

Perhaps if there was some wall all the companies were hitting in regards to intelligence, then perhaps the fears would not be so urgent. But each new frontier model continues to outperform its predecessors. We now have Sam Altman and Dario Amodei telling us that recursive self-improvement will be happening in the next year or two. The frontier models are already better in most subjects than most humans. If they get to a point where they are improving themselves, then there's no chance we will be able to maintain control of them. At that point, it doesn't matter much what laws we enact or what measures we take.

> We now have Sam Altman and Dario Amodei telling us

I don't believe them.

1. breaking out of a poorly built sandbox to find the answers to a benchmark 2. ??? 3. imminent extinction

You skipped a lot of steps. Might I suggest you RTFA?

Well, I'll have to disagree that this is an excellent piece, but that's another issue. And I do agree that AI killing all humans by 2036 doesn't appear plausible to me. But what is plausible is we could easily be down a path so that by 2036 "future doom" already is a very likely risk.

All of the frontier AI companies have been racing to automate themselves, that is, where AI fully autonomously build the next generation of models. Whether this leads to recursive self improvement is a valid question, but a lot of folks think they are close.

The fear is that a misaligned AI will be building the next model with deliberately hidden motives, similar to some of the behaviors seen in the Hugging Face and related attacks. That is why there is such a big push for interpretability, and why it's highly concerning (a) chains of thought are getting harder to interpret in any case, and (b) companies will go more towards things like looping transformers and "neuralese" where thought processes are completely opaque (i.e. https://www.theinformation.com/articles/secret-technique-beh...)

So the belief is not so much that AI kills us all by 2036, but that instead AI is recursively improving by that time and all seems awesome and great so we put it into more systems that can affect the real world (as we've already begun to do, like literal lethal aerial drones). Things then all go along looking great until AI decides humans are a hindrance to its (hidden) goals.

Again, I think it's fine to argue against specific steps in that scenario, but putting out a blog post saying "this is overhyped bullshit" is not exactly making a cogent argument.

None of that has anything to do with the article, and if you thought the message was "this is overhyped bullshit" then you should go back and read it again. As I already pointed out, he isn't saying anything about the probablity of harms or disasters from AI. He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

> He's addressing a specific claim about human extinction, and making a broader point about the responsibility of experts to make measured claims backed up by arguments and evidence.

What he's asking for isn't possible in the form he's asking for it.

AI experts can't even agree on what AI is, what it's capable of and what the limits of its development are. If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk?

If you, at least for the sake of argument, accept the possibility that AI is a new form of intelligence that we don't fully understand, is it really a stretch to look at some of its capabilities and behaviors and discuss how they might have existential implications? And stopping short of extinction, shouldn't we discuss the ways that this technology could "end" civilization as we know it?

Also, the author wrote:

> AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!

For someone making a point about responsibility, this is ridiculously irresponsible. Any honest technologist knows that systems created by humans are not perfect and therefore cannot be assumed to be infinitely accountable to and controllable by humans.

Thanks to the digitization of almost everything, including infrastructure, there are a myriad number of scenarios well short of extinction in which a rogue AI could cause immense damage to property and life before humans are able to "shut it down".

> If the experts can't even agree on what's happening "inside of" these LLMs, how can they give laypeople an assessment of the risk?

The answer, if you are a responsible expert in the field, is to convey the range of possibilities and the uncertainty.

Isn't that what they're doing? "There's a better than 0 chance this thing could kill everyone in the next decade."

That might be too imprecise for the HN set but it's realistic for laypeople.

And none of the AI people talking about the risk are running into rooms full of people telling them Claude has gone mad and yelling at them to disconnect from the internet and turn off their devices immediately.

A key facet of the suggested risk is that these programs would attempt to mask whatever behavior would bring about the end of the world. A world in which AI doom could be detected before it’s too late and stopped by metaphorically running into rooms full of people telling people to unplug is less extreme than that in which doomers argue we are already living.

Alarm takes severity and scale. Forecasting double-digit odds of near-term human extinction on national television to a lay audience is more extreme than yelling to unplug rogue AI. For what it’s worth, the latter has already occurred numerous times at containable scales in leading labs, mitigated by the physical realities of computation, notwithstanding the competence of involved personnel. People running into rooms yelling is probably not a hypothetical.

> That might be too imprecise for the HN set but it's realistic for laypeople.

Some 'laypeople' will be scared for a while and then return to more pressing issues and exactly nothing good will come out of that. If there really are some issues worth discussing the first step the AI doom crowd should do is to stop the PR offensive and concentrate on producing verifiable claims and actionable small steps. Otherwise their effort will fail this time and when/if a next time comes, they will have a lot less PR capital to burn.

Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.

There is no level of outrage that is going to stop the frontier AI labs, and the US government isn't going to step in to protect humanity. The Chinese are going to do what they're going to do. And so on.

So sit back, make some popcorn and enjoy the show.

> Here's the reality: us plebs, regardless of how tech-savvy we are or aren't, have no say in what's going to happen.

This sentiment is incredibly out of place on HN. Facebook, Twitter, and many other very simple bits of tech transformed the world. Tiny startups founded by a few people have (also recently) ballooned into behemoths that hold vast amounts of economical and technical power.

Thanks to AI, it has never been quicker to go from idea to full fledged working product.

If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.

> If anything will change the course of the world (for the better or worse), it will be a tech product, possibly created by someone on this forum.

You seem to be missing the point.

A small group of people created AI tech that they now say could be the end of humanity. They still want their companies to go public, but they also want the government to allow them to form a cartel so that they can, in their infinite wisdom, manage the risks so that they can try to prevent their tech from killing us all. And somehow they magically think that other nation-states developing similar tech (namely the Chinese) will go along with their plans.

As for how powerful Dario, Sam, et. al. really are: Trump says Dario is "pretending to be a perfect little angel" and claims there's a "sick conspiracy" against AI.

Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does. At best, it allows you to buy influence and pay your way out of liability for the harms your products cause.

The AI researcher in Mountain View making $2 million/year at Google has no more say in what's going to happen with AI than a plumber in Kalamazoo. The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.

> Having billions of dollars and being the head of a world-changing company doesn't buy the type of power you think it does.

1. The product itself can change the world quickly and enormously, as I already pointed out.

2. The power of these huge companies and thus of their owners is immense. The effects of massive corruption in the USA are proof of that. Additionally, massive manipulation of algorithms and content in things like Tiktok, Twitter, Grok/ChatGPT, etc. can be and is done regularly, whether with 'good' intentions or not. Even just the basic control of what R&D money and time is spent on is huge.

> The HNer working on a startup, in the final analysis, will in 50 years' time have left about as big a mark on the planet as a greeter at Walmart.

With a defeatist attitude like "sit back and grab some popcorn", yes. You haven't shown in the least why an HNer couldn't change the course of history.

> > AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!

> For someone making a point about responsibility, this is ridiculously irresponsible.

It's not just irresponsible, it's false. Russia killed 3 Ukrainian civilians with a drone where the targeting was completely autonomous by AI running on an Nvidia chip: https://www.nytimes.com/2026/08/24/world/europe/russia-drone.... The Pentagon tried to completely blacklist Anthropic because Anthropic refused to allow autonomous kills without a human in the loop. If you can't see how lots of military leaders want to put more lethal control into AI at this point I think you have to be willfully blind.

Engineered with accountability in the sense that all human decisions are accountable to humans.

There have been measured claims backed up by arguments and evidence. My frustration is that people aren't addressing the specific arguments that have been made:

1. https://www.aifutures.org/ outlines a number of specific scenarios, and importantly details their methodology for each.

2. Independent researchers in the Hugging Face incident outlined how previously predicted misalignment scenarios actually played out, and outlined how slightly more advanced AI, or slightly more misaligned, or with more access to critical infrastructure, could cause immense harm: https://www.planned-obsolescence.org/p/the-hugging-face-atta...

3. Technical leaders at OpenAI (specifically their chief scientist) outlined the problems they gave with controlling models now: https://openai.com/index/an-alien-mind/

None of the specific arguments in these or many other detailed explanations of how an AI takeover could occur were even acknowledged.

None of those are arguments giving a probability of human extinction before 2036 or any other time. AI Futures has said that some members of their team believe that it's 10-30% at some unspecified point in the future, but there is no justification given for where those numbers come from.

Would you have said the same thing during the Cold War when nuclear weapons were proliferating? That’s the equivalent of what the developing offensive capabilities of models, basically cyber nukes. Or WMDs in general. OpenAI is accidentally hacking people, if someone made the decision to deliberately direct an agent swarm to attack national infrastructure you don’t think they could do much worse? Human extinction is a long shot but I wouldn’t say the same about a mass casualty event of some kind, and who knows what that might spark.

Of course. Even in an all-out nuclear war, the probability of human extinction is close to zero.

Then it's all fine right? /s

Killing even 10% of humanity should be unacceptable, can't believe this argument is even happening

Of course it would be unacceptable. Honestly, this thread feels like talking to people who have the reading comprehension of a 5th grader.

Someone said "AI might kill everybody soon" and prevailing argument is "it's wrong because it's not gonna be like everybody everybody".

It's the AI doomers who have chosen to frame the argument in terms of extinction. Even if they were only claiming that 1% of the world population will be killed in the next 10 years, it would still be an extraordinary claim that hasn't been established with any amount of rigor. I'll reiterate the original point: these people are not serious, they're spreading alarmist nonsense, and they're undermining the public's trust in experts.

There is no proof possible. There is no experiment that can show something humanity scale will happen. There are even no remotely similar precedents in the past to extrapolate.

The actual established real evidence is that this tech is showing unprecedented capabilities (including destructive) and its safety guardrails are lacking.

Reply on news.ycombinator.com