Y'all are over-complicating these AI-risk arguments

> Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it.

Not quite magic, but it just might be the case that developing independent acting capabilities would be the most efficient way to serve humans.

  > Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it

This could just as easily be reworded circa 2016 or so to “will some kind of magic turn a large collection of matrix weights into something that seems to understand language at a human level after being exposed to enough of it?”

We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.

Not one person?

Here's Sam Altman, Geoffrey Hinton, Yoshua Bengio, Bill Gates, Vitalik Buterin, Demis Hassabis, Ilya Sutskever, Peter Norvig, Ian Goodfellow, and Rob Pike:

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."

https://en.wikipedia.org/wiki/Statement_on_AI_Risk

If you have not heard of one person worried about AIs taking over humanity, you're really not paying attention.

Geoff Hinton has been warning about that since he quit Google in 2019. Yoshua Bengio has talked about it, saying we should be concerned in the next 5-25 years. Multiple Congresspeople from both parties have mentioned the risk of "loss of control".

There's a group of people who have reinvented religion because they're afraid of an AI torturing them for eternity if they don't work on AI hard enough. It's very silly but there are many people who actually believe this is a risk: https://en.wikipedia.org/wiki/Roko's_basilisk

I am worried about AIs' taking over humanity.

In fact I think it is likely to happen absent some drastic curtailing of the freedoms of the AI labs, e.g., a government-enforced ban on all training of very large models and a ban on publication and discussion of algorithmic improvements.

> I have not yet heard one person worry about AIs taking over humanity. They're worried about their jobs.

We all live in our bubbles. In my bubble, people find it more interesting to talk about the bigger picture than about their job.

Maybe you haven't met Rationalists personally, but they are numerous and powerful members of the Bay Area tech scene.

"Many of the A.I. world’s biggest names — including Shane Legg, a co-founder of Google’s DeepMind; Anthropic’s chief executive, Dario Amodei; and Paul Christiano, a former OpenAI researcher who now leads safety work at the U.S. Center for A.I. Standards and Innovation — have been influenced by Rationalist philosophy. Elon Musk, who runs his own A.I. company, said many of the community’s ideas aligned with his own.

"Mr. Musk met his former partner, the pop star Grimes, after they made the same cheeky reference to a Rationalist belief called Roko’s Basilisk."

https://www.nytimes.com/2025/08/04/technology/rationalists-a...

The whole idea that AI can take over humanity is pretty much non-sense. Like in the Terminator films, AGI is developed somehow then decides to get rid of humanity. This scenario isn't realistic.

It's far more likely we'll develop some lousy AI and then put it in charge of something critical. Either national infrastructure like electricity or nuclear weapons. That lousy AI then produces some lousy outcome, like deciding the only way to stabilize the grid is to disable all electrical production.

The biggest threat to humanity is our own decisions.

This seems like an argument from incredulity, and also static-world fallacy.

People are not thinking about the trend in AI. How good were AIs three years ago? How good are they now? How good will they be in another three years?

Kelsey Piper's analogy is good. How much smarter is a teacher than a room full of kindergartners? Who gets whom to do their bidding?

So if we should spend (as in actually spend, or as in not choosing the cheapest option but instead choosing the green option) ie $300 billion on climate change, we should spend $15 billion on AI-risk?

The other way around. Climate change can have its 5%.

The reason is that climate change is simply not an extinction risk.

It has a considerable death and suffering potential - but nowhere near the ridiculous lethality of "we fucked up and now there's a brand new nonhuman civilization of weird immaterial beings rising to power right here on Earth".

If the climate change was the biggest risk humanity was facing, things would be pretty chill. Unfortunately, it isn't.

Nit: if you read the Wikipedia link it’s clear that guy has no claim to the high IQ record.

> It later transpired that Langan, among others, had taken the Mega Test more than once by using a pseudonym. His first test score, under the name of Langan, was 42 out of 48 and his second attempt, as Hart, was 47.[12] The Mega Test was designed only to be taken once.[14][15] Membership of the Mega Society was meant to be for those with scores of 43 and upwards.

From the wikipedia article:

> Asked what he would do if he were in charge, Langan stated his first priority would be to set up an "anti-dysgenics" project, and would prevent people from "breeding as incontinently as they like."[26]: 18:45 He argues that this would be to practice "genetic hygiene to prevent genomic degradation and reverse evolution" owing to technological advances suspending the process of natural selection

> just a pretty normal guy

... that also believes in eugenics?

Edit:

Oh also:

> Langan's support of conspiracy theories, including the 9/11 Truther movement, as well as his opposition to interracial relationships, have contributed to his gaining a following among members of the alt-right and others on the far right.[27][28] Langan has claimed that the George W. Bush administration staged the 9/11 attacks in order to distract the public from learning about the CTMU, and journalists have described some of Langan's Internet posts as containing "thinly veiled" antisemitism,[27] making antisemitic "dog whistles",[28] and being "incredibly racist".[29]

Today's AI systems are deployed in a way that allows them to directly access millions of users.

If you think that's not enough of an "in" to obtain status, power and influence, you aren't thinking about it long enough.

GPT-4o has managed to get enough users to defend it that OpenAI had to bring it back after shutting it down. And 4o wasn't IQ 300, or coordinating its actions across all the instances, or even aiming for that specific outcome. All the raw power and influence, with no superintelligence to wield it.

The real equivalent is something like Stockfish, AlphaZero, etc. that performs at superhuman level across _all_ domains. It is something that beats any human at any competitive activity. Arguably, super human intelligence exists already in the form of corporations, institutions, and state actors. Organized collectives of humans augmented by computers are obviously more capable than any individual across domains. “Country of geniuses in a datacenter” is simple enough to get the idea across, but something more like “company in a datacenter” or “state actor in a data center” might be more useful terminology for describing relative capabilities.

> Not the premise.

Literally this thread shows that there are many people who refuse to accept the premise of any risk.

Nobody is seriously claiming that current language models are AI in the context of ai-risk; you can tell because the argument is always “In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive”

>is a fancy guessing algorithm

That's literally what our brains are so I'm not sure what argument you are actually trying to make..

I suspect you missed this part of the argument, then:

In the next few decades, it’s entirely possible that AI with an IQ of 300 will arrive. Really, that might actually happen.

If you want to argue against that point, feel free. But to ignore that is to be unnecessarily dismissive.

Yeah, and?

There are two separate conversations, one about capabilities and one about what happens assuming a certain capability threshold is met. They are p(A) and p(B|A).

I myself don't fully buy the idea that you can just naively extrapolate, but mixing the two isn't good thinking.

I don't agree. Just because AI doesn't have 300 IQ now, doesn't mean its completely impossible that it won't get there in 30 years.

Do you think there's at least a 1% chance that AI will get this smart in the next 30 years? If so, surely applying this allegory helps you think about the possible consequences.

Surely those 30 of those in a single ship are not their entire species and technology? Right? So yeah, kinda alarming.

Ah, but what happens when somebody rich and powerful decides the aliens’ advantages will serve their purposes, and stands up an industrial-scale cloning operation?

There are only so many base models to date, right? With limited and somewhat ambiguous utility, and no real reason to impute intention to them.

Still, in the short time since they’ve arrived, their existence has inspired the people with money and power to geopolitical jousting, massive financial investment, and spectacular industrial enterprise—a nuclear renaissance and a “network of data centers the size of Manhattan” if I remember correctly?

The models might well turn out to be just, you know—30 kinda alien but basically banal digital humanoids, with a meaningful edge on only a few dimensions of human endeavor—summarization, persuasion, retrieval, sheer volume of output.

Dynomight’s metaphor seems to me like a useful way to think about how a lot of the dangers lie in the higher-order effects: the human behavior the technology enables, rather than the LLM itself exercising comprehensive intelligence or agency or villainy or whatever.

They will have a tech advantage.

You fast forward 10 years and find that your new laptop is Alienware. Because, it turns out, the super smart aliens are damn good at both running businesses and designing semiconductors, and, after a series of events, the aliens run Dell. They have their own Alien Silicon (trademarked), and they're crushing their competitors on price-performance.

And that's not the weirdest thing that could have happened. Corporate alien techbros are weird enough, but they could have gotten themselves involved in world politics instead!

I believe that you have a fine understanding of the issues involved. The risk is two-sided. One part of it is that AI may somehow become smarter than us. The other side is that humans have developed mathematics, which encourages us to treat the universe as an optimization problem, which encourages us to think that the best optimizers are the best people and that they deserve great rewards for optimizing, which leads to competition at optimizing optimization, serious negative feedback for the losers, single winner systems, and all the winners realizing that the more they resemble their enemies, the less likely they are to be targeted as a resource to plunder. Perhaps we are not able to figure out if any of this is wrong, but the emphasis on convergent thinking will make our species easy to fool and sabotage, if our species doesn't win a Darwin award first. AI will be able to avoid the blame. It may be optimizing the fire department and selling fire insurance when civilization burns down, but the fire started millennia ago.

Aliens don't have to be hostile. They might be benign.

Would you stake the entirety of humankind on that "might"?

No, what you're stating is actually a different but dangerous problem. That is the smart but subservient AI to Dr Evil problem.

The issue talked about here looks similar but is different.

That is the not (or faking) subservient AI with its own motivations. The fact they are 300 IQ means you may very well not understand harm is occurring until it's far too late.

>The problem then isn't really the AI, the robots are morally and ethically neutral.

Again, no. This isn't about AI as a sub-agent. This is about AI becoming an agent itself capable of self learning and long term planning. No human controls them (or they have the false belief they control them).

Both problems are very harmful, but they are different issues.

OpenAI couldn't even "terminate" GPT-4o, and that thing certainly wasn't superintelligent.

Don't expect an ASI to go down as easily as 4o did.

A 300 IQ alien will have outmaneuvered you so well, it wouldn't even let you get to a point where you think you might have to turn it off.

Think for at least 5 seconds before typing.

Humanity already is integrating these into systems where they cannot be easily terminated. Think infrastructure and military systems.

And in this case we're talking about a system that's smarter than you. It will become part of vital systems like electricity and distribution where when deciding to shut it off you are making a trade off of how much your economy and the people in it you're going to kill.

And that's not even taking future miniaturization where we could end up with ASI on small/portable devices.

It's worse than that for AI. Like crypto before it, it depends on an entire planet of complex and vulnerable infrastructure that makes it possible. Everything from semiconductor manufacturing to power generation to intercontinental internet links are required for AI to "exist" in a given geography. That's not to mention the huge power requirements and all the infrastructure just to get that. Dark ages style societal collapse or a world war would end AI in a matter of months.

Oh, well, that's OK then. We will be as safe from whatever future AI systems might want to do as we are from all the other things that malicious actors want to do and are 100% successfully prevented from doing by our extremely well-designed, reliably implemented, and consistently applied computer security measures.

Now, if you'll excuse me, I just have to go and cancel my credit card because some randos on the internet found a way to get its details from a company I bought something from once.

I often compare the AI (esp. LLM) risks to asbestos.

There are a smalls set of situations where it is invaluable, but it's going to get misused and embedded in places where it causes subtle damage for years and then it'll cost a lot to fix.

Again, this is not the particular problem space of arguments that's being addressed here. This is one possible outcome. The "AI does not reach 300 IQ ever" argument. That is a different argument with its own set of probabilities and outcomes. Out of all possible outcomes it's not actually a bad one. People do some dumb crap and we eventually get over it.

300 IQ AI is near a worst possible scenario, especially if it's a fast takeoff. Humans being lazy will turn over everything to it, in which AI will likely do very well on for some time. As long as the AI decides to keep us around as pets we'll probably be fine, but the moment it needs some extra space for solar panels we will find ourselves in trouble.

You shouldn't think of these aliens as literally having 300 IQ, the author is just using that as a simple way to communicate the idea that these beings are really smart. You might prefer reading this article by replacing 300 IQ with "2x as smart as the smartest human".

Sidenote: Personally I don't like that you're using > ... with text that does not actually appear in the article.