We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
A pandemic is perfectly plausible.
https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...
https://www.youtube.com/watch?v=7wy3xyoXYt8
Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1
"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k
https://news.ycombinator.com/item?id=49737985
Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
What is missing from that to say AI safety is a reasonable position?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
https://owl.excelsior.edu/argument-and-critical-thinking/log...
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.