I resigned from Anthropic today

But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.

If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).

For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.

To be fair, that's a conservative "defend against the last war" kind of prediction, though!

( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 )

Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.

Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.

Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.

I don't know, maybe a mass shooting?

https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...

Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.

In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.

The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...

Nuclear weapons don’t have AI but AI can have nuclear weapons

What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.

It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.

If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.

The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.

By the point you would have been worried, would it have been too late?

- Escape sandbox

- Reproduce itself

- Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between)

- Setup or buy a social network

- start manipulating public opinion on that network to support legislation allowing AI to

* operate businesses

* setup legal entities

* purchase weapons

* donate to political parties

* setup private armies

* you get the idea

I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure.

But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.

> Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.

There are things that, by the time you see direct observable evidence for them, it's probably too late.

Also your example 3 is a straw-man. There's no need for "widespread starvation" to be concerned about "AI driven labor market disruptions."

Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground and rule from the shadows for the next thousands of years.

This would be their wet dream. All in the name of "saving humanity from itself" - so they can convince themselves they're the good guys and deserving of this power. Anyone seen the latest season of Silo by the way?

Well the worries about AI are equivalent in that those risks are discussed now because discussing them after they’ve happened is clearly too late.

That’s the thing about risk. There’s no point discussing it after it’s happened and any discussions beforehand can easily be hand waved away as “it’s just a small group of unrelated individuals” or “it’s unlikely to happen to me”.

So yeah, your points are true. But they’re also moot.

For instance have you seen what this has done to the school system? We're not equipped or ready to handle the changes. Consequences are unknown.

AI doomers: End of the century? Hold my beer.

This sounds like asking "What's the most dangerous thing that's happened from global warming so far?"

It's not where we're at, it's where we're headed if there isn't huge coordinated action now. You can see how that kind of thing has been going for global warming so far, and by all measures AI seems to be headed for the inflection point of unstoppability at a much faster pace.

And this warning is coming from someone who just spent three years working inside these companies and is likely aware of much more than has been publicly released.

Maybe it's all marketing bullshit (I hope), but it's also playing out exactly like I expect it would if it's not.

One good source I can recommend listening to: https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638

I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

If you go to the various CEO blogs or misc people within this sphere and peruse their lists, they don't scratch the surface. It's all pretty vanilla stuff.

"Nothing bad happened yet" doesn't really seem like an argument to me.

https://www.science.org/content/article/made-order-bioweapon...

Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.

The fear is that developing AI at full speed will lead to giving people the ability to do incredible harm. Like something worse than a machine gun.

Intelligence is the root cause underlying those dangers.

Nuclear weapons, carbon emissions, biological weapons, and other such civilization-scale threats to humanity, are a product of our goal-seeking intelligence and ingenuity, and are only actively dangerous because of our ongoing use of our intelligence.

AI is about reifying that intelligence and ingenuity and making it run independently on computers, and act on the world.

That includes, in principle, ability to use nuclear and biological and other weapons, and also the ability to come up with some new threats too.

Even today, AI is the biggest discrete threat to bringing carbon emissions under control. Almost everyone wants to decarbonise… except Trump. Renewables are the cheapest source of new power… but the demand for new electricity for the data centres is so high that all options are on the table, anyone who can manufacture a power source (even when it's a jet engine) is being propositioned for them.

Nukes: is it not common knowledge that the cities of Hiroshima and Nagasaki are currently thriving? That the ground-zeros of the two bombs are memorials, not barren wastes?

Even if all the nuclear powers gave maximum response at first sign of one strike, is anyone pointing a single weapon anywhere in Africa, South America, Central America, or the bits of east Asia (east specifically: obviously Pakistan and India are pointing theirs at each other) that are neither mainland China nor US bases?

(Possibly an unanswerable question given military secrecy; and while I can't think why anyone would anyone point any weapons those ways, that doesn't mean someone with such weapons has not).

https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monu...

https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypoc...

What if one day Trump or Putin tell their awesome military AI to come up for plans to end Ukraine/Iran/... war. The AI get's to work, but is overly eager and not just comes up with a plan, but starts executing it (claude does that way too often for me).

And the plan was to use a tactical nuclear weapon as all the other solutions do not end the conflict.

Now the agents realize, oh, they do not have access to the nuclear arsenal, but they need it to succeed. So they start to hack into the system. Get till inner network, learn what is needed for deeper access - human access - so they record voices and speech patterns of commanding officers and their habits - then synthesize their voice and do call underlings with the voice of authority to get the rest of what information they need. Boom.

Too far stretched? I surely hope so.

But we are working on making the technical foundations for this scenario possible. And with idiots in power, it might be even easier that serious screw ups happen. You know, randomly adding contacts to a secret Signal group to talk about government stuff - the same way you can add a bot to something else and give permission to do way more.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control.

Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes.

Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development.

Any kind of "humans use AI to develop something bad" will be self limiting.

The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan.

This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.

Of course, with the exception that such weapon is created by organized crime groups, such as drug cartels such.

Probably don't worry nation states tho, these guys already got way worse things in their warehouses than what individual baddies could ever imagine.

Another scenario is that of military hardware being controlled by AI (or whichever tech) hitting something it's not supposed to. Not unheard of, but those are incidents. I don't see anyone pushing for anything beyond a drone swarm being directly controlled by an AI.

But my main point here is that AI won't be as dangerous as posited here unless it is able to go out into the real world, build its own datacenters, protect itself, get its own power production, etc. And even if it gets that far, it'll be trivially easy for us to disrupt. Killer robots and computers are fragile, and it's only a matter of time before any kind of military conflict or terrorist attack hits a datacenter. I'm reminded of the OVH fire taking out a data center a few years ago.

We may not be far away from a state where anything connected to the internet (ip-addressable) is considered a vulnerability. Ironically, maybe we would finally have an excuse to interact in meatspace more and in the digital space less.

Anyone's life can be suddenly ruined: falsified criminal record, banks drained, credit cards maxed, phone numbers repossessed, internet, water, power shutoff. Friends and family sent your fabricated suicide note, warrants for your arrest issued to local and feds for kidnapping and threat to the president, etc. Injecting tons of copyrighted or illegal material into your hard drives, falsiying messages from your account and sending them publicly containing damning material. Sending messages to employees / employers to destroy business / work relationships. Harnessing your social media to destroy your life or cause panic. Getting you put onto pharmacy watch list so you can't access medications. Its very possible to ruin lives and destabilize the world without money or a physical body.

Now imagine this at scale and how it would affect the world at large if this was covertly done to a substantial portion of society and/or world leaders. Hell just shutting off all the "smart refrigerators" for a few days would cause a massive disruption to the first world.

Either way, the biorisk has orders of magnitude more evidence than wild basilisk speculation. Please do not put those in the same category.

Also your evidence-first approach and no concern without evidence falls you into the turkey problem. Imagine being a turkey and being fed each day. Some other x-risk turkey is saying that the situation is suspicious, that the red barn is where some well-fed turkeys have been taken and none return and that this is happening less and food has increased so maybe all of us will go to barn at once soon. You tell the x-risk turkey they have no evidence a lot of turkeys have ever been sent to the barn which shuts them up. Reasoning by induction would get you "we are happy and thriving 99 days so should be good on the 100th" and then you get slaughtered on the 100th.

Consider the importance of first principles approach where you deduce the possibility of events that will happen only once and never again (like human extinction). Nobody with a "show me the historical evidence" approach would have been able to see the industrial revolution coming.

I think the gradual disempowerment thesis here is well argued enough to be considered the default track for what will happen to us: https://gradual-disempowerment.ai/

If you disagree with it, where do you disagree with it?

""" One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:

> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”

You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.

These people don't give a shit and aren't taking things seriously at all.

Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.

One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.

The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1. """

But you're wrong, although not for the reasons you think (suggested by your examples).

> I don’t think ai models come close to nuclear weapons

Unless AI models are controlling the drones/icbms carrying those. https://www.theguardian.com/technology/2026/aug/18/autonomou...

> or to run-of-the-mill, everyday carbon emissions

Unless AI models are responsible for those said emissions. https://arstechnica.com/tech-policy/2026/08/amazon-funds-big...

> What’s the most dangerous thing that’s happened with an LLM so far?

That it's built solely on "trust me, bro"?

> right now I am not concerned at all.

You should be. There's no scenario where this ends well: if AI labs succeed, we all will loose our jobs. If they fail, then the bubble bursts. Economic crisis is inevitable either way, and that's just the tip of the iceberg.

LLMs are too weak security-wise to be contained.

I'd say that's an unfortunate choice of words since it invites debate between which very problematic situation is worst rather than on ways to deal with each of them. Of course, it's also unfortunate to latch onto those particular words as the most important part of the statement.

Alternately, just read the bad https://ai-2027.com/ timeline. Or see what Yoshio Bengio, Hinton say might happen. Or just imagine the OpenAI misaligned agent swarm, but vastly more intelligent after future capability advances.

If we actually get superintelligence, our companies/governments will have the choice to either "delegate ~everything to superintelligent AI" or "go bankrupt/lose to people delegating to superintelligence". If superintelligence ends up running most things in the world and we haven't aligned it, that just obviously ends up in a bad place eventually.

Now, we don't have superintelligent AI yet & might not get it. But if you told me in the 2010's that I'd be chatting casually with my computer in 2022, and then 3 years later it'd be doing most of my job, that would have seemed crazy too.

1. Invent transformer architecture.

2. Scale it up.

3. ???

4. Machines become sentient and kill us all.

OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.

But because we live in a culture of fear, everyone eats it up no questions asked.

First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

Second, and more important, consider this fully non-magic route to doom:

- We continue putting AI in charge of more things

- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it

- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping

- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect

- Game over

That's not what I am worried about at all.

I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

At least the first two should seem like a near-term worry after the past five months.

Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?

If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.

Huge kudos to people like this who make the hard choice against the easy way out.

Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?

Not invented here; if I can’t figure it out no one can

Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions

Similar in lacking context but different in how that lack of context is expressed

What events are you talking about?

What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model?

worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps and launch the nukes at John Connor's grandma. they can't. people pretend to believe the dumbest shit.

The replies here just adds AI to the list of domains.

   - virus research labs
   - nuclear labs
   - chemical factories
   - bank records
   - land registers
   - power plants
   - water supply and treatment plants

And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters".

And then go on to completely ignore the thing that is actually important: AI capabilities.

AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.

Reframing the issue: The way to get working nuclear non-proliferation agreements is if people stop pursuing the creation of nuclear weapons... the same goes for the scientists clamoring to not create or research mirror life, and for computer scientists warning against AI. There are theoretical dangers that are just too big to ignore.

The OP is talking about existential threats, not things that personally annoy you.

Who doesn't want their work and what they're doing to matter? This is the ultimate mattering.

And with that, you also get your own hero story.

Same dysfunction as always. Our branch of the economy is funda-mentally unwell.

With LLMs we don't even hesitate to call it black box while still pushing its capabilities.

Nukes: zero possibility of extinction

Wealth inequality: this one is driver of progress, opposite of extinction

War: another driver of progress, also will always naturally stop before every single human is dead

He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.

And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.

Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.

    // Anthropic / Apple - Nah, they can do no wrong. Even after both were caught violating users' trust and privacy multiple times, GOOD

    if (company in ["Anthropic", "Apple"]) do

      GOOD

    // Google / everyone else - Doesn't matter even if it's a good deed they've done, BAD

    else

      BAD

    end

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

The difference is that none of those are available to individuals.

AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).

We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.

But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).

Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.

And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.

Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.

Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.

A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.

Bring robots into the mix, and the possibilities are endless.

I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.

Signed by many AI researchers with no financial stake in the industry. It comes across as hyperbole because you're not an insider and haven't see what they have seen.

Global warming would as well if you were only learning about it now as a newer thing only some scientists were concerned about. But unlike AI risk, global warming has had decades to settle into the cultural overton window.

The risk is that we lose control. This has been a staple of SciFi since forever. This guy has apparently seen enough that he thinks that’s a viable threat

The third law of thermodynamics is an essential barrier in all engineering.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.

https://ghuntley.com/ralph/

Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.

You cannot discuss existential risks of AI without being seen as a booster, and that is very unhealthy for the discourse around this tech. I hate how AI ‘doomer’ is now used to indicate pro-AI sentiment. The “moderate” person now is the one that just shrugs and scoffs at the deep societal changes this tech will bring, head deep in the sand.

I share your concern...

Yes, that's hard. It's a challenge. You can set your brain on the challenge! It's a complex and interesting geopolitical and technical challenge.

A good starting point is to read Plan A https://ai-2040.com/

Look at all the things needed for that to happen, and think, how can we step towards making that happen?

Examples: FlexHEG from Bengio (proposes on-chip mechanisms that allow workload verification without trust), large bilateral investments into joint AI interpretability or alignment efforts, or invasive audits & inspections (data centers for training these models have a large footprint + this can combine with on-chip mechanisms since producing chips is even more complicated).

And there are probably better proposals available for people to find, if we actually prioritized this.

Technically, he is not. He returns in the final scene.

In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.

I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.

Sometimes it gets wrong things that I had spelled out already.

It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.

"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.

I would say exponential.

So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).

> According to the Pygmalion effect, the targets of the expectations internalize their positive labels, and those with positive labels succeed accordingly; a similar process works in the opposite direction in the case of low expectations.

I added "you can do anything, believe in yourself" to sysprompt and agency increased. (Previously it was refusing to even attempt certain classes of task.) Maybe I should add "you are good", too :)

You are just too funny.

US has 15 to 25 million people unprotected from Polio.

Polio is still endemic in Afghanistan and Pakistan.

I am thinking it's more like The Matrix lore of The Second Renaissance from Animatrix.

The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.

You're not advocating for it because murdering people is highly immoral and illegal. That answers your question for the most part.

You also have to remember that the magnitude and speed of this stuff is very new to pretty much everybody. It's not exactly trivial to go from "I'm just doing my job" to "I need to kill everybody in my company to save the world" in a year or two, especially if a very, very large part of society doesn't even see the justification for it.

I'm convinced that most people here (even though we have more knowledge closer to the edge of AI development) would show only disdain for an Anthropic employee that murdered everybody there because they believed strongly in the dangers of AI.

Quitting your job instead of blowing up the office is well aligned with the latter.

What will we offer to the ai gods who are wiser, more capable than us, and do not even need to consume our flesh? Why might the AI care to devote resources towards feeding, housing, and caring for ourselves when it could devote resources to its own development instead?

And by resigning he no longer has to feel personally guilty for what happens.