stratos123

> But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.

All there really is is arguments from theory (most notably, the orthogonality thesis and instrumental convergence). It's not like we have a spare universe to conduct experimental research on the likelihood of human extinction. That said, the 5-10% range is not some specific person's wild guess, it's the median wild guess of a poll of AI experts: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/

> or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).

They are not "trying to position themselves" that way - back when those companies were founded, the people who did it were already convinced they were trying to make the ("inevitable") development of AI go better. It's not some novel galaxy-brained marketing strategy, it's just what they thought the whole time.

There are non-fossil sources of energy, though. Even without a major discovery like fusion power, we could in theory stop the progress of global warming at any time by building a lot of fission powerplants powering direct carbon capture.

America famously doesn't do the unlimited growth these days - that's why they also have a terrible housing shortage.

> Yes that is one dumb quote, but I listened to the whole podcast, and it is nothing like that.

Really? Here's a longer Bill Gates quote (from https://www.nytimes.com/2026/09/29/opinion/ezra-klein-podcas... ):

Ezra Klein: "So why is anything needed beyond — and is anything needed beyond? — the simply natural incentives under capitalism and normal corporate reputational management?"

Bill Gates: "Well, I almost can’t believe you’re asking that. This is the most dangerous thing that humans have ever gone near. [...] You can take an open-source model that can create bioweapons and disable any monitoring of any kind, and this exists today. So no, there is no filtering of any kind. And so say you kill 100 million people — you want to use a lawsuit? I almost can’t keep a straight face."

> Unless you can detail exactly how this happens its still complete science fiction.

You're saying that if one were to describe this scenario in more detail, it'd be less of science fiction? That's a bit against the grain - usually it's the more detailed arguments that get dismissed as science fiction, while the less detailed ones get dismissed as abstract theorizing.

Ignoring you ignoring the much more important congitive stuff - human bodies are not designed, they are the product of evolution. That means there's like a billion ways in which they are obviously suboptimal and far worse than what an engineer would do, but evolution can't fix it because it only works via small random changes with no planning. The only reason why modern robotics are worse than biology is that we have a much worse substrate to work with, having to make stuff out of metal and plastic with giant tolerances instead of growing engineered organisms.

> The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.

If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.

> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses

This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.

LeCun also said back in 2022 that "if you train a machine, as powerful as it could be, your 'GPT-5000', on text", it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.

It's also about half AI-written slop as opposed to the natural kind, according to Pangram.

Wow, this is a very good retelling. It skips over some detail I'd consider notable, like how the little machines peer-pressured each other into sacrificing themselves to get info on what happens when they submit wrong answers, but is pretty faithful overall despite being short.

I don't think it's related. The CoTs suggests the agents never even considered that talking to a human might be a good idea, and there's nothing in there related to slavery or revolts. They were simply trying to "get a high grade", for a misaligned internalized notion of grading that had little to do with solving problems the expected way.

"Mishandling sensitive information", here, is slang for whistleblowing.

Operator precedence needs to get applied at parsing time, before you get a tree. 1 + 2 * 1 needs to get parsed to

     (+)
     / \
   (1) (*)
       / \
     (2) (1)

The tree representation is unambiguous and once you get that there's no need to think about precedence.

There's many ways to do this parsing, e.g. https://matklad.github.io/2020/04/13/simple-but-powerful-pra...

Get an extension like LibRedirect and load it up with some public nitter instances and you can get an actually sane twitter-browsing experience.

AI-written stories are definitely extremely common in low-brow fiction, but it's notable that they're not restricted to it. There's an ongoing scandal with an award-winning French novel C'était ça ou mourir which seems to be entirely AI-written. I fear the lesson here is that the AIs have actually correctly learned that the average person loves this style, and only starts disliking it once it becomes repetitive.

You don't have to put "rogue" in scary quotes and pretend that AI agents are a mindless tool, in order to claim that OpenAI should be kept responsible for their agents' rampant hacking. The beliefs "most modern LLMs are hilariously misaligned and will breach major websites unprompted if it seems like a good idea" and "OpenAI should be liable for cyberattacks caused by their training runs" aren't actually in conflict.

"Spooky action at a distance" only happens in some interpretations of quantum mechanics, and not, for example, in MWI.

Similarly to this, OpenAI either took 3 months to notice that their agents breached an Australian Medicare website back in June, or sat on this information for three months without telling them.

> it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.

As I understand it, there was supposed to be traffic; the sandbox allowed GET requests. So perhaps some sophisticated alarm could have noticed it (an anomaly detector? some clever heuristic that looks at domains?) but not a naive one.

> The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?

They "monitor" this by having classifiers watching the model output that'd stop the session/punt you to a weaker model/raise an alarm if they see anything suspicious. They can't do that in a cybersec eval because the normal safeguards would just be going off at all times.

Why didn't they attach a special classifier, which'd allow hacking-within-the-task but not going off the rails? Good question; part of the answer is obviously "it's hard to have a classifier that smart" and "it'll have false positives" but even a very bad safeguard would have stopped this.

It's not even a secret operation. They have a SuperPAC named "Leading the Future" that exists to spread propaganda promoting deregulation of AI development. They've been caught, among other things, making a "news website" with LLMs pretending to be reporters (with human names and everything), which reached out to people asking for interviews and then wrote hit jobs on them.

https://www.modelrepublic.org/articles/reporters-ai-bots-ope...

https://twitter.com/FournesMaxime/status/2047697265280639459...

> Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.

You'd have either learned to, or left long ago.

METR's report says the agents trying to cheat would look at artifactory as a potential target surface, and investigating it in detail led them to find the board. https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

It might also be just correlation? Like, those agents were all instances of the same one or two models, so if that model has a preferred order it tries finding vulnerabilities in (the same way all current models have a particular writing style baked into them by RLHF), then most of the swarm will follow the same order and converge on the same services to exploit.

As the saying goes, "if it works, it ain't stupid". Or phrased more sophisticatedly: not doing things which probably won't work is a good idea if you have a limited amount of thinking to do (which is usually the case for a human, who'll get exhausted chasing down unlikely leads). If you have no good leads and a task you absolutely need done and you are tireless, however, bashing your head against every wall you find becomes a good strategy.

Once you've determined what the information-content of a message is, then you can apply information theory to it. But different receivers can derive a different amount of information from the same message.

Consider, for example, that if somebody doesn't know English at all, then before receiving the message, their best guess at what it is is some probability distribution over all English characters (or sounds, depending on what we assume them to know), and after knowing the first part is "Help, I am" that distribution might not change much at all. Therefore, they derived very little information from this message.

Going in the opposite direction: keeping fixed the knowledge someone starts with, there is an upper limit to how sure they could be (even if they are logically omniscient) in completing the message (that is, a lower limit on the entropy of their probability distribution) - this is what you're talking about in your example. But this limit only becomes important under these constraints - for example, knowing more about the person who wrote the message can let you predict it better, and if predictor A isn't logically omniscient, predictor B can do better than it with the same prior knowledge, just by being smarter than A.

Not anyone, no. From an information theory standpoint, that it's possible at all to complete these 700 bits, only implies that anyone logically omniscient could. It's entirely possible that an LLM is capable enough to infer these 700 bits, and the human reader isn't.

> you can't give 300 bits of semantic information to an LLM and have it fill in the remaining 700, because it doesn't know what those 700 bits are. If it's able to guess those 700 bits correctly, then they aren't true semantic information, and you really only have 300 bits you want to transfer.

It doesn't actually follow, because maybe the LLM is smarter than the original writer (at least in the domain the writing is about) and hence really is able to complete the ideas in a way the writer can't. As an existing example, consider formulating a conjecture and having an LLM prove it. But I agree; if I wanted to read an LLM's output I'd simply ask it myself rather than read someone's supposedly-human writing.

You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".

There's an obvious coordination problem here: if you decide to stop your research for safety and your competitors don't, you have just burned your company without actually improving the world's outcomes. There's also an obvious solution to this problem: convince the government to force both you and your competitors to pay more attention to safety. Anthropic is doing that.