Can you give concrete examples of "something impossible happens based on known physics"? I have followed the AI debate for a long time but I can't think of what those might be.
The annoying property of prohibitively exponential (ignoring geometric) space complexity is that it places a severe bound on computational complexity per unit time. The exponentially increasing space implies an increase in latency for each sequentially dependent operation, bounded at the limit by the speed of light. Even if you can afford the insane space requirements, your computation can’t afford the aggregate latency for anything useful even for the most trivial problems. With highly parallel architectures this can be turned into a latency-hiding problem to some extent but this also has limits.
This was thoroughly studied by the US defense community decades ago.
The tl;dr is that efficient learning scales extremely poorly, more poorly than I think people intuit. All of the super-intelligence hard-takeoff scenarios? Not going to happen, you can’t make the physics work without positing magic that circumvents the reality of latencies when your state space is unfathomably large even with unimaginably efficient computers.
I harbor a suspicion that the cost of this scaling problem, and the limitations of wetware, has bounded intelligence in biological systems. We can probably do better in silicon than wetware in some important ways but there is not enough intrinsic parallelism in the computation to adequately hide the latency.
Personally, I find these “fundamental limits of computation” things to be extremely fascinating.
You studied ML before discovery of "double descent"?
EY’s assertions regarding a fast “FOOM” have been empirically discredited by the very fact that ChatGPT was created in 2022, it is now 2025, and we still exist. But goal posts are moved. Even ignoring that error, the logic is based on, essentially, “AI is a magic box that can solve any problem by thought alone.” If you can define a problem, the AI can solve it. This is part of the analysis done by AI x-risk people of the MIRI tradition. Which ignores entirely that there are very many problems (including AI recursive improvement itself) which are computationally infeasible to solve in this way, no matter how “smart” you are.
The FOOM argument starts with some kind of goal-directed agent (that escapes and then it) starts building a more capable version of itself (and then goal drift might set in might not)
If you tell ChatGPT to build ChatGPT++ and leave currently there's no time horizon within it would accomplish either that or escape, or anything, because now it gives you tokens rendered on some website.
The argument is not that AI is a magic box.
- The argument is that if there's a process that improves AI. [1]
- And if during that process AI becomes so capable that it can materially contribute to the process, and eventually continue (un)supervised. [2]
- Then eventually it'll escape and do whatever it wants, and then eventually the smallest misalignment means we become expendable resources.
I think the argument might be valid logically, but the constant factors are very important to the actual meaning and obviously we don't know them. (But the upper and lower estimates are far. Hence the whole debate.)
[1] Look around, we have a process that's like that. However gamed and flawed we have METR scores and ARC-AGI benchmarks, and thousands of really determined and skillful people working on it, hundreds of billions of capital deployed to keep this process going.
[2] We are not there yet, but decades after peak oil arguments we are very good at drawing various hockey stick curves.
(2) It is quite trivial to Ralph Wiggam improvements to agentic tools. Fetch the source code to Claude Code (it's minimized, but that never stopped Claude) or Codex into a directory, then run it in a loop with the prompt "You are an AI tool running from the code in the current directory. Every time you finish, you are relaunched, acquiring any code updates that you wrote in the last session. Do whatever changes are necessary for you to grow smarter and more capable."
Will that work? Hell no, of course it won't. But here's the thing: Yudkowsky et al predicted that it would. Their whole doomer if-you-build-it-everybody-dies argument is predicated on this: that take-off speeds would be lightning fast, as a consequence of exponentials with a radically compressed doubling time. It's why EY had a total public meltdown in 2022 after visiting some of the AI labs half a year before the release of ChatGPT. He didn't even think we would survive past the end of the year.
Neither EY nor Bostrom, nor anyone in their circle are engineers. They don't build things. They don't understand the immense difficulty of getting something to work right the first time, nor how incredibly difficult it is to keep entropy at bay in dynamical systems. When they set out to model intelligence explosions, they assumed smooth exponentials and no noise floor. They argued that the very first agent capable of editing its own source code as good as the worst AI researchers, would quickly bootstrap itself into superintelligence. The debate was whether it would take hours or days. This is all in the LessWrong archives. You can go find the old debates, if you're interested.
To my knowledge, they have never updated their beliefs or arguments since 2022. We are now 3 years past the bar they set for the end of the world, and things seem to be going ok. I mean, there's lots of problems with job layoffs, AI used to manipulate elections, and slop everywhere you look. But Skynet didn't engineer a bioweapon or gray goo to wipe out humanity - which is literally what they argued would be happening two years ago!
but saying that "since it was possible already many years ago and we are still alive" the whole argument is false doesn't stand up to scrutiny, because the argument doesn't make any claim on speed nor does it depend on it. quite the opposite, it depends on accumulating infinitesimal gains (of intelligence and misalignment).
FOOM (fast take off, intelligence explosion either through deception or by someone asking for a bit too many paperclips) is simply one scenario. also notice that even this (or any sudden loss of control) doesn't depend on the timimgs between first self-improving agent, FOOM, and then death.
like I said the hypothesis is pretty coherent logically (though obviously not a tautology), but the constant factors are pretty important (duh!)
... I think spending time on the LessWrong debates is a waste of time because by 2022 the neuroticism took over and there were no real answers to challenges
This is flat-out false. I will send $50 (in the form of a personal check or bitcoin) to the first person that can link to anything written or said by Yudkowsky since 2021 stating or even just implying that humanity definitely or probably won't survive past the end of the year.
He has said that he can't entirely rule out the possibility that the end will come today or tomorrow, but the context suggests that he thought the probability was very small, and even if we ignore the context, a reasonable person will surely concede that that is a much different level of probability.