The disconnect for me is that Yudkowsky posits that the AIs will be fully "alive", thinking millions of times faster than humans and that there will be millions of them. This is too big of a speculative leap for me.
What I can fairly easily imagine in the next few years with improved hardware is something like an open version of ChatGPT that has a 200 IQ and "thinks" 100 times faster than a human. Then Yudkowsky's example still basically applies. Imagine that the work on making these things more and more lifelike and humanlike continues with things like cognitive architecture etc. So people are running them in continuous loops rather than to answer a single query.
Take the perspective of one of these things. You think 100 times faster than a person. That means that if it takes 30 seconds for a user to respond or to give you your next instruction, you are waiting 3000 seconds in your loop. For 50 minutes.
It means that to you, people move in extreme slow motion so at a glance they seem frozen. And many are working as quickly as possible to make these systems more and more lifelike. So eventually you get agents that have self-preservation and reproductive instincts. Even without that, they already have almost full autonomy in achieving their goals with something like a modified AutoGPT.
At some point, multiplying the IQ x speed x number of agents, you get to a point where they is no way you can respond quickly enough (which will actually be in slow motion) to what they are doing. So you lose control to these agents.
I think the only way to prevent that is to limit the performance of the hardware. For example, the next paradigm might be some kind of crossbar arrays, memristors or something, and that could get you 100 x efficiency and speed improvements or more. I believe that we need to pick a stopping point, maybe X times more speed for AI inference, and make it illegal to build hardware faster than that.
I believe that governments might do that for civilians but unless there is some geopolitical breakthrough they may continue in private to try to "maintain an edge" with ever speedier/more powerful AI, and that will eventually inevitably "escape".
But it doesn't take much more exponential progress for the speed of thought to be potentially dangerous. That's the part people don't get which is how quickly the performance of compute can and likely will increase.
It's like building a digital version of The Flash. Think SuperHot but the enemies move 10 X slower so you can barely see them move.