That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating.
Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology.
If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk.
Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous.
Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.
One more thing to add is that as the AI performance increases and surpasses human decision-making ability, the hyperspeed will push competiting companies and countries to give them broader and broader goals and more autonomy. Because waiting overnight for human input means your competitor's AIs race ahead the equivalent of weeks.
I think we need to set a limit for the performance of new hardware at some point. Also we need people to understand the dangers of full autonomy and imitating animals in the context of hyperspeed or superintelligent AI. We will need criminal penalties and maybe some type of cooperative digital immune system.