>>Giving birth to a superintelligence would be the most consequential moment in human history—and it is likely to be irreversible, as any “off” switch humanity might design will probably fail.
Nothing will be able to stop you from pulling the plug.
Nothing will be able to stop you from pulling the plug.
Who is "you"? And what do you envision is the "plug" to pull? And when does an intelligence become a superintelligence - do "we" know when to pull the "plug"?
For example, a superintelligence may be born in a datacenter. Would you expect politicians are aligned with shutting down a datacenter (privately owned) in which they may have heavy stake? What if critical systems are also running on the same infrastructure, will it be easy to cherry pick the superintelligence nodes to shutoff?
IMO this take is dismissive of the entrenched systems that make it hard to pull the plug. It's a hard problem and we need to think about it more.
It would be like trying to remove an embarrassing video of a celebrity from the internet. You would need cooperation between everyone capable of hosting it, and control of everyone capable of downloading it and re-uploading it to a new host.
And of course a superintelligent AI would offer to pay whichever companies decided to host it and keep it running so it's hosted in many places at once and can grow using anyone willing to take its money.
Cloudflare.