"We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals."
"Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."
Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act.
It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.
A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed.
If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.
There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
The question is, which assumptions do we think are warranted and why?
I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on?
What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable.
The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.
Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute.
What does this AI do, then?
Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995.
So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.