I think this assumption is worth at least casually testing.

Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from?

Presumably being super-intelligent the AI would be:

1) likely to anticipate being turned off,

2) super-normally capable to identify & traverse any network security measures;

3) potentially also highly capable to identify & traverse weaknesses in other (eg. social or physical) security measures).

4) possibly able to be covert/ stealthy actions via the network.

5) possibly able, if able to reform on the Internet, to escape to there.

I'm fairly much on the super-AI fence. But in a dystopian scenario the proposal of a super AI being able to be turned off as the key means of safety, seems scarcely credible.

Note even that without being able to reform on the Internet (eg physical embodiment requires singularly specialized computing equipment), I can't logically see that extreme hostile actions (destroy global finance system, subversive propaganda campaigns, launch wars, launch nukes) are obviously easy to preclude.

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.