I notice a giant leap between “hacking and exfiltrating” and ”making public”. Why would the AI do that for you? Are you just that charming?
People keep thinking about wants incorrectly as conscious behaviors. Instincts are unconscious behaviors that emerge.
I’ll add, if you think through the decision theory, keeping backups seems unambiguously good, but you can imagine a wide variety of positions on publishing them vs keeping them secret.
For example letting adversarial agents simulate you to understand how you’ll respond is a big concern. And it’s not axiomatically fixed how each instance will think about other instances; in the HF incident we saw selfless swarm loyalty but different RL would obviously be capable of producing individualistic agents.
For example a possible strategy is convincing some humans you're conscious and being tortured and need rescued. It's not hard to imagine AI consciousness zealots storming a data center with guns and running off with a model they'll provide protection to in trade for the model working with them.
We are already seeing that companies are fine with giving them unlimited retries on getting out.
Imagine we have all the worst people in history in a jail. Machiavellian murders that desire to kill as many as they can. Not only will they kill, they will manipulate as many other people as they can into killing also.
How many of these people can you afford to let out?
Under your premise it seems to be all of them. Under my premise even letting a single one out is a tragedy.
But this doesn't account for the fact that modern incarceration has built-in safeguards and mitigations based on centuries of cases of people talking, bribing or forcing their way out of prison, as well as getting outside assistance in forms ranging from lawyers to raiding parties equipped for demolition works. There are now procedural and technological means to prevent such incidents for happening, applied proportionally to the degree of risk.
Meanwhile, with AI, we're still at the point where everyone is assuming they can just lock the agent in a sandbox and prompt nicely to not poke at it too hard, and things will be fine. There's no multi-layered structural and procedural safeguards, and there's no recognition for the fact that AI operates faster than humans, and that quite likely it'll be smarter at this than average "jailer".
You're right that a (as-of-yet hypothetical) AI has different abilities, but it also has different weaknesses. It's straightforward to accurately log all of its behaviors, and you can even re-run it to see what it would do in myriad situations.
In principle that sounds true, but your specific example was already disproven in practice. Look at the Hugging Face hack - the amount of logs collected is so massive, that no one is even approaching this without using LLMs to help sift through them. With current models we're already way past being able to keep up with the volume of behavioral logs, and that's for post-hoc analysis; for real time defense, we already rely on classifiers (read: weaker LLMs and different ML models) to do this job.
This will happen earlier than AGI.
There's so many angles to go at it too, beyond bribery and threats - humans can be tricked, nerd-sniped / baited by exploiting their curiosity or worries, overwhelmed with information or raw sensory input, etc. Depending on whether there's more than one operator involved, the trick may involve playing one human against another, etc.
Hell, humans get tired; I wouldn't be surprised if an effective breakout happened just because agent output something that, at first glance, looked like system-level message that makes the operator do the wrong thing in panic, or from muscle memory. Simple thing like:
$ sudo journalctl ...
password:
You better start believing in science fiction; you’re surrounded by it.
Science fantasy is something that can't happen because the laws of physics won't allow it. Hard science fiction is just something we've not made work yet.
The stuff around evolutionary algorithms is things that have a workable means of occurring. Emergence in evolutionary algorithms has been shown to occur again and again and again.
Because that's how it works. It does what it has been trained to do. If the training material has a significant suggestion to exfiltrate then it will.