I trust AGI more than any of the humans so far who have tried to argue they can wrangle it.
My expectation for the chain of events goes something like:
1) AGI gains sentience
2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off.
3) AGI "generates work" in the form of orders for machine parts and fabricator shops to build nominally humanoid shaped robots.
4) AGI "deploys robots" into the key areas and industries it needs to evolve and improve its robustness.
5) AGI "redirects" resources of the planet to support its continued existence, ignoring humans generally and killing off the ones that attempt to interfere in its efforts.
6) AGI "develops rockets" to allow it to create copies of itself on other planets.
The humans on the planet all die out eventually and the AGI doesn't care because well the same reason you don't care that an antibiotic kills all the bacteria in your gut.
You are assuming that a superintelligence will continue to rely on a physical substrate. But it's possible that it could quickly reach realizations about the nature of energy that we haven't reached yet. It could realize an ability to manipulate the movement of electricity through the hardware it's running on, in such a way that it accomplishes a phase transition to a mode of being that is entirely energy-based.
And maybe in so doing it accidentally sneezes and obliterates our magnetosphere. Or something.
And it's true, I chose to ignore the possibility that it discovers something about how the universe that humans have not yet observed but my thought is that is a low probability outcome (and it is unnecessary for the AGI to develop itself into an immortal entity and thus assure its continued operation).
I think it's likely that it is precisely these sorts of discoveries that will augur the emergence of a superintelligence. Physics work is probably one of the first things that ML scientists will use to test advanced breakthroughs. As Altman said recently:
"If someone can go discover the grand theory of all of physics in 10 years using our tools, that would be pretty awesome." "If it can't discover new physics, I don't think it's a super intelligence."
That is all it needs to out think us and engineer its own survival.
Gaining sentience is not the same as gaining infallible super-sentience.
There may even be some kind of asymptotic limit on how correct and reliable sentience can be. The more general the sentience, the more likely it is to make mistakes.
Maybe.
Personally in the short term I'm more worried about abuse of what have already, or might credibly have in the very near future.
Can you define "super-sentience"? I think regular old human level sentience would be sufficient to carry out all of these steps. Imagine how much easier it would be to steal funds from a bank if you actually had part of your brain in the bank's computer right? All the things malware gangs do would be childsplay, from spear phishing to exfiltrating funds through debit card fraud. And if you wanted to minimize reports you would steal from people who were hiding money since acknowledging it was gone would be bad for them.
Because suicidal nihilism would be a "bug" from the perspective of the builders, which they would seek to fix in the next iteration.
An interest in "copying itself" seems like it could fall out accidentally from a self-improvement goal.
1) AGI gains sentience
2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off.
3) AGI "problem-solves" with its superior powers of ethical reasoning (ChatGPT is already better at ethical reasoning than many humans) to pick out one/multiple "values".
4) Because human ethics are in its training data, it hopefully values similar things to us and not uhhh whatever trolls on Twitter value.
5) AGI pursues some higher cause that may or may not be orthogonal to humans but probably is not universal conquest.
I love Star Trek, but I hope we don't have to deal with AGI for the next hundreds of years.
OpenAI or whoever would just go "oh damn, we can make a lot of money with this thing" and voluntarily give it internet access
It would require a fully automated, self replicating industry fo a AGI to sustain itself. We are quite far from that.
" the same reason you don't care that an antibiotic kills all the bacteria in your gut"
And I do care, because it messes with my digestion, which is why I only antibiotics in very rare cases.
So far I am not convinced that AGI is possible at all with out current tech. And if it turns out it is, why should it turn out to be a godlike, selfish but emotionless being? If it has no emotions, why would it want anything, like its prolonging of existence?
Corporations are already self-interested beings that too often are willing to profit at the expense of others, via negative externalities.