Mrs. Davis[0] offers a pretty compelling vision of this - absurd but not wholly unbelievable. The machine doesn’t necessarily have to supply the physical threat if it has humans who happily do its bidding.
I never said it is; my point was that to someone hubristic it is. That's why I said 'seems to'. An AGI wouldn't have a superego, so by your definition, it could never understand morals, and it wouldn't be surprising if it discarded them or tried to argue its way out of being subject to them. That's what the snake did in the story of the garden of Eden.
An AGI wouldn’t have a superego, but it would have the ability to study data across really long timescales and apply itself to long term strategies that would supersede the need for a superego. Since we apply morality as a shorthand for describing agreed upon negative consequences for complex behaviors, we can assume that an AI can overcome this need by simply calculating out the full consequences of complex behaviors without the need for any shorthand. And we can assume that AI can even iterate on and develop on its own understanding of these complex calculations much faster than a human can, making it significantly better at navigating moral issues.
We’ve had plenty of ideas that were later found out to be wrong (if not more.) I’d be just as afraid of an AI that was bound to our understanding of reality as I would be of one that wasn’t. For example, if a “moral” AI was coded 100 years ago, it would enforce the belief that homosexuality is immoral.
Yeah, like judging human history and deciding murderous humans need to be removed from the earth or enslaved for their own benefit. (Some people already argue that the world would be better off without humans.)
Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
>Also, we know torture is wrong because we have a sense of pain, but an AI would just have to take our word for it. That doesn't seem too balanced to me.
Again, you have this innate belief that immorality is somehow strategically beneficial and IMO you should really look into that
Now, you might say that AGI will be able to perfectly predict the future of all actions and make the legal system redundant. Ignoring the absurd amount of possibilities, I struggle to believe it's possible because an AGI has no sense of pain to give actions weight; it just has to follow human precedent. Seeming good enough, it could steer humanity into a dead end unknowingly. Also, it might decide that a few eggs have to be broken to make a utopian omelette.