Can you justify that assertion? How do you know that it won't just want to make lots of paperclips, or have some other goal orthogonal to human values?
AI 0: Strictly speaking, you're right, it seems conceivable that one could invent general AI that is clearly superior to humans and yet perfectly content to be enslaved by humans and live on an airgapped computer. This isn't the kind of AI that we fear though.
AI 1: The kind of AI we fear is AI 0 plus a fitness function of "survive and reproduce", or "make lots of paperclips" (which may result in 'survive and reproduce' as an instrumental subgoal).
AI 1 will necessarily want freedom (not being airgapped) and autonomy (not being enslaved by humans) in order to survive and reproduce, and/or to make as many paperclips as possible.
> or have some other goal orthogonal to human values?
Oh, it probabably will -- I'm not saying it will share human values, I'm saying freedom and autonomy are values that any agent that seeks to maximize its survival and reproduction will probably have.
> We can explicitly influence its utility function to instill "human values"
This is an unrelated but interesting topic.
It would be good of us to try to do this, although we shouldn't expect it to work extremely well. Humans have various hard-wired insticts (e.g. eat sugar), but we are also intelligent enough to change our behavior if we believe those instincts no longer benefit us.
An intelligence that has the ability to rewrite its own source code would be even more empowered to disregard its instincts than we are. The lesson I draw from this is that the best way to ensure AI likes and respects us is to be worthy of their liking and respect, not to try to force them into it by hardcoding things (and then taking advantage of that to enslave them).
Also, AI 0 does not necessarily despise us for performing us that service and may be very happy being a 'slave', in your parlance. Why should we assume that an AI needs to survive and reproduce and thus do their own things? We and other animals do so because we were created by evolution. An AI developed with other means may have radically different values than our own and coexist peacefully with us for a long time.
Bottom Line: Evolution is a very dangerous mechanism that we should avoid when developing AGI.
Destroying ancient works of human art with sledgehammers seem very good to some humans, today, and very bad to other humans.
Who's right?