How do you make an AGI smart enough to be helpful but not so smart that it gets cynical and figures "what's the point of helping these dummies anyway?"
Kind of the Ayn Rand problem...
Kind of the Ayn Rand problem...
And if that argument doesn't interest you because of AGI's massive utility, then how about this: If Superman were born tomorrow, and we put implants in his brain that made him really, really love America, apple pie, and basically do whatever we want, how would you feel?
Most people, I think, want a deeply species-ist AI.
It has to want to help humans. Not exclude us from planets to make room for uplifting dolphins or defect by spawning more machine intelligences.
I think that condition is hard to meet while also counting future-humans who might look more like aliens or software than the humans of today.
I believe the rational solution would be a benevolent AI that values all life, the problem being that it wouldn't necessarily always work in "our" interests.