Well, here's the thing. Even the worst villains of history had human values and feelings: In other words, alignment. A superoptimizer AI might have the ability to wipe out the whole human species, in a way we won't be able to understand in time to prevent it, and all for an instrumental goal incidental to whatever it's doing.

(In a way, this thread is a data point for why we need a more sophisticated debate about AI.)