Humans will use AI to destroy each other long before AI uses itself to destroy humans.
If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.
It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.
(Think of some of the most intense suffering we're capable of; it doesn't all depend on higher cognition, some of it is very base and physical.)
First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers.
Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips.
[1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...
Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth.
Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
Lots of humans have pretty despicable goals, including some very intelligent ones.
I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons.
Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.
The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps.
Interesting take on the AI takes over the world theme.