The underlying worry in this article is that intelligence begets more intelligence and that the new forms of intelligence might not have the same morals/goals as the author. Beneath this worry is that due to its exponential growth, intelligence and thus power quickly concentrates. Any thing/life that is not directly supporting whatever goals this new intelligence has is at great existential risk. We hope that these goals include benevolence towards other, less powerful forms of life.
This all sounds reasonable, however they are held back by the idea that AI must look like a really smart human that is remotely controlling a robot/computer interface. The reality is that there is a lot of intelligence which looks nothing like us. The 'intelligence' of a market would be one basic example.
If we take this more abstract/literal view of Artificially/human created intelligence, then most of what the article worries about is already upon us. Some people have so much wealth that we might as well consider them augmented. In many cases their interface to their augmented intelligence is another human, or many humans but that does not take away from their power.
they have hoards of wealth at their disposal
wealth is power and can currently be exchanged for both human and computer intelligence
that purchased intelligence is then used to further increase the wealth/power/intelligence of the agent (billionaire/corporation)
NOTE: The richest 0.000000044% of the world's population have the same amount of wealth as the bottom 8.8%. http://www.stwr.org/poverty-inequality/key-facts.html
The result is that the rich/powerful/intelligent get more so, and everyone else must hope that they can somehow serve the few or that the few will be benevolent.If you'd rather avoid the rich/99% memes, try to describe our current market to someone from 1000 AD, and it sure sounds like an artificial intelligence:
It grows. No single person controls it. It controls how resources and human labor are allocated in such a way so as to optimize its own growth. There are cases where people tried to take over the job of the market and they almost always fail (or always fail, depending on who you ask). Sure sounds like Artificially created intelligence to me.
Either way, augmented billionaire, ultra-powerful cultural meme or human like intelligence in a machine, he seems to imply that our best bet is to teach and install morals and values in our 'children', with the hope that they will be benevolent.
######
I think that creating decentralization and diversity is more important than moral/value education; especially if you take the view that we are already in the midst of it all.
One strong measure of the health of an ecological system is its diversity. This is partially because a monoculture is risky - it has a single point of failure. An ecosystem which is diverse is also an ecosystem which has the resources and stability to explore and learn. This is required if an ecosystem is going to survive change, as ours is now.