Silicon Valley Is Turning into Its Own Worst Fear (2017)

"Speaking to Maureen Dowd for a Vanity Fair article published in April, Musk gave an example of an artificial intelligence that’s given the task of picking strawberries. It seems harmless enough, but as the AI redesigns itself to be more effective, it might decide that the best way to maximize its output would be to destroy civilization and convert the entire surface of the Earth into strawberry fields. Thus, in its pursuit of a seemingly innocuous goal, an AI could bring about the extinction of humanity purely as an unintended side effect.

This scenario sounds absurd to most people, yet there are a surprising number of technologists who think it illustrates a real danger. Why? Perhaps it’s because they’re already accustomed to entities that operate this way: Silicon Valley tech companies.

'Consider: Who pursues their goals with monomaniacal focus, oblivious to the possibility of negative consequences? Who adopts a scorched-earth approach to increasing market share? This hypothetical strawberry-picking AI does what every tech startup wishes it could do — grows at an exponential rate and destroys its competitors until it’s achieved an absolute monopoly.'"

Somehow that quote had way more resonance when read within 24 hours of reading this (warning: 85-pages!) detailed critique of our FAANG overlords and their ability to handle tasks of forum moderation or plain old customer support: https://danluu.com/diseconomies-scale/

~ Senator Palpatine / Darth Sidious

Of course the SV companies will present themselves as the underdogs that you should root for when they are, you know, the underdogs. And of course they'll do anything and everything to cement every inch of power they gained after they start gaining it.

If anybody has been surprised by this then they are despairingly naive.