“It’s tough to build if you think you’re building a weapon. Especially if you thought you were going to be making helpful tools.”

Every tool is a weapon if you hold it right.

Humanity is a superorganism. Thanks to the internet, it’s more connected than it’s ever been, and can “think” and change faster than it’s ever been able to before. Facebook is trying to build a neutral platform and useful tool for communication & socialising. Because of that neutrality, it has no inbuilt “immune system” for preventing the superorganism from being infected by memes created by individual humans. Quite the opposite: it rewards memes and viral popularity because the humans who run its “brain” can extract value from that effect. Most of these memes are benign (jokes and entertainment, advertisers trying to sell more widgets and widget-related services) but some are malignant/parasitic (e.g. influencing political outcomes for personal gain).

Any sufficiently large social network is going to have to contend with this and make the hard choice: do you remain neutral and let humans influence humanity, or do you try to prevent “social infection” when it’s deemed detrimental? But what are the criteria for intervention—sharing false information, for example? Who has the power to dictate what’s “true” or whether to intervene? Facebook? Governments? A benevolent AI—developed by whom? &c.