The risk is that the naive and opportunistic application of decision-making, “judgement” and “ethics” embodied in AI systems will lead to the well-intentioned accidental regulation of important domains of human life, stifling those with relentless automatic gatekeepers and negative feedback loops which are pervasive, inscrutable, unaccountable, inimical to the fertile chaos of human negotiation and hostile to human social and political evolution. They will take human activity that is currently diverse and distributed and will force it through narrow channels controlled by self-appointed gatekeepers.

In an analogy to censorship, consider a world where “prior restraint” and “chilling effects” apply to every aspect in our lives, where deviations from the ideal are reliably detected and rejected, where we would not be able to earn credentials, get a job, find a partner, buy a home, evade surveillance, obtain necessary health care, or safely express a political opinion unless we either conform to or cheat the prevailing machine inference gatekeepers. What an insane additional cognitive burden this will be for everyone, everywhere, all the time.

This kind of AI is a mechanized, high-speed race to the bottom. And it will operate to discover undreamed-of bottoms to race to.

In the name of a few extra percent of efficiency we will doom our children to a rigid and lifeless dance with pattern matching algorithms.