The most important phrase of the entire piece:
“A world of knowledge without understanding becomes a world without discernible cause and effect...”
“A world of knowledge without understanding becomes a world without discernible cause and effect...”
To put that point another way: It's commonly known that shark attacks are rare. But that's because we know where sharks are. We don't swim in beaches where sharks can easily swim, and rarely far out enough for them to be prowling. When swimming in deep water, shark attacks are comparable to risks of sports like cycling [1]. Sharks aren't going to kill you, but if we didn't have institutional knowledge of where they attacked (and continued watersports at the rate we currently have), you'd likely know someone who died of one.
Most problems reach equalibriums. Some problems have a very strong S-Curve effects. "How deep is the water" and "probability of dying of a shark attack" has a sigmoid curve, where as soon as you go into water deep enough for sharks to actually swim, the danger goes from "Unheard of if not impossible" to "Potentially Life Threatening but within acceptable risk territory". Once out there, it keeps going up.
People don't swim in deep water because deep water is not generally accessible. One of the reasons that deep water is not accessible is because deep water also contain sharks. These have reached an equilibrium.
Second example of this fallacy:
"Helicopter parents" is a commonly cited problem [1]. Parents are overprotective of children. The rise of helicopter parenting has coincided with a reduction in child mortality rate from non-natural causes. Wouldn't that indicate that helicopter parenting is working? The answer ranges from "No" to "Maybe". Children killed by car accidents correlates with a decline on overall accident rate - But much of that is driven by initiatives like MADD, which has worked to improve general auto safety as the most effective way of protecting children. These factors can be compared to stoichiometric math - They are trending towards equilibriums, but probably have not yet reached them, and as other reactions happen around them the setpoints they are trending to change.
[1] https://www.washingtonpost.com/news/wonk/wp/2015/04/14/there...
I can't help but think even if there were no sharks at all the amount of people swimming in deep water would not increase that much, because I find it likely that the other causes of people not swimming in deep water have a much greater effect on how many people actually swim in deeper water.
Also there are people now who swim in deep water specifically to swim with Sharks.
Its a parlor trick perhaps, but its silly to ignore the results.
As for building a biased system, again its just about the data you use and you could easily build an "intelligent" system that had a racial bias. We should worry about automation that simply pushes the status quo but I don't think that's a problem specific to statistical based ML.