The coming war on the hidden algorithms that trap people in poverty

> He realized that nurses were telling patients, “Well, the computer did it—it’s not me.” “That’s what tipped us off,” he says.

This seems to be the real crux of the issue. These industrial-scale algorithms should be treated the same as any other industrial equipment. If you run over someone with a bulldozer, you can't say "the bulldozer did it, not me." Using the excuse "I don't even know how to operate a bulldozer" only makes it worse. I think it's perfectly reasonable that if an institution doesn't train people to properly use an algorithm, take reasonable precautions to make sure it doesn't cause severe damage, and/or has no reasonable system in place for fixing problems it creates, they should be liable for negligence the same way any other operator of potentially dangerous equipment would.

The economy has become impersonal. In the past, there was one county bank, one regional college, and usually one large employer in town. If someone never paid their debts, the bank manager wouldn't lend them any more. If they didn't do well in school, the dean wouldn't admit them to the college. If they didn't show up for work, the foreman wouldn't hire them back. Did people accuse these decision makers of "keeping" unreliable people poor? No. Today there are many banks, but if you don't pay back a loan, your bank tells the others and you wont get another from any of them. There are many universities, but the all have similar admission criteria, etc. The fact that computers instead of people make these simple calculations isn't a problem. My concern is that algorithms will be used to create affirmative action for things that were never possible before. Some people will start paying higher prices on Amazon and not know it. Groups that organize politically will be able to tip the scales in their favor by claiming the unbiased algorithm "keeps them down" or that the biases people develop against them must be offset by opposite biases in the algorithms. Whereas before every outcome was the product of decisions that every human being got to make about their voluntary interactions, now the reigns of power in the economy are centralized in the hands of whoever gets to design the algorithms and the economy is superseded by a zero-sum political struggle. That is the biggest danger of algorithms.

Things like this make you wonder how system works at all, if a decision can't be explained. Shouldn't it - obviously - be thrown out in a case of a dispute for this reason alone?