Cathy O’Neil on Weapons of Math Destruction

It's true -- we also, however, ban firing based on race or sex. Purportedly, in the view of the managers who would wish to do so, their company would be more efficient if they were allowed to hire and fire whomever they wish (i.e. only employ white people, for example) -- perhaps they believe, in their racist world view, that having a company full of one race would create workplace unity, etc.

We could just let the market decide, and since we know the antiracist hypothesis is true, racist companies would miss on a huge amount of qualified labor and get beaten in a free market, forgetting for a moment that free markets are a myth and don't exist, and that the markets we do have aren't even close to efficient.

But we don't. We force companies to act a certain way even though they don't wish to because we are forcing ethical standards on them. This is, akin to your point at the expense of all of the workers not protected by the law. Every time we prevent a black person from being fired because he is black, it's at the expense of the white person who would take his job. The costs are passed down to white people. Neither choice is completely without consequences, yet we've made the one against racism.

> It's important that while we keep in mind the ethical considerations of minorities there is also an ethical consideration of marginalized users of financial capital, who are denied credit because banks are unable to afford them a lower rate due to lacking information.

That's true. If we didn't give credit to more than a few outstanding black folks, then some poor whites could get credit cards at lower rates.

The most important thing to realize here is that DISCRIMINATION CAN WORK. If everyone agrees that redheads are no good, and everyone is extra careful about lending money to redheads and extra reluctant to hire redheads and extra-strict when deciding how to prosecute and sentence redheads, then network effects will make it a self-fulfilling prophecy. A redhead will be more likely to get caught up in criminal proceedings, will be more likely to get fired (or not hired in the first place), and therefore will be more likely to default on their loans.

For the most part, society has decided that this is either a moral outrage or a case of tragedy of the commons. From the moral point of view we say it's just not ethical to discriminate against people based on race, sex, family, and such. From a purely utilitarian point of view we can say that discrimination can benefit one party at a cost born by all of society. As is usual with tragedy of the commons situations, we can repair the problem with regulation. Regardless of whether you prefer the moral approach or the utilitarian one, there is a pretty strong case to be made that it is GOOD (on a society-wide basis) to give better deals to some (those marginal whites) than others (those marginal blacks) by prohibiting the use of certain information in granting credit.

good pithy way to put it, "ignorant but considerate". I think our society entering into the whole data space will unveil more and more ACTUAL disparities and inequalities with such accuracy that economists may very well introduce concepts and notions such as the hidden costs to willful ignorance. Perhaps call it data omission costs, blacklist data costs, model error from black list data omission. Stuff like that. You hit the nail on the head, neither choice is without consequences.

Those laws were passed because of cultural values that are widely shared. You make it sound as if bankers would prefer racist policies if only the law would allow it. GP's point is that even though most bankers do have such high-minded motivations, regulation is necessary to avoid bad actors from gaining a competitive advantage in a race to the bottom.

> You're making it sound as if lenders don't apply criteria such as race out of high-minded civic duty and a commitment to ethics.

I work for such a lender. Yes, that is precisely what I am saying. Although I might call it "basic ethics" not "high-minded civic duty".

Typically now "you" (someone raising this issue, because I don't want to put words in your mouth) point out that corporations are not ethical and simply act to seek maximal profit. Then I point out that corporations have a culture, which may have values other than just profit, and that corporations are made up of individuals who ALSO have motivations other than just profit.

The company I work for also has resisted the urge to open accounts that customers didn't ask for (as recently revealed of Wells Fargo) and many other ethical deviations. I am sure we have our own sins, and should work to improve them. I agree that laws constraining corporate behavior are ONE good tool for managing corporate behavior, but it is not true that absent such laws there would be no other constraints.

Given the absence of such a perfect algorithm, allowing or favoring certain types of errors over others is absolutely a decision with ethical consequences.

Sure, but there's still a question on how to fix that. Should we prevent these companies from applying more targeted models, and hence impose a tax on the random subset of consumers who happen to use the same services? Or should we take tax money, which is taken from everyone in a democratically decided way, to help people of color get loans?

Or to put it more specifically, why should a middle-class white family have to pay higher rates on their loan to help some family of color get a loan, while the rich white family that buys the house outright doesn't?

Shouldn't we help people of color using a system that distributes the costs in a widespread and progressive way?