"Machine learning systems trained to minimize prediction error may often exhibit discriminatory behavior based on sensitive characteristics such as race and gender. One reason could be due to historical bias in the data."

The thing about this discussion is that it aims to balance social welfare goals and profit maximizing goals but without any criteria for attaining basic fairness on a general level.

To wit, suppose a machine learning algorithm is looking at twenty or fifty piece of data about a given individual, all of which are entirely irrelevant to the individual's chance of repaying a loan. But by random chance, one of those pieces of data, say handedness, happens to be correlated with a group's repayment history. So, some individuals with a good handedness are given loans more frequently and some individuals, with a bad handedness, are given loans less frequently. This situation doesn't matter to the company since the data is irrelevant and they only give out so many loans anyway and shitting on, say, left-handed people, gives them no grief. Moreover, if this trend is noted by all the companies soon it will be made "true".

Which to say, companies making life-defining decisions like mortgages or parole-granting, just should be prohibited from the using a lasagna of random data to make their decisions and instead should be required to use specific rules with specific reasons behind them.

And cry me a river about missed chances for optimization. This about the structure of society and optimization doesn't benefit society here imo.