You need to watch Person of Interest and Minority Report.
It's a similar problem to using ML to give people credit scores.
If the training data includes a lot of minorities and poor people breaking laws / delinquent payments, then your ML will simply key on race/economic status as a predictor.
So you've built a system that simply targets those groups.
But you might object and say that this race/economic status targeting gives the highest accuracy! It was only learned in the training data, after all. You can make a great classifier that is extremely unfair.
So you have to realize there is a conflict here between accuracy and fairness. This means there is a conflict between observational data (training), and using that data to produce decisions/outcomes.
If you make decisions/outcomes that reinforce the training data, you do not give racial groups/low economic status people a chance to improve their lives.
That is extremely inhuman, predatory, and unfair.
Lady Justice doesn't wear a blindfold as a fashion accessory. Discarding information is a key factor in nearly every established system of justice / morality. Refusing to do so (i.e. "just" running a ML algorithm) places you directly at odds with society's hard-earned best practices.
I never noticed that before. Thanks for pointing this out!
Ok, and to what end?
I assume someone else will be consuming these predictions, else you wouldn't bother at all.
What are your customers/users going to do with these predictions?
Or is that simply not your responsibility; someone else's problem?