The issue here isn't that machine learning gives wrong answers, it's that our definition of 'fair' is irrational.
The issue here isn't that machine learning gives wrong answers, it's that our definition of 'fair' is irrational.
Hypothetical possibility: members of group X are not perceived as 100% as effective as group Y because of pervasive bias by the employers that assumes their incompetence. They are generally perceived to be 80% as effective as a standard Y member despite actual 100% performance, and paid accordingly. A member of X needs to be 120% as effective as a Y member to be perceived at 100% Y efficiency because of stereotypes coloring their perception and an inability to objectively evaluate their performance.
Some non-hypothetical studies touching on this:
http://www.nber.org/papers/w9873.pdf http://www.pnas.org/content/109/41/16474.full.pdf+html http://advance.cornell.edu/documents/ImpactofGender.pdf http://www.socialjudgments.com/docs/Uhlmann%20and%20Cohen%20...