Bias detectives: the researchers striving to make algorithms fair
nature.com
nature.com
TBF there is at least one leading ML scientist that has made a huge narrative on discriminating AI, particularly against women.
(In the insurance case, ProPublica attempted to compare areas with comparable crash frequencies and show that rates were higher in poor and minority areas. But the data they had was moving accidents, especially with injuries, and the data they didn't have was stuff like "rate of car break-ins" and "odds of being hit by an uninsured driver". Which you would obviously expect to vary by region even when serious-injury accidents don't.)
If you want to claim you know what fair is in any given situation, then go and hardcode all your own fair rules, because you aren't going to find "fairness" in machine learning.
This strikes me as eerily similar to the argument that type systems are impossible because of the halting problem. It's sort of true in some sense, but not in an even remotely useful way. So it mostly functions as a way of derailing the conversation away of the more subtle distinctions that do matter (e.g., could we design an easy-to-use type system that rules out this particular type of non-termination/other class of bugs).
There's a large middle-ground between "hard code all your own rules" and "completely unconstrained learning". Learning under constraints is not a new idea.
A classical programming analogy to your argument might be "well the halting problem is undecidable so ignore all this high-level language stuff and just go code up your own turing machine; it's the best you'll ever be able to do".
> because you aren't going to find "fairness" in machine learning.
Why not? The human notion of fairness is fuzzy, which is why researchers have provided various formal notions of fairness in machine learning tasks. Obviously, these formal definitions may or may not correspond to your own gut instinct about what is "fair". And there might be friction between different notions of fairness. None of that should be surprising; otherwise, fairness wouldn't be something that philosophers continue to bleed ink about.
But it is equally obvious that for some notions of fairness, there will exist machine learning algorithms that learn well under the given constraint.
I'm not actually saying hardcore all your rules. I was making an example that if you really know exactly what fairness is, then program it. But we both don't know there are no definite rules. So why hardcode at all?
I think it is never the ML that is "unfair", it's the one who made it who is responsible. If you're finding yourself running into issues in your ML with unfairness, I think you're just using it wrong.
EDIT: Rewording last sentence
... to the point that every attempt to define "fairness" in one (arguable) aspect leads to (codified, inescapable) unfairness in another.
It's good that some Computer Science researchers are ready to work in such politicized fields though, it's definitely necessary. I find it admirable because I personally wouldn't enjoy those discussions.
> [then] it was given biased samples
David Deutsch had an anecdote in his book The Beginning of Infinity about explaining Arrow’s theorem to a US congressman and eventually getting him to the point of understanding how it applies to the electoral college in principle and there is no simple legislative change that could mollify it (I think the discussion was about how preferential voting would improve upon FPTP voting).
The congressman replied something like “this is lamentable” and Deutsch wrote a big passage about whether it makes sense that anyone should ever find a mathematical fact to be “lamentable.”
I do think there will be pragmatic varieties of this sort of impossibility theorem for machine learning fairness and that society in a broad sense will have a hard time processing it, and the possible legislative reactions might be totally unreasonable, even unintentionally harmful.
[0]: < https://en.m.wikipedia.org/wiki/Arrow's_impossibility_theore... >
No matter how accurate your metric is—nay, because of its accuracy—people will be mad at you because it reflects an underlying reality they are in denial of or believe is itself unfair, and want to rectify by requiring fictions elsewhere.
I don’t envy people working on this because you by definition can’t win. Some of the powerful political forces, on the one hand, and the demands for accuracy and effectiveness, on the other, are irreconcilable.
The article is about the unfairness of a higher percentage of <racial group> getting incorrectly flagged as “criminal activity risks”. The fairness of the expected output is essentially out of scope for this kind of research, it's all about the distribution of correct vs. erroneous decisions. The underlying reality is an imbalance in data quality or similar, not whatever you were thinking about.
Even if this is a case of "the data is biased, not the algorithm", it's important to remember that the data is "who was caught doing this" rather than "who did this". Concluding that people who live in stop-and-frisk areas are more likely to get drug convictions isn't actually wrong, but it's still unhelpful.
IMO, the focus should instead be on not overselling algorithms as being infallible rather something which will have some bias and needs overriding time to time. If a system is fully automated without checks and balances we might have serious problems. A good non-ML example was discussed couple of days ago on HN where a person was terminated by a machine without much oversight:
This references two different, almost opposite, meanings of the word bias. The meaning in TFA is being individually subject to judgements about groups (eg an individual AA prisoner being denied parole because AAs as a population have higher rates of recidivism) whereas in the context of ML it refers to a model ignoring important features of a dataset.
For instance, US law has "disparate impact" provisions at least since the civil rights act (a time which was probably not dominated by political correctness), which requires outcomes to be not too different between races or other groups.
(Though disparate impact is not a particularly good metric of fairness and doesn't seem to be used so much in algorithmic fairness nowadays.)
By contrast imagine for a labor job I decide to never hire anybody who can't left and carry at least 120 pounds. That would also have a disparate effect negatively impacting a protected class, but it would be justified. Machine learning takes all data and draws conclusions that map strongly against the data and ideally generalize to new scenarios. So long as the behavior trying to be predicted for was relevant to the task at hand, using the recommendations of such algorithms would certainly be justified and thus not fall under disparate impact.
In a nutshell disparate impact is not to ensure equality of result, but to prevent discrimination by proxy. And discrimination not being of the form 'I'm not going to hire women because they can't lift as much as men' but of the form 'I'm not going to hire women because I don't like working with women.'
It has led to some unfortunate rules. For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but seems to be okay (in practice) to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
What's mostly controversial is fictitious applications of it.
> For example, it’s (generally and presumptively) illegal to hire based on intelligence tests, but totally okay to hire only from elite universities that select students largely on the basis of SAT scores, which correlate very strongly with IQ.
No, it's not. Both policies have differential impact, and both have the same requirement of a tight connection to job performance under disparate impact analysis. The main relevant differences are:
(1) There is an widespread cargo cult belief, including among many people involved in hiring, that IQ tests are categorically illegal in hiring, so people avoid them in part based on a prohibition that does not actually exist, and
(2) The same false belief above also is common among workers, so they are particularly likely to seek legal counsel if they experience an adverse hiring decision and were subjected to an IQ test; at which point they are likely to discover the actual disparate impact rule. People who have adverse results and were, even overtly, subjected to another criteria which might invoke disparate impact analysis are less likely to seek a remedy.
(3) IQ tests are overt and the failed applicant is unlikely to be ignorant of the standard used; places that filter strictly by elite universities rarely disclose that, they just require a resume and apply opaque criteria to it. Even if those criteria (such as an elite university filter) would provoke disparate impact analysis, it is difficult for an injured party to know what criteria were used (and to prove what criteria were used even if they know or suspect that one was used that would be subject to disparate impact analysis.)
That's not what “disparate impact” requires; it requires that acts producing differential outcomes for protected classes are demonstrably tightly aligned with a legitimate interest (e.g., in hiring, criteria with such an impact must be demonstrably tightly tied to job performance.)
Differences, even extremely wide differences, in outcomes associated with protected class are still legal, if they have a demonstrably legitimate basis.
> Comparing black and white defendants, the journalists found that a disproportionate number of black defendants were ‘false positives’: they were classified by COMPAS as high risk but subsequently not charged with another crime.
It goes on to note that there are other potential measures of bias, such as false negatives, and that external factors such as different arrest rates may bias the data. But it's clear that this issue is more complex than "researchers are trying to get the algorithms to be politically correct".
It doesn't make sense to say that someone classified as "high risk" which is not charged again was misclassfied. Unless "high risk" is supposed to mean "100% probability of being charged with another crime in the next two years".
We wouldn't say that patients with "high risk" of a heart attack were misclassified because they didn't have a heart attack. If we applied a similar reasoning, we would conclude that cardiology guidelines are biased against obese people.
> In the case of the ProPublica example, the claim is that a particular subset of the results of an algorithm is more skewed than the full set.
https://www.propublica.org/article/how-we-analyzed-the-compa...
"black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)."
If we define "high risk" as 60% chance of recidivism, the algorithm is getting it essentially right and according to your previous comment there are no misclassifications (if we assume that the small variation is due to statistical variability). If we want to be hyperprecise there is a small bias: white defendants are getting classified as high risk more than they should and black defendants not as often as they should.
Their claim is not that 60% is too low. They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
If their argument was about miscalibration and they considered that getting 60% is too low, the threshold to define "high risk" could be raised. The score is fine-grained, their analysis is based on the score deciles and they consider the binary partition Low=1,2,3,4 and High=5,6,7,8,9,10.
> Nowhere is it argued that individual samples can be explained probabilistically.
I don't understand what you mean.
> you can’t say that a particular person was misclassified, you are making a statement about the calibration of the algorithm
But no one is arguing that a particular person was misclassified; both the article and the study are about results in aggregate being skewed in some way.
> They never define what the desired precision is, but their use of the terms "misclassify" and "false positive" makes sense only if one expects "high risk" to predict recidivism with 100% precision.
They're not trying to find an absolute number for accuracy, they're comparing data for white and black defendants. Excerpt from the ProPublica article below:
> Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).
In this case, the equation for each race is (defendants who were labeled high risk but didn't recidivate / defendants who didn't recidivate). The accuracy in absolute terms isn't in question here; the claim they are making is that the algorithm mistakenly classifies black defendants as high risk at twice the rate it does white ones.
Note that testing for true positives is different from testing for false positives, and both can be true simultaneously. If of 15 defendants, the algorithm predicts ten will recidivate, of whom six actually do, that gives 60% accuracy with ~45% of those who didn't misclassified as high risk.
(I'm also not sure if you're making the semantic argument that "it just said recidivism was likely, not guaranteed, so even if they don't it's not technically wrong." That's a weird assertion to make; if every high-risk classification never recidivated but every low-risk one did, by that definition the algorithm still wouldn't be "wrong", but it would obviously be broken. Classifying a defendant as high risk is a prediction that they will recidivate, and we can judge the algorithm by whether it turned out to be true.)
I'll also note that other replies to my original comment have been good rebuttals to the ProPublica study specifically, but what they're attempting to do — draw conclusions about false positives based on the "high risk" classifier — makes perfect sense.
There are two groups, A and B. For both groups, when a person is labeled high risk they have a 75% risk of reoffending (ie no bias).
10% of Group A people are labeled high risk. Which means that (1 - 75%) * 10% = 2.5% are misclassified.
20% of Group B people are labeled high risk. Which means that (1 - 75%) * 20% = 5% are misclassified.
As you can see, this is simply an artifact of the higher reoffending rate of Group B. If 1) the two groups have different reoffending rates, and 2) the algorithm for labeling people high risk is unbiased, it necessarily follows that the two groups will have different misclassification rates.
The entire ProPublica piece (and your arguments) is based on misunderstanding this. Part of the issue is that extremely misleading table titled "Prediction Fails Differently for Black Defendants", where "Labeled Higher Risk, But Didn’t Re-Offend" actually refers to P(labeled higher risk | didn't re-offend).
The article claims that each and every particular person that was classified as high risk but didn't commit a new offence was "misidentified as high risk".
The ProPublica article and analysis claim that each and every particular person that was classified as high risk but didn't commit a new offence was "misclassified" and "falsely flagged as a future criminal".
There are ways to control for these sorts of effects, but I don't see that done with the COMPAS data. In that case researchers got the data years after the fact so it would be hard to separate misclassification due to bad biases vs those from feedback of the system working correctly.
I think it is fair to be suspicious of systems like COMPAS, but I don't think the misclassifications identified are a slam dunk against COMPAS.
For reference, here is the overview of their process: https://www.propublica.org/article/how-we-analyzed-the-compa...
What the ProPublica analysis showed was that a black person and a white person with the same risk score have the same probability of recidivism. There is no racial bias.
Black people have higher rates of recidivism, which means that they will also have a higher rate of false positives. This is mathematically unavoidable. I recommend creating a toy example in excel and playing around with some numbers to get an intuition for it.
Also check out this analysis: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
Garbage in, garbage out.