Attacking discrimination with smarter machine learning
research.google.com
research.google.com
A real life example: In many (all?) countries, it's common to pay more for your car insurance if you are young, if you are male, or if you recently acquired your driver's license. One might frame this as a prejudice towards young, male drivers as being reckless. But statistically they are just that, so the prejudice is balanced (within a reasonable tolerance). It's unfair to the careful young, male driver, but well, life is not always fair.
Fortunately I'm sure people are hard at work finding "proxy factors" which happen to map to pretty well gender but have an acceptable parallel explanation for why they're meaningfully descriptive of an applicants risk profile :)
Plus, women want equal rights, so why not be equal in this, too?
I think for things like insurance, or healthcare taxes (the model adopted by most countries) it's fair to normalize the price for everyone. Otherwise, the system doesn't really work.
It is not fair to normalize the price. It penalizes people who are healthy and good drivers (for example not smoking and speeding).
It's interesting that insurance premiums can act as behavioral incentives in some cases, but I doubt it has ever led someone to change their gender or race.
For example, the price of bond insurance is not normalized between bonds with different ratings.
Supposing this price discovery were perfect, customers who make a claim would have already paid premiums equal to the claim, and customers who never make a claim would pay nothing (plus fees in each case). This aspect of insurance obviously serves no core purpose. The purposeless of it is just hidden in the noise of uncertainty, and we see pricing differences emerge out of the uncertainty.
The actual purpose is to blunt the effect of catastrophic losses by spreading a smaller burden across a larger population. One caveat is that insurance can sometimes have the effect of "rewarding" risky behavior and "punishing" cautious behavior. Now, it's interesting that price differences can sometimes balance that by reversing the punishment/reward incentives, but only if prices are based on risk factors that the customer can actually control.
My male Montana cousins always had slightly cheaper insurance than I did, but my female Montana cousins more than made up for it.
That's the computer science problem being worked on here.
Not everyone shares your politics.
I'm perfectly happy with algorithms detecting that certain people are more likely to be safe drivers than average, and giving them lower rates, and concentrating premiums on the groups more likely to be in accidents, even if I don't understand why Armenians (in your example) get in more crashes.
The distinction is that with young male drivers, we have two supporting classes of information:
* A clear statistical observation
* A conceptual understanding of why the observation is likely to be valid
With machine learning, we might have neither of these classes of information.
<shrug> What's the null hypothesis here? From my point of view, you're the one who defined into existence as a problem something that is not a problem.
A great example is how very resentful many young white men of college age are that universities are requiring them to take sensitivity courses designed to reduce the instance of campus rape, but strictly speaking men of that age are the overwhelming majority of bad actors in that environment. Statistically and logistically speaking, it's smarter and cheaper to just require all men of college age to take courses reminding them that rape is not okay rather than dealing with the moral, legal and healthcare costs of the alternative.
In some cases, our relatively primitive algorithms pick up on correlations that should not be acted on because we're actively working to correct them. For example, it would be inappropriate to pre-reject job applications based on skin color if in a certain culture, it's less likely for that person to have a college degree.
Even if that insight is correct, it's usually part of something that society hopes to correct or that applicants should be given the benefit of the doubt about, otherwise very serious negative responses will emerge.
Acting on existing categories may reinforce. It may not. For now, it's a case-by-case basis we'll have to act on. Maybe one day, modeling techniques and data sources will become sophisticated and robust enough to make every decision for us. That day is not today.
I'm sure that I've experienced increased costs based on this.
The issue is that I don't consider that the morality of forcing my will on other people depends at all on whether their current behavior is advantageous or disadvantageous to me.
Why on earth would I believe anyone who says this? I don't think humans are capable of such abstractness. It's not a matter of wanting to, biology itself is at odds with this mindset.
But also, there are many markets which are not effectively free markets. Insurance is a good one, and health service is an especially good one. In these cases, it's very dangerous to start agreeing that insurance agencies can start to pay "pass the puck" with human life.
And I'm not sure what sexual orientation has to do with any of this.
Nice bait, I guess?
Even if that was true[1], where are you getting the data from? Choosing which types of data to use is just as important as the model.
[1] as others have already pointed out, it depends on the technique/etc
To use your Armenian example, it could be that while being Armenian doesn't actually affect your driving, a "true" model could still end up being bad for Armenians if being Armenian is correlated with the things that actually do affect crash risk.
But what about this: we don't need to solve all social ills every single time. What if we let the algorithm correctly decide crash risk, and if we notice that it unduly impacts Armenians and that's not an outcome we want, we via a separate channel compensate the Armenians? That is, acknowledge the fact that Armenians may crash more, but give them a government subsidy to offset the higher premiums, and work to bring the premiums down (i.e.: fix the underlying issues that being Armenian is correlated with).
It's related to something I've been thinking about lately with regard to minimum wage. I like the idea that everyone should have a livable income, but tying the implementation to businesses that have low wage jobs seems like mixing concerns. For example, my company doesn't have any minimum wage jobs, but shouldn't my company chip into this social ideal same as any other?
What if we let the businesses pay whatever the market will bear, and if we decide that as a social concern people should get more than that, we subsidize them from the government, which is wear this concern is coming from in the first place.
I'm just incredibly impressed that you came up with a way to make this sound even worse.. can you imagine the government giving subsidies to Armenian drivers with good records because insurance companies are overcharging them due to the statistical performance of their category?
A tax benefit for being an outlier from the mean? My god. Not only does that sound like a terrible idea, but intensely complicated to organize in the general case.
And what happens when an individual is in multiple categories? What if only gay Armenians who listen to disco are the at-risk driving category? How do we aggregate the tax subsidies, multiplicatively, additively, ..? This sounds like an administrative nightmare, I congratulate you on your evil ingenuity.
Lets say that I am male, and am young. Lets say that as a young male driver, I am likely to be a cause of an accident 20% of the time i.e. 1 in 5 young male drivers will cause an accident. Lets say as a young female driver, I am likely to be the cause of an accident 5% of the time.
So on the face of it, young male drivers are riskier, and should pay appropriately.
But lets say there was another measurable factor, such as a 'recklessness' score, which describes how recklessly you behave. And lets say that if you are 'reckless', you are likely to be the cause of an accident 90% of the time. And the maths works out that if you take the 'not reckless' group from men and women, they are equally likely to be the cause of an accident, i.e. there are more reckless men than women.
If you were to take the stats at face value, then a good proportion of people are being over and under-charged, because you are using a proxy measure rather than digging deeper for a root cause/don't have the data available. I think this is exactly what people are afraid of, businesses/people being lazy and making assumptions that are correlative, based on characteristics that one can't change.
This research is about what happens when combining the recklessness score with gender-based distributions provides more accurate results than recklessness alone. Many people consider this to be "unfair". This article shows how much profit you need to sacrifice to get fairness.
However with a lot of ML classes we are told to look for the simpler explanation which would be nationality in this case, in the absence of collection of the real reasons.
Not sure how that can be gotten around. I'll read the paper.
If a computer program spots an irrelevant unhelpful correlation its generalization error will noticeable go up. If it is an irrelevant "helpful" correlation it means there is a problem with the data (such as leakage), not with the algorithm. If there is a problem with the data, all bets are of, both for black and white box models.
A blackbox model will probably not find that being an Armenian alone will lead to more crashes. Being non-linear in nature it will find interactions (young male Armenians are more likely to crash than young males in general). If the importance of such a feature is not significant enough to distinguish it from noise, then regularization may automatically remove it.
Even if we observe that 99 out 100 Armenians crash their cars, and you decline someone a loan, because he/she is Armenian, you may just have discriminated against the 1 Armenian who is a safe driver. Young male drivers who drive safely have a worse time getting loans, because their group (the set of young male drivers) spoiled it for them. So their only hope of getting a loan is you adding more features (like nationality), to be able to distinguish them as safe drivers, not removing them and lumping them into the status quo.
The simulator is a great illustration of exactly what they did; the entirety of the work is generalizing that to arbitrary predictors (subject to a few conditions) and bounding the accuracy penalty.
Feel free to cite the theorem improving accuracy if you disagree.
[1] One idea I'm kicking around is the following. Banks/others are legally required to issue bad loans for fairness. I suspect there is a lot of money to be made hacking this, I just haven't figured out how yet.
But sometimes they loan to people they wouldn't normally loan to, just because that loan adds the kind of data they would otherwise not get, and they take a risk on default in exchange for more data. I'm assuming smaller loan amounts.
So I guess that is one way for lending institutions to say they are issuing "bad loans for fairness" while still benefiting from it in terms of getting more training data and making the model more robust.
The problem being worked on here is "what if Armenians shouldn't get car loans because they don't pay them back as much as other groups?" I.e., algorithms rightly classifying people leads to results that we believe are "unfair".
This research is illustrating that you can't simultaneously have accuracy and fairness. You need to explicitly decide how much accuracy you are willing to give up to get fairness. I.e. it's computing the tradeoffs needed to evaluate the ethical question: how many Armenian deadbeats should you extend credit to in order to be "fair"?
Go play with the simulation to see. The various fairness criteria all achieve lower than maximal profits.
A key result in the paper by Hardt, Price, and Srebro shows that—given essentially any scoring system—it's possible to efficiently find thresholds that meet any of these criteria.
The crucial thing is that this "credit score" doesn't actually predict defaults quite well, because blues who are 50% likely to default have higher score than oranges who are 50% likely to default. If you base decision on a simple credit score cutoff, the blue defaulters will screw you up because of their higher score. You can defend from this by tweaking cutoffs per-group to achieve profit maximization or various "fairness" goals. In a sense, it's an attempt at producing non-garbage out from garbage in.
And of course whether real-world implementations of this idea are "good" or "bad" is another can of worms altogether and varies case by case. They probably could have came up with some better examples than credit scores and labeling people by colors.
The best predictor is one which is explicitly discriminatory based on race: it takes both FICO score and race into account. There are worse predictors which also discriminate explicitly based on race, but in some "fair" way.
Finally, the worst predictor throws away directly relevant racial information.
This is not about garbage input data - that's not the problem being addressed here at all. The input data is perfectly fine. The problem is just that the input data says "race discrimination is the best way to make money".
> The problem being worked on here is "what if Armenians shouldn't get car loans because they don't pay them back as much as other groups?" I.e., algorithms rightly classifying people leads to results that we believe are "unfair".
No. If you stop playing with simulation and look at the input data, Armenians don't pay back less. They pay exactly as much as Iranians except that Iranians consistently have higher FICO scores because Iran infiltrated FICO with their suckxnet(TM) worm which replaces R binaries with hacked versions. Or something like that :)
The general abstract idea is: you have some input "score" which is know to inaccurately predict the outcome, use the input score and measurements of its biases to produce more accurate prediction than naive threshold classifier would.
And yes, the other guys talking about women's healthcare costing more or men causing more traffic accidents got it wrong too. I somewhat arbitrarily responded to you because you said something about "the real computer science problem being worked on here" and then continued to talk about other things like everybody else.
Why don't you quote the place in the paper where they make accuracy go up, fix overfitting, or build an improved risk score? Or even just quote a place in the paper where the risk score is treated as anything other than an accurate black box?
They pay exactly as much as Iranians except that Iranians consistently have higher FICO scores because Iran infiltrated FICO with their suckxnet(TM) worm which replaces R binaries with hacked versions.
In the example provided in the paper (see Fig 7) that's explicitly NOT true. Blacks pay back their loans a lot less than asians/whites holding FICO fixed. For example, at a FICO score of 500, blacks pay back their loans 10% of the time while Asians do about 40%.
I.e., blacks have consistently lower FICO scores because they don't pay back their loans. Further, FICO score is biased in favor of blacks. If we made it more accurate we'd be actively discriminating against blacks. For example, a black person with a financial situation reflecting a FICO of 500 would have their FICO score lowered to approx 450 to reflect their higher default rate.
Did you even read the paper, or the linked article?
I think we can agree that using the same cutoff for both races gives less accuracy than using higher cutoff for Blacks and lower for Asians, for some values of "higher" and "lower". You are right that "race discrimination is the best way to make money", but at the same time I think I'm right that this happens because FICO score already is racist to begin with, which is the "garbage in" I talked about.
But this paper is NOT about correcting that bias. If it was then it would be a pretty short paper:
Abstract: Go use isotonic regression in scikit-learn, once for each group.
References: Some papers from the 1970's when isotonic regression was developed.
http://scikit-learn.org/stable/modules/calibration.html
...but at the same time I think I'm right that this happens because FICO score already is racist to begin with...
This is also incorrect. The conclusions of the paper would remain valid if FICO were calibrated identically for all groups.
Can I suggest reading the paper and working through the math?
I guess you are right, this was just the first disparity I noticed in the orange/blue example and I got fixated on it. And yes, I haven't read the whole thing and my maths may be a bit rusty nowadays :)
Now I see that the problem they attempt to solve with "equal opportunity" is FICO's (in)ability to fish reliable borrowers out of the whole population. Currently, FICO identifies a small number of reliable Black borrowers whom they give high scores, plus there are many Blacks who would pay back diluted in a sea of unreliable borrowers with low scores. While in, say, Asians, the ratio of reliable borrowers who had been given high scores is higher (fig. 8).
I think the issue is a bit more nuanced than "algorithms rightly showing that fairness is opposite to profit".
In particular
> The best predictor is one which is explicitly discriminatory based on race: it takes both FICO score and race into account. [...] the worst predictor ["race blind"] throws away directly relevant racial information.
As I noted, this is a case of reverse bias directly compensating for FICO's bias. Max profit simply grants loans to all FICO score buckets whose default risk is sufficiently small to be worth it. If FICO vs risk was race-independent, "max profit" would use the same FICO threshold for every race and hence would be equivalent to "race blind".
Analogously, "max profit" could be equivalent to "equal opportunity" if FICO was better at finding Blacks who can pay and putting them in low risk buckets (high score) so that it becomes feasible and profitable for banks to grant them loans. I believe this is what authors meant by incentivising classifiers to improve accuracy and yes, I was wrong suggesting that they found a way to improve accuracy here by using the input classifier as a black box. Their ideas only compensate for the aforementioned race-biased score inflation, which isn't the entirety of the problem, and put some financial burden for some other mispredictions on banks/FICO to pressure them into getting their shit together.
I think trying to apply "equal opportunity" in the real world may indeed turn into handouts to people who can't pay, because <handwaving> it's possible that poor people are just hard to classify correctly and if certain ethnicities are poorer than others, they will appear to be given less opportunity even though actually it's simply poor people in general who are being given less opportunity. If FICO finds ways to classify poor Blacks better, it may turn out that applying the same solutions to other groups will improve their true positive rates too and hence "Black opportunity disadvantage" will stay.</handwaving>
But otoh, it also is possible that <handwaving> for some reasons Blacks are classified with less accuracy than Asians, which contributes to the lower overall true positive rate of the race.</handwaving> Authors appear to be assuming this possibility, though I don't think the distinction can be made from data presented in the paper alone.
TL;DR: I think you were oversimplifying things, and so was I :)
Your dispute may be with the authors.
We accept that younger drivers pay more for car insurance. And that disabled people pay more for their health insurance. And that the government (in the UK) has a special business grants scheme for ethnic minority entrepreneurs.
Obviously discrimination depends on context. If a car insurance firm had a special policy for ethnic minorities, people would (rightly) be outraged. But in the context of a government intervention, based on the evidenced disadvantages that ethnic minorities face, discrimination is accepted.
Collectivism is just fine when you are the one on the receiving end. Either we set absolute lines of discrimination across all industries or we disallow it entirely.
The parent comment explicitly said "before factoring in pregnancy". So this is not about continuation of the species, if that comment is correct.
This does not seem fair to me, because if this is applied then your race (group) would determine your credit score threshold which feels discriminatory to me.
I feel that, by definition, it is not discriminatory only if none of your attributes that you could be discriminated against (gender, race, ...) are taken into account at all.
Maybe the concept of 'equal opportunity' is just some compromise between discrimination and making less informed decisions.
In practice, the first variables to be used for classification are the ones that have a biggest effect. Then you're going to use them (from order of importance) as eliminatory or classificatory
> because if this is applied then your race (group) would determine your credit score threshold which feels discriminatory to me.
But the opposite is also discriminatory, which is what the article is showing. Because then you're using the same ruler to evaluate different groups, and of course the minority person with 2 jobs can't match the credit score of an Ivy-League educated WASP
For example, let's take blacks in the US. The data tells you that a black person is more likely to be a criminal than a white person. There are two possible reasons for this: (1) blacks are more prone to crime, or (2) blacks are more likely to live in circumstances that make them criminals. With access to only anecdotal data, I strongly believe that (2) is true, and that if you took into account enough circumstances (e.g. single parent, school district, income level, parents' wealth) you'd be able to remove race from your model and still arrive to the "equal opportunity" result. That way, you wouldn't discriminate based on race, but you would still help the people most needy of help (poor, uneducated, etc.). I think the same applies to colege applications and sex wage gap, which is why I strongly oppose any kind of affirmative action.
Life expentancy sex gap might be a different case. IIRC, men die earlier because of some behavioural/social tendencies (working dangerous jobs, supressing emotions, risky behaviour (speeding, smoking)), so maybe there are some non-discriminatory (or at least less-discriminatory) ways of determining life expectancy (e.g. testosterone level, job description, ...). However, there are also biological differences that seem very strongly linked to the quality of being male (i.e. the Y chromosome). E.g. prostate cancer is less lethal than breast cancer, and women are more likely to get MS than men. These particular examples suggest that men should live longer, but I'm guessing there might be other gender-specific ilnesses that might reduce their (our) expected lifespan. If that's the case, I don't think it's too discriminatory to have different insurance fees for different sexes.
In all such scenarios, however, there could still be broader societal goals that would override specific instances of (non-)discrimination. For exapmle, AFAIK women's health is more expensive (because of pregnancy), but having more children benefits everyone in the society (in the West), so it makes sense to "discriminate" against men by letting women pay less for their health insurance.
I'd like to add a third possible reason for your consideration. Since "criminality" i.e. guilt of committing a crime is determined after a process engaging the law enforcement and justice systems, we have to examine whether there are inherent biases in those systems that result in skewed statistics. For instance, do police officers selectively target blacks for monitoring and investigation? Are blacks discriminated against in the courtroom as a result of procedure or human nature?
Do White Police Officers Unfairly Target Black Suspects?
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2870189&...
Using a unique data set we link the race of police officers who kill suspects with the race of those who are killed across the United States. We have data on a total of 2,699 fatal police killings for the years 2013 to 2015. This is 1,333 more killings by police than is provided by the FBI data on justifiable police homicides. When either the violent crime rate or the demographics of a city are accounted for, we find that white police officers are not significantly more likely to kill a black suspect. For the estimates where we know the race of the officer who killed the suspect, the ratio of the rate that blacks are killed by black versus white officers is large — ranging from 3 to 5 times larger. However, because the media may under report the officer’s race when black offic-ers are involved, other results that account for the fact that a disproportionate number of the un-known race officers may be more reliable. They indicate no statistically significant difference be-tween killings of black suspects by black and white officers. Our panel data analysis that looks at killings at the police department level confirms this. These findings are inconsistent with taste-based racial discrimination against blacks by white police officers. Our estimates examining the killings of white and Hispanic suspects found no differences with respect to the races of police officers. If the police are engaged in discrimination, such discriminatory behavior should also be more difficult when body or other cameras are recording their actions. We find no evidence that body cameras affect either the number of police killings or the racial composition of those killings.
There are lots of ways for the system to be racially skewed/biased without being a product of personal bias - so many that I think the focus on "racist police" makes it hard to recognize a lot of easily-provable problems.
Stop-and-frisk is my go-to example of a system that produces bias regardless of the race or biases of the officers involved. In theory, it's an efficient use of limited police resources, it can be implemented race-blind, and it "only catches criminals". There's room to talk about harassment of non-criminals, but at least regarding the people who get arrested its an understandable idea.
In practice, criminality is a product of conviction. Stop-and-frisk mostly catches 'possession' crimes like personal-use drugs and illegal weapons (and since a 3-inch pocketknife is illegal in many cities, we shouldn't mistake this for violent intent). As a result, living in a stop-and-frisk area massively increases your odds of being charged with a low-grade crime - it's not as though carrying marijuana or a Leatherman is rare among un-policed groups. Even if you attempt a crude race-blind implementation, like policing based on neighborhood crime rate, you end up with a vicious cycle where crime rates are high because enforcement is high.
So I think we do a disservice when we limit our discussion and investigation to officer bias. Even when it's not present, it's still easy to build an unequal system.
True, but there has been more than one paper written on the subject, which don't all agree with the one you linked vis over-representation for crimes.
The black/white marijuana arrest gap, in nine charts
https://www.washingtonpost.com/news/wonk/wp/2013/06/04/the-b...
As you're probably aware, black Americans are arrested for marijuana possession far more frequently than whites. You may also know that there's not much evidence that black people consume marijuana with greater regularity than whites do.
...And this is a uniform phenomenon. It's not that some states treat the races equally and others treat them really unequally. Only in Hawaii are the rates even close to equal, and that's biased by the fact that blacks make up only 1.6 percent of the population. In the state with the second-lowest disparity, Alaska, blacks are 1.6 times more likely to be arrested. In the state with the biggest, Iowa, blacks are 8.34 times more likely to be arrested. D.C. has the second biggest; in the District, blacks are 8.05 times more likely to be arrested.
I'll defend the parent comment by observing that we can reproduce this effect with non-judicial metrics, like "murder rate in majority foo-race communities". Assuming the reporting rate for murders is very high across the board, this escapes bias in both policing and conviction rates, and sends us back to other explanations.
Even so, the policing/judicial question is really important.
There are some sociology results (I haven't seen a replication failure?) suggesting that not only race but stereotypical appearance within race influences sentence duration. There are all kinds of low-level-illegal practices whose effective criminality is shaped entirely by policing - think of any "side hustle", where selling loose cigarettes leads to lots of charges, but selling moonshine (the closest analogy I can think of) leads to relatively few.
More obviously, incidental discovery and stop-and-frisk policing produce wildly imbalanced charge rates. Walking down a city street with a pocketknife and pot is always illegal, but the odds of it being a crime depend heavily on who you are.
I'm not saying anything new, but it's a soapbox worth getting up on. It's a big deal in race, but it's also worth remembering the power of selective policing whenever we talk about criminalizing something widespread.
A community has a fair amount of power in deciding where crime happens. The police (by policy or culture) can choose to push illegal activity to certain neighborhoods. If the police come down hard on drugs, prostitution, etc. in white neighborhoods, then that activity will move to non-white neighborhoods. The problems that come along with increased illegal activity, including higher murder rates, will be concentrated in non-white neighborhoods.
This is pretty much the only way to help people who were born addicted to meth inside a trailer in the mountains who also happen to be blessed with being white.
The problem is, when given access to a large number of classifiers, some of which have inevitably been affected by a pre-existing racial bias, a black box machine learning algorithm will likely become discriminatory as well if race is not in some way represented and equalized.
For instance, many justice systems in the U.S. use machine learning software to determine the likelihood that a criminal will reoffend, and use that prediction to determine sentencing. Race is never used explicitly as a classifier, but the program ended up being significantly more likely to rate blacks as more likely to reoffend [1]. Classifiers like "had parents with previous criminal convictions" can be misleading when blacks are more likely to be convicted for the same crime as whites. It doesn't mean that the white person's parents didn't engage in criminal activity or other reprehensible behavior that might cause their child to become a violent, repeat offending criminal - just that they were able to get away with it more easily because of a biased system.
Machines end up just as biased as the data they've been trained on, so if we are going to use computers to judge things that have such a significant impact on people's lives, we can't risk racism slipping through the cracks.
[1] https://www.propublica.org/article/machine-bias-risk-assessm...
This is simply not true. Black box machine learning algorithms will have the tendency to correct bias in their inputs. Insofar as they do systematically deliver wrong answers, this is actually called "variance" and has no particular sign. It's just as likely to be biased in favor of $protected_class as against that class.
https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
Also, you do know that Pro Publica's R script actually found no bias, right? The bias was actually in the selection of anecdotes in their article, which obscured the fact that their statistical analysis could not reject the null hypothesis.
https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
Past payment history: if you're black, prior discriminatory behaviors may have limited your ability to open credit accounts, and thus you have less history to go on. Available balance has the same reasoning. Zip code correlates with race.
I'd hope very few people are including an "is_black" feature in their classifiers. If you eliminate anything that is informative towards race though, you're likely going to have a classifier that doesn't work very well.
The problem is that we have datasets that have arisen from a history that included significant racism, both overt and latent. There is no way to separate those effects from the data. You either get an "optimal" classifier that is racially biased in ways we don't want, or you get one that intentionally gives up some perceived performance in favor of fairness.
[1] https://www.amazon.com/Weapons-Math-Destruction-Increases-In...
[2] http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...
Personally, I find this as the key outcome here. The accountability is on the people/systems who make the decision and that leads to an appropriate incentive. Win-win as a start.
It's a sensitive topic, because sometimes we're actually tampering with the data, trying to eliminate known human or selection biases.
The first defence against discrimination is, in my honest oppinion, for everyone working with data to be aware of these problems. To know that, besides ROC, precisions and recalls, we should measure the impacts of the models in sensitive demographics (gender, race, nationality, sexuality).
And one of the things that I learned (in [1]) is that, even if you're carfull with the features you use, you might still have a negative effect.
One needs to understand how these features interplay with each other. For example, you may not directly use a protected class feature (race) to make your prediction but you might end up using a secondary or tertiary variable (like location) to end up learning a protected class feature due to statistical correlations.
I would imagine that if we are separating people into groups based on demographic or social factors to make decisions, then those decisions may have an impact on that entire group. (In this example, maybe granting more loans to the blue group alters the group characteristics and leads to higher profit in the long term despite higher immediate risks).
Is there an area of ML research that considers this kind of concept?
My guess is that at most companies spending time on making an algorithm non-discriminating will be viewed as a waste of time and money.
It is an interesting question. However, there are ways to answer this question as we have been measuring discrimination before algorithms. Algorithms are a way of reaching a decisions. Therefore, we can measure discrimination if we audit decisions generated by algorithms. In other words, look at the input-output of the algorithm and measure the impact. In the US, the doctrine of disparate impact has long been used a guideline to evaluate discrimination [0].
[0] https://en.wikipedia.org/wiki/Disparate_impact#The_80.25_rul...
Banks are built on trustworthiness. Having your bank's name in the headlines for discriminatory practices can have a severe negative impact on trustworthiness. They have a whole teams of people devoted to this topic and every year at most banks every employee has to learn about, "reputational risk."
Having worked at big banks for over a decade now I am 90% certain that discrimination by banks at this point is primarily due to carelessness.
The consequences of these things are very bad for stock prices! This is doubly true for banks which take reputational risk far more seriously than most businesses.
Only if the cost * probability of a PR disaster outweighs the cost of a poor credit risk model, is a bank economically incentivized to not discriminate.
Then again, even if you take care to not directly discriminate, you will probably indirectly discriminate. For instance, in your credit risk model, remove the `gender` column from the features, and use the remaining features to try to predict it. If performance is better than random guessing, you are using proxy features (like `income`) to discriminate on `gender`. You will find that nearly every feature you use is correlated with `race`. Now what? Throw away all these features and let your competition eat your lunch?
Along the lines of what Pedro Domingos said [2], you can not solve the problem of discrimination by making poorer performing machine learning models that adhere to your view of what is ideal. Discrimination won't disappear because you made a model that makes you feel good. Want no discrimination of women? Work on closing the wage gap. Don't cripple your statistically correct ML models or sweep the discrimination under the rug, covered by correlated variables.
It is not so much carelessness, as it is the nature of the beast. And banks remain in business by how much they can trust their customers first and foremost, trustworthiness by customers is a second (and customer trust is very much malleable: It is the perception of trust, not objective trust like "can we trust this customer to pay back their loan").
It also depends on how you (mathematically) define "fairness" [3]. You can define fairness in ways that still allow you to discriminate.
[1] A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
[2] https://www.youtube.com/watch?v=furfdqtdAvc
[3] https://algorithmicfairness.wordpress.com/2016/09/26/on-the-... https://arxiv.org/abs/1609.07236
Secondly, it isn't mathematically possible to be "nondiscriminatory" - there are multiple definitions of that term and they are mutually conflicting. For example, as this article shows, "equal outcomes", "equal opportunity" and "equal treatment" (group unaware) don't make the same decisions.
So no matter which definition of "fair" you choose, some intrepid reporter can choose a different definition and then write a clickbait article calling you racist.
What is important to note here is that we need to tweak the mathematical model to the culture we want to achieve. In other words, the objective function of the optimization problem needs not only match the current state of the world, and provide an hindsight in one's own economic interests, it also needs to take into account the culture that we want to reflect, and influence. Otherwise, these statistics are just a giant status quo amplifier.
Good point. Systems should work for people, not the other way round.
What I mean, say the model identifies that a certain group has a greater risk due to systemic problems. If you change something about the group, you can change the calculated risk without changing the model. And this may very well be a better way to achieve the outcome you want.
Specifically, by preventing insurance companies from using a more accurate mode, what you're demanding is that the random people who happen to have taken the same insurance packages but are not part of the group should make an extra contribution to fix these systemic problems.
But why them? Shouldn't we all contribute instead, hopefully using a fair system for assessing how much should each pay?
Instead of tweaking the model, you can change the inputs by providing a state-backed guarantee to the underprivileged groups. Isn't it more fair overall?
Specifically, a machine learning will produce different numbers for 2 individuals with the exact same characteristics except the race. And that is the problem that needs to be addressed.
Let's put it in another context. Let's say I'm a white athlete, and I'm very good at running the 100m race. Actually I run just as fast as a black person who is my main rival. Now if someone has to select one of us to go to the Olympics, they should toss a coin to decide who goes. If you use a ML algorithm, it would absolutely send the black person, because no white people has won a 100m race in the last 20 olympics. That's the kind of bias ML does and that needs to be addressed.
The key thing to understand is that we're talking about discrimination that is usually unintended and unexpected by the designers of these systems.
This is presented as if it's an unambiguous fact, when it's largely a political stance.
It's a subjective political view that any or all of those things are injustices, of course, since justice is a subjective thing.
The problem (or at least, one of the more important problems) being addressed in this work is the unintended amplification of the status quo --- the implicit notion that if something is a certain way now, it is best that it always be that way.
That is surprising. And given this discussion about only wanting stats that have a valid explanation as well as fitting the facts, is interesting.
Have you seen statistics to the contrary? Why would it be surprising if you don't have an informed prior in the opposite direction?
As in I am surprised that the parent claimed ... err the parent I replied too seems to have vanished and my comment is looking pretty weird
Anyway from memory he raised that young white men had a higher rape stats than any other group - I was surprised by the inclusion of race in that
We detached this subthread from https://news.ycombinator.com/item?id=13006361.
It is a sensitive subject I suspect
And I cannot find the original parent comment in the discussion and I look like a total dick with my comment coming out of nowhere.
I will try and understand what moved where and get back to you
Aha:
"""A great example is how very resentful many young white men of college age are that universities are requiring them to take sensitivity courses designed to reduce the instance of campus rape, but strictly speaking men of that age are the overwhelming majority of bad actors in that environment. Statist"""
It is only my memory but I am fairly sure that the above comment said "strictly speaking young white men ..." which what prompted my comment. Maybe I transposed the white in the first part of the paragraph to the final part. Not sure. Please do check, let me know if I was being a fool or not.
Darn I need to md5 hash parent comments I reply to now :-)
I should have gone with "citation please"
Anyway, I have written up a sort of overly long comment which won't fit in 2000 chars so I can't submit it and so this is the link
http://www.mikadosoftware.com/articles/HNdisaster
I apologise for thinking this was not a slippery slope and for not reviewing my text for misunderstanding. And I apologise for putting a piece of text up there that is so blatantly... horrific. It's awful to have that in my permanent history even if I know how it was a mistake.
As it says in the article I am going to go and have some reflection time.
What a cock up.
See you in a few weeks.
Do you have anything to back up your claim that African American officers are more likely to use force against African Americans than white officers?
It's very counterintuitive that police officers around the country would have a homogeneous culture that diverges so strongly from the cultures of the diverse communities from which they come and in which they live and in which they work.
It's necessarily "spontaneous" that police officers around the country coming from diverse backgrounds and living and working in diverse communities would spontaneously adopt a culture that biases them against one race in particular, especially when many of those officers are of that same race.
> Do you have anything to back up your claim that African American officers are more likely to use force against African Americans than white officers?
2 separate analyses come to mind, but there are others (I don't have the time to dig up at the moment). As far as I know, this isn't disputed among criminologists.
https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&c...
Secondly you imply that police diversity somehow matches community diversity, when this is manifestly untrue, particularly in places where it is asserted that discrimination is active.
Take Ferguson Missouri, 67.4 percent of the city’s 21,000 residents are black, and 29.3 percent are white.
What is the makeup of the police department? 50 white officers, and 4 African American officers. http://www.politifact.com/punditfact/statements/2014/aug/17/...
The facts do not bear out your conjecture.
2. I made no such implication; only that police come from and work in diverse communities. This means it should be surprising that police from all over and in all places would spontaneously converge on a single, hateful culture.
The facts may not bear out to my conjecture, but you've done nothing to demonstrate this.
Your second claim is false. See my reference to the diversity of the ferguson police department. You are simply shown to be wrong on this count.
Also, nobody has said anything about convergence on a single hateful culture, and nowhere is there anything that requires a biased culture of policing to have occurred spontaneously. That is simply a straw man.
It only demonstrates that there are a lot of search terms that match those criteria. Without diving into each source, we don't know what's even relevant to this conversation (we have no idea how "police culture" is defined in each of these sources, for example).
> Your second claim is false. See my reference to the diversity of the ferguson police department. You are simply shown to be wrong on this count.
I saw your reference to the diversity of the ferguson police department, but it has no bearing to my statement. Pointing out the racial disparity between the Ferguson PD and the community it serves does not disprove the notion that police come from diverse communities, nor that police serve diverse communities. It merely proves that in Ferguson, the racial composition of the police don't match that of the surrounding community.
> Also, nobody has said anything about convergence on a single hateful culture, and nowhere is there anything that requires a biased culture of policing to have occurred spontaneously. That is simply a straw man.
Unless you're proposing that police bias is organized (e.g., conferences about how to better oppress black Americans), then it is necessarily spontaneous if it exists at all. If you want to make a compelling case that racial disparities in criminal justice are principally caused by racist police, you had better at least provide a good explanation for how such a diverse occupation might become racist (in particular because the officers most likely to abuse force are themselves minorities). In particular, it is well-documented that African Americans commit more crime per capita than other American racial groups, so your case must sufficiently explain why these data are inaccurate or how to better interpret them.
Having read that, what would you assume is race and economic background is?
After criminal proceedings he was sentenced to community service.
Now, what would you assume is race and economic background is?
PS: Bias is insidious and really hard to control for.
However, most people will make assumption on race and class from things like PCP even though zero information about that in my example.
No, I had zero problem with your response (assuming it was genuine). As I said, it sounds like you are avoiding the question, however avoiding making those associations is the correct response.
Anyway, I was not asking if you thought they where guilty or anything. This was a friend of mine and he had zero problem admitting what happened.
White, because most people in the US are white.
PS: PCP has a reputation as a rural poor white persons drug. I wonder if you would have had a different impression if I started by saying he got community service.
I assume you mean "PCP" and not what circuit boards are printed on, yeah. You're right, I've never heard of a rich person taking PCP (cocaine, fancy liquor, etc. for them, right?) although I'm absolutely unsure of how PCP use breaks down racially. (If you have any source for those statistics, you've piqued my interest in them and I'd love a link)
Again, my guess of "white" was based on the fact that most people are white, and thus most cases of violence (without further stratification) are probably by whites. And what, do African-Americans not get community service? That probably wouldn't have changed my mind, and actually sounds a little bit racist.
Further, you pulled poor from thin air so you did make an assumption without evidence.
> Findings on social class differences in crime are less clear than they are for gender or age differences. Arrests statistics and much research indicate that poor people are much more likely than wealthier people to commit street crime.
> [...] most criminologists would probably agree that social class differences in criminal offending are “unmistakable” (Harris & Shaw, 2000, p. 138).
http://catalog.flatworldknowledge.com/bookhub/reader/3064?e=...
And again, as you admitted, PCP has the connotation of being used by poor people. I can't find stats on it right now, having just run out of time, but you even admitted it directly.
I didn't pull it out of thin air.
Now, the same identical reasoning happens all over the place in the criminal justice system. Making the link between crime statistics and crimes almost meaningless. Arrest statistics are just that arrest statistics and they don't tell you about who was actually committing crimes.
PS: I suspect you feel very confidante about his race, except I never confirmed or denied your assumption.
Edit: TLDR; Making a judgement with limited evidence is a really bad habit. People soon forget they chose something because it was slightly better odds and reinforce the judgement so something that may have been even odds to start with often feels much more likely over time.
Here's the big fallacy: "the two groups have different thresholds, meaning they are held to different standards." They are not held to different standards because they're different groups, but because of other reasons that indicate different loan default rates. So you cannot call this "discrimination". This is how things should be.
If you don't want to be banned, you're welcome to email hn@ycombinator.com. We're happy to unban people if they give us reason to believe they'll only post civil, substantive comments in the future.