Algorithmic decision making and the cost of fairness (2017)
arxiv.org
arxiv.org
Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.
[1]: https://www.economist.com/news/science-and-technology/217349...
Not paywalled.
That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement.
Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people.
And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions.
If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.
So when you ask 'x' people to independently judge something, let alone something that is multiple choice with a correct answer, you're going to get answers that are far more accurate than a single individual would give you. So looking at the average answer of 'x' people, comparing it to an AI system, and arguing that they're relatively close so individual people are relatively close to the AI is completely fallacious.
---
As a tangential aside, there's another quirk to the wisdom of the masses. When you let the people communicate and try to intelligently organize and use expertise to come to answer, this effect disappears and the final answer again tends to be very wrong. Kind of an interesting perspective on the current zeitgeist of society and work.
===EDIT===
The authors were obviously aware of the wisdom of the masses. Quoting the paper itself:
To determine whether there is “wisdom in the crowd” (7) (in our case, a small crowd of 20 per subset), participant responses were pooled within each subset using a majority rules criterion. This crowd-based approach yields a prediction accuracy of 67.0%. A one-sided t test reveals that COMPAS is not significantly better than the crowd (P = 0.85).
That's a quite silly p-value and on top of that I'm not sure how they claim their system actually controls for the wisdom of the masses.
Only if people are an unbiased estimator!!
For another example of it consider things like the television show, 'Who wants to be a millionaire?' It's a quiz show where one of the choices is for the participant to ask the audience. And the audience tends to do absurdly well on even the most esoteric questions, though independently they are certainly far from trivia experts. But very few are randomly guessing - their own experiences and biases leads them to entirely different conclusions. Yet somehow it produces the correct result time and again.
It's a strange phenomena and one that has to be constantly guarded against in anything involving sampling of people. This is a text book example of a study that gets destroyed by it.
>> Only if people are an unbiased estimator!!
> You'd think so, but that's not correct.
Either you're misinterpreting the technical term "unbiased estimator" here, or you are aware of some research that I would like to read.
In context, "unbiased" means that if you pick people at random and ask for their estimates, then on average the too-high estimates cancel out the too-low estimates (i.e. there is not a bias in one direction or another).
But people as a whole have poor understanding of many things. One common one that appears in social science research and is often replicated is that people grossly overestimate the size of the homosexual population in the US (the "wisdom of the crowds" often estimates it around 20% whereas best available polling data suggests 3-5%). Here's just one source for this phenomenon
http://news.gallup.com/poll/183383/americans-greatly-overest...
"Wisdom of the crowds" is occasionally reliable, but it should not be assumed to be reliable for any particular problem without verification. It often fails terribly even on problems that are not very esoteric.
Edit: changed phrasing of final paragraph
Where it works phenomenally well is in areas where biases have not been directly instilled into people. This does not mean people are unbiased, however. Again the knowledge of trivia is a good example since while the crowds can generally do phenomenally well even at very esoteric questions where biases would lead them to individually come to very different conclusions, yet they will invariably fail to answer ostensibly trivial questions like 'What is the capital of Australia?' You'll get Sydney, it's not. You'd likely get a similar result for things like the capital of Pennsylvania.
In a way I view the wisdom of the masses as analogous to machine learning systems. They do an oddly good job of providing extremely precise answers to a wide array of questions even when trained with models that do not directly represent the 'questions'. Yet you can also break the systems, at times comically, with certain types of queries designed to do precisely that.
And as was the case with machine learning for quite some time, I think people remain reluctant to utilize it due to the black box nature of it. The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse.
You are simply mistaken in your interpretation of the technical term "unbiased estimator". This has a specific meaning in statistics, and is required for the convergence property you specified earlier. From wikipedia [0]
"In statistics, the bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated."
In lay terms, this means that the estimator process "ask lots of people and average the result" is unbiased only if all the too-high errors, in aggregate, cancel out the too-low errors.
"The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse. ... "
I'm not sure if you even realize all the assumptions you're making. You are assuming, for instance, that 'guesses' are regularly distributed. That assumption may be correct in some cases - I expect in many it is not. Alternatively there is the possibility is for you to claim that you're referring not to the individuals in question as the estimators, but the entire group. In that case you've spent a lot of time saying nothing as it boils down to "people are only correct if they're correct."
https://www.scientificamerican.com/article/economic-inequali...
The "wisdom of the crowds" is pseudo-nonsense.
There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways.
Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.
What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth.
There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.
Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.
Another issue I have is that we don't know is whether statistical bias exists in the metrics they train on, something they discuss in the paper. We already know for example blacks are arrested and convicted harsher than whites for some of the similar crimes[0]. The paper just says that COMPAS predicts likelihood of violent crime, not whether that means they'll be arrested of crime. Moreover while Table 1 talks about a prediction regarding violent crime, Figure 2 talks about recidivism with regards to all crime, so which is it? It still seems nebulous to me what COMPAS actually is for.
I'd have to read it deeper than I have, but not sure how I feel about the paper.
[0] http://www.politifact.com/truth-o-meter/statements/2016/feb/...
2. "Your brain isn't doing anything a neural network can't do." Really, now?
3. The paper mentions "over 100 factors" the algorithm uses. Ad agencies have way more than 100 factors on me, and I still get irrelevant ads every day.
Humans aren't perfect, but they can catch things that stats won't, and vice versa.
They're both Turing machine approximators after all. Granted, ANNs aren't quite sophisticated enough to match the brain yet, but this capability gap is a difference of degree, not kind.
But I don't think the similarity of the brain to ANNs really matters to my argument, as interesting as this discussion might be --- so I've edited my original comment to make it more general and to focus on my higher-level point.
> I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile
I think we'd all agree here. The point is that both methods should always be used together, because each has huge limitations on its own.
The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Machines are already better than humans at other tasks for which we imagined humans exercised fine and subtle judgment: https://www.sciencedaily.com/releases/2016/04/160421133831.h..., so I see no reason to believe that bail-setting is some unapproachable problem domain.
> The point is that both methods should always be usned together, because each has huge limitations on its own.
I disagree. Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system, and fairness and transparency dictate that we be up-front about those biases. I see no evidence that a computer model couldn't reproduce biases we considered desirable.
Hormones, reflexes, instincts, empathy, etc. Again, human inference and learning are (as of yet) insurmountably more complex than statistical regression.
> regression (which I wouldn't characterize as "relatively simple"
I encourage you to study brain chemistry, and compare that with any stats course ever.
> Why would the use of some kind of regression (which I wouldn't characterize as "relatively simple") a problem if the overall result is nevertheless more accurate than the human equivalent? We judge models by their results, not their complexity.
Because there is no way to universally judge "model results". https://en.wikipedia.org/wiki/Confusion_matrix
Even a 99% true positive rate + 5% false positive rate can be worse than a 5% true positive rate and a 0% false positive rate, when you're talking about irreversible decisions.
> The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Yes, that's very true. Executing the model is easy, and maybe even works similarly in NNs and brains. Figuring out that "some kind" part is what we formalise as "learning", and turns out is the hardest part of this equation.
Again, clearly, current regression methods are nothing like human learning. Else, they'd be able to learn a language like a human, without seeing billions of annotated data points.
> Machines are already better than humans at ...
Neither the article nor the paper claim the machine performs better. Their strong claim is that the machine is faster.
> I see no reason to believe that bail-setting is some unapproachable problem domain.
I think justice is a more important domain for many people. My guess is it's because it can set precedent.
> Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system
For drastic contrast: unstated, socially-determined biases is what prevents nuclear wars. https://en.wikipedia.org/wiki/Stanislav_Petrov
More plainly: people have compassion, empathy, and can understand things about convicts that are (still) very difficult to encode into a computer.
Complexity doesn't mean better. Humans are absolutely garbage at statistical thinking. Even simple statistical regression almost always beats human experts. Let alone more modern methods. E.g. http://zaldlab.psy.vanderbilt.edu/resources/wmg00pa.pdf or http://lesswrong.com/lw/3gv/statistical_prediction_rules_out...
To be able to decide if something is fair, one has to be able to understand the process by which a decision is made, no? And not just a woolly, after the fact justification that a person might well invent to explain their decision. If we can't reliably look at the inputs and determine what the outputs would be, how are we able to decide if the process is 'fair'?
As someone who is looking to understand a system, the argument of "But it couldn't make a human decision" triggers alarm bells of "What you mean is: It couldn't make a non-rational, unfounded, biased decision influenced by whether the person is hungry or not". Is that how important decisions should be made?
Is this the sort of hormonal influence you find desirable? http://www.economist.com/node/18557594
All these factors you mention can be modeled as constant or contingent biases applied to the inputs. There is no magical aspect of the brain here. To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
> I encourage you to study brain chemistry, and compare that with any stats course ever.
You're hand-waving. This is not a useful discussion technique. If you have a point about how some aspect of brain chemistry produces decision outputs that are difficult to encode, please make it. Otherwise, please stop this aspect of your argument.
That something is "insurmountably more complex" than some other thing doesn't make it better than that thing. Sometimes, it's just insurmountably more complex for no good reason. The blind watchmaker doesn't build elegant watches.
> For drastic contrast: unstated, socially-determined biases is what prevents nuclear wars. https://en.wikipedia.org/wiki/Stanislav_Petrov
Unstated, socially-determined biases prompted Europe to slaughter itself for 200 years in absolutely pointless wars of religion. So what?
Nope. I said nothing about desirability. The question was: "What is the brain using except conditional probabilities derived from past inputs?"
> All these factors you mention can be modeled as constant or contingent biases applied to the inputs.
It's possible, but we can't do it yet. If you have proof that we already can, please share it.
> To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
Fairness is human-defined, and difficult to encode in a computer. I conjecture that human + computer will always result in more fair and safe results than either one in isolation.
> You're hand-waving.
Yep. The number of variables (and their interactions) affecting a human decision far surpasses the number of variables in any regression method. In a lot of literature, number of variables characterises complexity.
> So what?
My example is empirical evidence of a human element identifying a false positive better than a computer. Europeans fighting wars is not such an example.
Philosophically speaking, I think the "key component" that humans bring, besides still-superior decision-making in many cases (not all), is creativity. So far, I haven't seen any machine learning that can outperform humans in light of limited brand-new data.
You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself.
What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence?
Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.
In some other cases it might be justifiable.
The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.
The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive.
But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned.
>Justice should be individualized or it's not justice.
There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.
In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.
(The easy ones are male/female; racial differences are far less.)
Yes, but...
>>and there is no magic algorithm that bypasses this fact.
Maybe there is, it's just the method used wasn't able to find it either due to limitations of the method itself, not enough information or bias in the training set.
As a toy example: assuming you only have race and age to make your decision on then to optimize for public safety you need to include race to make good decisions. If you have race, age, number of friends who committed crime then maybe you don't need race anymore. The problem is that we are likely not getting enough data and then race is a proxy for that uncollected (and maybe uncollectable) data.
I understand that your argument is a toy argument so it doesn't make sense to discuss it specifically, but I feel it's important to point out that the issue here isn't just which variables are used to make the decision, but what the decision ends up actually being. That is, maybe you find that you can make a very good decision based solely on age and number of friends who committed crimes, and don't take race into account at all -- but then if this algorithm ends up yielding "yes" to most white people and "no" to most black people (even if your algorithm doesn't use race at all), you haven't solved anything. [edit: "solved anything" was poor word choice on my part, obviously you have solved something, but you remain in the state described by the paper]
Another issue is that while you can "whitewash" variables, it's very difficult to scrub race out entirely. For example, in practice we can't use "committed crimes" as an indicator because we can never know a ground truth: we'd have to use "were convicted of crimes" instead. Unfortunately, you're far more likely to be convicted of a given crime if you're black than if you're white, so you're already mixing race into your variables even if it isn't named. With the disparity in convictions, enforcement, etc., it's very, very difficult to come up with measurable signals that are not in some way already tainted by racial decisions.
It would actually solve the problem. It's ok if I give more "no's" to black people as long as black people are more dangerous in general. It's only not ok if I punish a specific non-dangerous black person just because they are black.
That's what fairness is: you get what you deserve because of your decisions and wrongdoing not because how you look or where you were born. That some groups end up with more convictions is expected and doesn't contradict fairness principle.
> Maybe there is
No, there isn't. We actually have a mathematical proof (which is quite simple) why this is impossible.
Specifically, following conditions can't be true at the same time: 1. groups differ in base rate 2. prediction isn't perfect 3. decision is correct at the same rate for groups 4. decision is correct at the same rate for groups, restricted to positive/negative class.
1 is a brute fact. Your toy example insinuates at 2. 3 is called calibration and what is usually optimized by machine learning. When people say algorithm is unfair, it usually means 4.
Even in your post you go from a statement that amounts to "statistically some groups have a larger number if dangerous individuals" to "black people are more dangerous". The two sentences do not mean the same thing!
The point stands: using statistics to meter out justice IMO amounts to collective punishment. Of course that leaves the question if whether more conventional methods are any better, but now we're opening up a new can of worms, namely what is the goal of criminal justice systems and how should those goals be achieved...
You cannot fix a problem by pretending it doesn't exist.
I'm not usually such a stickler for language, but in this instance it does seem to me that this kind of profiling serves to perpetuate that narrative of criminality being a feature of a group of people, and doesn't help to get at the causes.
(Not to mention that "public safety" still excludes the harms of centuries of slavery and decades of Jim Crow...)
You are right, but that's not the problem with the algorithm.
A critical assumption with large-scale data mining is that past trends continue - the problem is that the existing data fits the algorithm. It is just a conservative What-If decision maker operates on existing facts (i.e bad present day situation), just wrapped into code (or worse, encoded as opaque literal "biases" in a decision tree).
I see somewhat similar patterns in lending interest data (redline zipcode -> credit ratings) and the problem is that bigger the historical trend data, the less forgiving a "past trends" algorithm will end up being.
Since this ends up being a prisoner's dilemma, if you are a rational actor in this system and the system keeps playing a defect card on you, then the obvious move is to always defect - cut your losses.
Algorithms can't improve the job prospects of the people released. And that's not the algorithm's fault.
And therefore, the you're right - the algorithm can't change the world beyond its output result.
Not necessarily. It would be easy to use only recent data and ignore data from decades ago. The prediction accuracy wouldn't go down much as long as there is a large enough sample size in the recent data. It may even go up if the old data really is inapplicable now.
Then if a past trend stops happening, the old data gets purged eventually and only the post-trend data is considered.
It's the same root idea: the goal of the system is not to find the person who committed the crime. The goal of the system is, for each crime, to find a person who can be nailed for it. And you can manufacture a class of people such that whenever you need an offender you can go grab some of them and stand a good chance at getting a guilty plea or a conviction.
You start by selectively hyper-enforcing small violations against a chosen subset of the population. Get them for minor traffic violations (which you can ensure turn into arrest warrants by setting the fines and fees and other costs high enough!), get them for "paraphernalia" offenses (where you assume everyday objects in Person A's possession are evidence of drug habits while they wouldn't be assumed evidence in the possession of Person B!), get them for all sorts of things.
Now whenever there's a crime you just go pick some random people from the demographic you've been doing this to, figure out who you can bribe to testify against whom, and then march into the courtroom with witnesses and a defendant who has a thick file of previous "encounters" with the system, and off to a cell they go for a while.
And it gets even easier each go-round: people with criminal records don't usually have the resources to move on and rebuild their lives, so once they're released they're going to be right back where you found them last time. And they've got an even longer record now, which is just proof that you've been doing a great job in figuring out who these dangerous recidivists are! The model works!
I know it goes against some cherished beliefs, but this really is how much of the US handles "justice".
That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race."
In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?
Pretty odd concept of "fairness" if you ask me.
Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.
Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem.
Why should I be jailed because other people with similar skin color to me committed crimes?
Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?
People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc.
As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair.
In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations.
Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black.
As I said, I don't think many people would agree with that definition of "fair."
* I don't know exactly what the model inputs are, I just made these up for demonstration purposes.
Also from the abstract: "In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk."
The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
The authors propose a system to mitigate the problems here, though I actually don't really think that's the solution - these AIs should simply be abolished and replaced by objective criteria.
But that's not really true: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
>these AIs should simply be abolished and replaced by objective criteria.
What exactly is "objective criteria"? Previously human judges made these decisions. They are far more biased than any statistical algorithm.
I did see that in the abstract, but then the only follow-up in the actual body of the paper was:
"...among defendants who ultimately did not reoffend, blacks were more than twice as likely as whites to be labeled as risky."
Which is simply restating the premise in a roundabout fashion - i.e. that those with criminal records (etc) are more likely to reoffend, and black defendants are more likely to have criminal records. It's a bit of a long bow to draw to say the classification was "incorrect" post-hoc because "those from the risky group who did not reoffend were initially labelled as high risk".
> The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
I believe it's a straightforward logistic regression, which is about as objective as you can be. It's not "AI", it's not opaque and it's not indirect. It's a simple calculation of the relative odds of recidivism based on factors such as criminal history, employment, and so on.
Given that adjusting for race makes no difference to the model output, it is literally the most colour-blind approach you could take.
The authors, on the other hand, would have you treat two people differently purely on the basis of respective race.
Now obviously there are historical/legacy social issues at play, not to mention the broader question of what's "best" to eliminate disadvantage. But I suspect not many people will agree with the authors' that this would be "fair".
If I were pushing their line of thinking, I would be researching whether this type of "positive" discrimination actually reduces the race gap long-term.
That's ultimately the only thing that matters, and in my mind, it's a much easier sell to say "this isn't fair, but it's what we need to do to reduce the criminal overrepresentation and disadvantage that exist because of our history with slavery".
Meaning your original comment was disingenuous. Good job.
Define "race".
Well, if you don't take the authors' approach, you don't need to. You just continue assessing pre-trial detention on the basis of each individual's background.
> As noted above, a major criticism of COMPAS is that the rate of false positives is higher among blacks than whites [2].
Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.
The line you quoted misses a lot of context - the 'standard' is an algorithmic risk score with a single threshold. The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment). So this standard is what we're stuck with. The paper asserts that it isn't fair.
This is really the clincher. If the algorithm is effectively aggregating multiple human decisions, it ought to correct for known biases in those decisions.
Citation needed. The fact that certain ethnic groups have problems with crime and violence can be attributed to the fact they grew up in single parent households, are less educated, and have less income than other ethnic groups, all things that contribute to a higher crime rate.
> The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment).
I don't see how anti-slavery has anything to do with this. If you are convicted of breaking the law, as a citizen you will be subject to the same punishments as any other citizen.
Citation needed. Cops visit black neighborhoods more, sure. But they do that because they commit vastly more crime. Which is the cause and which the effect?
It's also a lot more complicated than that. People throw out studies that blacks are more likely to be caught for drug crimes. But they are less likely to be caught for more serious crimes. Possibly because they trust police less and are less likely to volunteer as witnesses. E.g. https://www.wsj.com/articles/the-underpolicing-of-black-amer...
Citation needed
Citation needed
It's would be difficult to have a discussion if you had to provide citations for every single well known fact.
A. Blacks commit vastly more crime.
B. The sun goes up in the morning.
You can argue for being racist if you want but you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
No one is doing that. The algorithm does not take race into account. It's just an assertion that it uses other factors as a proxy for race. I've not seen this proven anywhere.
And indeed, if it were true, you would have to admit that blacks are more likely to be criminals. Even after controlling for all relevant variables. Which goes against the standard narrative that it's just socioeconomic status or whatever.
>you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
It isn't racist. And the word racist is becoming almost meaningless because how often you overuse it.
Quite possibly - and if that were the case, it's certainly something to be addressed. As a non-American, I can't really comment.
But the paper doesn't really deal with that. It's saying that society should tolerate higher rates of recidivism amongst one racial group in the name of fairness.
I do understand the authors' thesis - I'm just saying that "fair" is a very subjective concept. I don't think many people will agree with the authors' definition.
You want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?
That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc.
[1]Given they themselves don't officially collect race information.
Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
I don't want to punish people for living in polluted areas therefore I don't include that data in the training set.
>>Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
I am not ok with doing that therefore I don't include address in the data.
Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there.
I'm not sure I can reconcile that.
1. in the first option, we're not applying the consequences random people, we'd be affecting those already found to have committed a crime.
2. in the second option, if we think we'd be affecting innocent people in option 1. why do we think we would not be affecting innocent people (good drivers who happen to live among poor drivers) in options 2?
That's to say, either both are right or both are wrong. I see the same effects in both.
There are several reasons for this from both technical and legal perspective.
It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.
If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe.
from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing.
most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.
I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing.
"You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with:
"You are not getting parole because you willingly associated with people who committed crime".
Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal.
Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack.
Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.
What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.
Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"?
>>What you should do is to keep the ratio of people classified as dangerous same among different groups.
Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
Now imagine if people were unfairly punished based on something they were born with and not where they lived.
No, the algorithm just gives negative weight to locations where dangerous groups tend to live. You infer race from that measure.
here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf European Union regulations on algorithmic decision-making and a “right to explanation”
Here's a very simple refutation:
Is past arrest history a valid factor in the model?
What if past arrest history is biased due to biased policing (Hint: it is, well proven)
How about zip code?
If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC.
Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.
Was interesting, and worth a look.
Beyond that, I agree with nabla9:
> I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.
What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color?
These are exactly the things that justice is supposed to be blind to.
I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.
* Base decisions only on things that would be relevant in a trial, like evidence and criminal history.
* Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect.
* Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle.
* Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy.
If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.
Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker.
It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists.
Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all.
But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them.
This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not.
A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
Also, A.I. ‘Bias’ Doesn’t Mean What Journalists Say it Means: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...
That's really the beautiful thing about our shift toward algorithms: we'll no longer be able to rely on these subtle social pressures. If we want to optimize a specific function, we need to put that function in the open, where everyone can see it. Then, as the linked article demonstrates, we can compare the idealized and actually-desired functions and perform a real cost-benefit analysis.