To study possibly racist algorithms, professors have to sue the US
arstechnica.com
arstechnica.com
[1] https://www.propublica.org/article/how-we-analyzed-the-compa...
https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
That's probably why the article is nothing but a bunch of anecdotes.
(which I'm not trying to weigh in on the quality of the main article, just pointing out that they did publish a discussion of their statistics. I can understand the editorial decision to break it into a separate article.)
https://github.com/propublica/compas-analysis
https://github.com/propublica/compas-analysis/blob/master/Co...
It even contains the same weasel words "almost statistically significant" for one analysis.
But the fact that you keep referring to COMPAS as "machine learning" when neither COMPAS (in the various papers I've read on their site) nor the ProPublica article describes it as such makes me wonder if you're missing a few key things about the actual reality. AFAIK, COMPAS clearly involves statistical heuristics, but its not clear the data pipeline used to create those heuristics, nevermind how it would feed into a machine learning system. And that's kind of the problem here, with so much of this algorithm being proprietary, there's no way to keep it accountable, other than the imperfect ad-hoc way that ProPublica devised.
You have a highly emotional reaction to their process -- calling them flat-out liars -- despite the fact that they were the ones who not only procured the COMPAS scores via FOIA, but the also wrote their own scraper to automate the collection of criminal records from the Broward court system. They then reconciled this with the databases from Florida corrections and jail records from Broward's sheriff's office, and the databases required manual reconciliation. They then packaged and posted all of this data for public use, along with their workbook, on to Github.
And this is the work, in your opinion, of a group of deliberately dishonest journalists who are looking to sell clickbait. And you can't stand that they went out and talked to people (anecdotes, as if data weren't the plural of anecdotes...and as if COMPAS itself isn't built on anecdotes because it requires practitioners to go out and interview inmates in person, for fucks sake)
Again, I haven't run through the numbers myself, so I can't agree or disagree whether ProPublica has conclusively argued that race is a statistically significant factor when COMPAS fucks up the prediction. Or if the "bias" that they speak of is more focused on the "hazard ratios" differentiation between racial groups, and perhaps that's not as sexy a bias as its headline warrants.
But I'll be honest. Your conflation of COMPAS with "machine learning", and your simplistic view of the law and the justice system -- "we've taken a flawed and racist human system and replaced it with a much better machine learning system"...um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics -- makes me think that you come into this fight with at least as raging a bias as that of the ProPublica journalists.
https://www.chrisstucchio.com/blog/2015/no_free_samples.html https://www.chrisstucchio.com/blog/2015/ab_testing_segments_... https://www.chrisstucchio.com/pubs/slides/crunch_conf_2015/s...
Now, most of the folks pushing bad stats are simply not understanding the subtleties of a complex field. PP, in contrast, did understand and ignored the results they got. Yep, that's dishonesty in my book. What would you call it?
I have no idea what you are talking about with "machine learning" - it's a synonym for "statistical analysis", leaning a bit in practice towards external validation of models without regard to internal validity. Which is, incidentally, what PP was trying to do.
If you want to draw some fine line where a random forest is machine learning but a Cox model isn't, be my guest. I'm sure that's a great reason to ignore PP's own analysis.
And you can't stand that they went out and talked to people...
No, I can't stand that they cherry-picked anecdotes to push a conclusion that's the exact opposite of what their own analysis claims.
Now if you want to criticize the analysis, go ahead. But that's not really a good way to defend PP - after all, if the analysis is bunk, doesn't that discredit the whole article?
um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics
If you want to make that argument, do it. But that's not the argument PP made.
As for my own biases, I'm pretty open and honest in my beliefs in the power of technology. I also back up my claims with math that actually agrees with me, unlike PP. Do you have any objection to what I wrote beyond the fact that you dislike the tone, and might define "machine learning" slightly differently than me?
You have no idea what machine learning is? Then why do you use that phrase. My interpretation is that it is differentiated from statistical analysis in that there's a system that adapts its heuristics to trained and incoming data. For someone to call COMPAS "machine learning" is to have a faith that this system would be adequately tuned and adapted to changing data. For example, if the factors that predicted higher recidivism for blacks were first devised in an era where low-level drug possession was a criminal offense...and now that era no longer exists, and there's no mechanism to adjust for that...I would say that would be a big fucking problem for every criminal who is processed under COMPAS today (regardless of the argument that drug possession crimes were more punitive towards minorities).
But we don't know because COMPAS is secret, which was the main thrust of this article. And it seems to perpetuate the biases of the "non-machine" justice system, except in an untransparent way.
I said:
> um, there's an actual Constitutional process to adjust things in our judicial system that shouldn't just be overridden because statistics
you said:
> If you want to make that argument, do it. But that's not the argument PP made.
Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be.
> Scores like this — known as risk assessments — are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts — as is the case in Fort Lauderdale — to even more fundamental decisions about defendants’ freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing.
I agree with you that 0.05 < 0.057. What we disagree on is how much of ProPublica's article is based on that, and whether that difference is enough to call them bald-faced liars, given that they acknowledge the limitations of their data collection and show their work and their data. They're attempting to reverse-engineer a black box that's being paid for by taxpayer dollars. They shouldn't have had to provide an anecdote-laced story with statistical analysis to pressure the judicial system for more transparency, but that's what they've chosen to do.
The Cox model fits that definition.
Perhaps you mean ML is synonymous with reinforcement learning, which is I think the closest thing to "tuned and adapted to changing data"?
My interpretation of ML is that it's about evaluating statistical predictors without worrying about their internals. Those are the fundamental advances that have given us all the classical "machine learning" algorithms - random forests, SVMs, neural networks, etc.
And the entire point of those techniques is that you can use them to evaluate a secret algorithm.
Did we even read the same article? The entire fucking article is about whether the justice system is unduly influenced by a proprietary algorithm that has been sold to the system (and paid for by our tax dollars) but is not as smart or objective as it is sold to be.
The entire article claims the algorithm is both inaccurate and racially biased, which their statistical analysis directly contradicts.
They never once made the argument that an accurate and unbiased model should not be used.
> Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts
Which is exactly what you would expect to happen if the algorithm predicts that black defendants are more likely to recidivate. Being in the group predicted to recidivate more will make you more likely to be predicted to recidivate. It would be quite unexpected for that not to still hold for the people who didn't actually recidivate.
And if being black does correlate with recidivism, what is the algorithm supposed to do?
Algorithms don't think for themselves, and AFAIK, the COMPAS algorithm has not been touted as one of those self-healing types. We wouldn't know either way because the algorithm is kept secret even though it is broadly applied in the public justice system.
But let me get this straight, you'd be OK with an algorithm just using the heuristic of "well, he's black, so give him a -0.5, because that's what blacks tend to do" to avoid the complexity of identifying possible confounding variables that are correlated with being black? That's actually illegal in many contexts (judicial system, housing, employment).
According to the PP article, race is not an input to the algorithm so it can't give -0.5 to black people solely for being black.
It's the other way around. If you only use the other variables but most of the variables that correlate with recidivism also correlate with being black, the algorithm will still predict higher recidivism rates for black people even if it doesn't know anything about skin color.
The more accurate the algorithm is the more it reflects the correlations that exist in the actual outcomes.
Which is a different matter entirely. I don't think there is any good argument that they shouldn't have to publish the algorithm.
> I'm going to assume that race is not actually used as a weight, I would still like to see the methodology used in determining the validity of the other weights, some of which may act as a proxy for race while not having a significant causal effect on recidivism.
The problem is they're all proxies. Homelessness likely correlates with recidivism, but it isn't the homelessness that causes it, it's the poverty and lack of a support network it implies. Someone with affluent close friends who chooses to live outside because they enjoy communing with nature is in a different situation entirely. But you can measure whether someone has a street address more easily than you can measure their support network, so it gets used as a proxy.
I'm not sure causal effect is the thing you want to hang your hat on anyway, otherwise the existence of racism would imply that race has a causal effect on recidivism and should be used as a factor, which I take it is not the outcome you were after.
The legal standard for showing that someone has been discriminated against is that it is more likely than not that they have been discriminated against.
They go on to give supporting analysis in form of the false positive and false negative rates by race, which is pretty compelling evidence. You claim to not believe that because you cant find it in the notebook but its literally right underneath the Cox model section.
I was intrigued by this article and went a step further to plot the ROC curves and the evidence is solid. It's messy, but you can see it here https://github.com/stoddardg/compas-analysis/blob/master/my_... in cell 78. Its quite clear that the algorithm is choosing a different point of optimization on the ROC for white people (a more lenient one) than for black people. A white defendant with a risk score of 5 is as likely to commit a crime as a black defendant is with a score of 7. That's an obvious case where you could simply relabel and be more fair but their algorithm chooses not to.
I also hate when people abuse bad statistics and reasoning to sell page views.
Your analysis might be right - if so, that's interesting. I'll take a closer look and write a followup piece if true - among other things glancing at your ROC curve suggests they are pretty close, and perform better for whites in some regions and better for blacks in others. But it's 7:30AM (pre-coffee) and I haven't looked closely yet.
But since PP did not do any of this, my criticism of them holds - they ran an NHST, got the wrong result, and then spouted a bunch of anecdotes instead of admitting that their analysis went against what they wanted to find.
> Finally, the article includes a table of false positive probabilities (FPP) and false negative probabilities (FNP). This may or may not be evidence of bias - the authors would need to run a statistical test to determine that, which they don't. In fact, I can't even find the place in their R notebook where they did that calculation. Is this the result of bad statistics? Is it merely random chance? Who knows!
Looking at PP's Jupyter Notebook, the calculations seem to be performed at lines 50 onwards (if you're referring to the table that I think you're referring to).
FWIW, those "weasel words" you allege are in the writeup of the methodology, where the audience is expected to follow along and see how the 0.057 is calculated. I'm not sure how you're interpreting that calculation...My read is that it's not the bedrock from which all of the other analyses are based from. Where in the story do you see that particular calculation being used as the main (or even ancillary) thrust of the piece?
But I'm sure they look convincing to ProPublica's readers who are not statistically sophisticated.
ProPublica, in line [36] of their analysis, fit a Cox proportional hazards model on pairs (score, race) with three possible scores (low, medium, high) and six possible races (Caucasian, African-American, Asian, Hispanic, Native American, and Other). Thus there are 18 total groups; each person belongs to exactly one. This model tries to estimate the relative risk of recidivism for each of these groups. The "base rate" cancels out when you fit this model, so these risks are all expressed relative to a baseline; this baseline is chosen to be (low, Caucasian).
There are 17 degrees of freedom here. It doesn't matter what basis you choose (as there is no regularization), so all that's really happening is you're counting recidivism in 18 buckets. The basis ProPublica's formula chooses is 2 = 3-1 terms for score, 5 = 6-1 terms for race, and 10 = (3-1)x(6-1) terms for the "interaction"; this reflects their choice of coding using two categorical variables. But you may as well just have 17=18-1 terms each of which expresses the relative risk between each group and the baseline; this would be the equivalent of coding using one categorical variable for the entire pair. The advantage of such an expression is it'd be harder to make the mistake both you and ProPublica make: the "interaction" terms are not the droids you're looking for!
In the "Background" section of your "How to lie without statistics" article, you write: "the size of the second coefficient (relative to the first) is directly proportional to the degree to which the old predictor was biased". The first coefficient is the coefficient in front of the "old_predictor" variable (here, "score_factorHigh/Medium"). The second coefficient is the coefficient in front of the "is_black" variable. I'm honestly surprised, given how clearly you seem to write here, that you then focus on the wrong coefficient later.
The relevant coefficient is the one on "race_factorAfrican-American"! That coefficient is 0.27879 and very significant. In other words, the algorithm is "biased in favor of blacks". To be ever more precise, the numbers you're actually looking for are "race_factorAfrican-American + race_factorAfrican-American:score_factorHigh" and "race_factorAfrican-American + race_factorAfrican-American:score_factorMedium"; as one would expect, the interaction terms are correction terms that aren't the dominant effect.
Your error is a bit more understandable given quite how much ProPublica messes up line [37]. They forget the race terms entirely!! The correct calculation for "Black High Hazard" is exp(0.27879 + -0.18976 + 1.28350) = 3.95 and "Black Medium Hazard" is exp(0.27879 + 0.84286 - 0.17261) = 2.58; note that I included the race terms! Their sentence should read "High risk white defendants are 3.61 more likely than low risk white defendants [to recidivate in a given time period], while High risk black defendants are 3.95 more likely than low [risk white defendants...]". (They wrote 2.99 for the italicized term. For medium risk, they give the numbers as 2.32 white versus 1.95 black instead of the correct 2.58 black.)
To specifically isolate where your analysis is incorrect: two of your three conclusions are incorrect. Conclusion #1 is correct. Conclusion #2 is not: the analysis directly suggests that it is biased in favor of blacks and the interaction term is not the dominant effect. Conclusion #3 is also incorrect: that term is how much more likely (low risk, African Americans)s are to recidivate than (low risk, Caucasian)s. If blacks had a much higher recidivism rate but it was fully captured by the score, this term would be zero!
I'm genuinely curious what you think about this. I'm not in the US and it sounds awfully like you don't see a distinction between the two cases.
Or you genuinely believe yourself to be neutral and impartial - which would of course be ridiculous ;-)
You can write so as to describe an incident, you can get quotes to balance out a view. And what you choose to EXCLUDE matters just as much as what you include. Even down to how images are cropped. A writer who is self-aware of bias can do an outstanding job by understanding that what they think they know they may not actually know.
For example, a topic such as climate change, I may realize both that I have a bias AND I don't have very good info of the other sides of an argument. If I were being a responsible writer that should mean that I need to study up both on the topic AND on the view points, separate the valid from the invalid and go from there. Taking my own side on faith or from a narrow perspective is my undoing if I'm to be a reliable source as a reporter or journalist to 'inform'.
Also this is the age of clickbait so it's even easier to detect bias because the headlines are begging for scrutiny.
'bias', 'both sides', 'balance' etc all rely on the idea that there is a reasonable middle ground. Aren't you assuming the very thing I'm asking you to justify?
For example in the UK - the BBC has been criticised for 'false balance' in it's science reporting. i.e. giving equal weight to mainstream views and fringe views: http://www.telegraph.co.uk/culture/tvandradio/bbc/10944629/B... (this is from a right wing newspaper - bear in mind that the right in the UK doesn't strongly support climate change skepticism in the same way they do in the US).
Also I've found the concept of the Overton window to be useful and fascinating: https://en.wikipedia.org/wiki/Overton_window
If I properly quote Wm Shakespeare “You speak an infinite deal of nothing.” (Merchant of Venice) then within various limitations designed for academic quotation use, I don't owe ole bill a penny nor do I need a copyright license for that academic purpose quotation.
On the other hand if I were a musician and sampled 2 seconds of a song, supposedly I need a legal release or the owner of the sampled song will have our government give me the economic death penalty in the courts.
The CFAA is kinda weird legally having the infinitely powerful government team up with the infinitely powerful megacorporations just to make sure individuals follow arbitrary corporate defined local rules of behavior. I'm having trouble finding a non-CFAA example in the real world. I guess it would be like if a shopping mall privately published a rule about not saying "merry christmas" for inclusiveness reasons, then the local police shoot in the head any private citizen at the mall caught saying "merry christmas".
Next thing you know they'll be running extension cords over your fence and into your windows in order to power their experiments for academic purposes.
If you're a site owner and you're using bots to screw over your advertisers, that's one thing.
But otherwise, what's their claim here? I'm sure web crawlers interact with ads all the time in the course of their crawling, and no one's complaining about that.
That's not a real, legal, thing. "Click fraud" is just what some businesses term interactions that they might not pay for.
There is no legal, or moral, requirement to only click links if there is some chance that you will increase the income of some "advertiser".
Though the harms it causes are similar:
* advertiser is paying for clicks/impressions from a bot rather than a user
* publisher/network sending lower quality traffic to advertiser faces reputation and financial risk
Likewise, depending on the situation fake social media profiles also can harm the quality of said network.
In terms of damages caused by the researchers' action... clicks would be the most harmful. Especially against insurance, loans, and mortgages which they seem to be interested in. Those have some of the highest CPCs around. (e.g. http://www.wordstream.com/articles/most-expensive-keywords)
Personally I think it's a little unlikely, but criminal prosecution can all be very capricious and dependent on local and regional priorities. And so much the better that their suit against the US might bring about limits of CFAA via precedent.