A Discriminatory Algorithm Wrongly Accused Thousands of Families of Fraud
vice.com
vice.com
> Crucially, she said, we should place blame on the human individuals behind the creation and use of the algorithm rather than reify the technology as being the main driver.
> “Systems and algorithms are human-made, and do exactly what they’ve been instructed to do,” she said. “They can act as an easy and sort of cowardly way to take the blame off yourself. What’s important to understand is that often algorithms can use historical data without any proper interpretation of the context surrounding that data. With that pattern, you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop that increases social problems that already exist.”
As someone in an adjacent institution this affair has been frustrating to watch. In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors.
Interestingly though the administration resigned over this affair, it does not look like they'll suffer much over it in the upcoming election this wednesday. The responsible prime minister is likely to get a fifth term if things go as the polls are predicting.
We learned from 2000 years of human history - war, disease etc. - to create a transparent system of checks and balances. These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency".
I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
Well, the same 'we' that push voting machines and other crap like locking government records up in proprietary software.
I push for algorithms specifically because they're (as a written document) transparent. If a human has to judge a historical context you're never going to get repeatable results. But I want the system to be able to diagram, with weights, all the inputs that went into something before I'll trust an automated version.
And "cost." As in, "I'd rather get crap news online for free than pay for a newspaper or broadcaster to keep the government honest."
Part of it comes down to an effect that's been known for a long time with human-computer interaction.
Whenever you have a machine in the decision loop of a job, many workers WILL DEFER to the machine whenever there's a question of whether something is wrong or right.
By deferring to the machine, the workers absolve themselves of accountability and shift all responsibility to "the system". I think this is often deliberately "baked in" by management to customer service of all kinds. You can see this in a small way if you've ever had to call an internet service provider in the USA to deal with a billing mistake.
What I find very problematic in the use of automated systems for fraud detection is taking the system decision as final and not offering an easy and clear way of appealing decisions.
I dislike how people just carelessly make these claims without justifying them. For instance, another (far more likely imo) possibility is that it converges into certain state.
Back in autumn 2013 I got a letter stating that I had received too much in child care benefits, with a request to pay it back. On the back of said letter was the text 'If you are still entitled to child care benefits and want to settle this amount against upcoming entitlements, you do not have to do anything'. Given as I was still entitled to benefits, I proceeded to not do anything. A few weeks later I got a reminder, once again stating that if I wanted to settle, I wouldn't have to do a thing. The reminder was followed by a final notice with opportunity to pay, once again with the exact same text on the back of the letter. So, obviously, I once again did nothing.
Then we got a 'Dwangbevel in naam des konings', i.e. a writ of execution in the name of the king. Red envelope. No added text. So I had to pay up. Now, as you may well guess, the tax service had already also started settling against the benefits that I was still entitled to! This prompted me to write a strongly worded letter, asking the tax service in no unclear language what I should have done differently, and requesting them to not dock me for that particular money twice. Literally the only reaction to that letter from the tax service was in silently restituting to me the double payment.
Now, I'm a reasonably well paid and well educated software developer, who just happens to hold two nationalities. I am perfectly sure that I would not even have had a reminder had I just had the Dutch nationality.
I could have paid in the first place and not settled against upcoming benefits; I was just lazy. However, imagine you're scraping by on minimum wage and are put in such a situation. There is no other way you could have acted than I did, but you also would not have been able to pay up to the writ of execution. And there would have been no recourse!
Add to that the fact that in many cases, the people hadn't even actually been paid too much, as I had, but merely been tagged as possibly fraudulent and put in to the system for reclaiming paid benefits, so the government could take a stance of being hard against benefits fraud while figuring out if any fraud had taken place in the first place.
"In one of the more egregious examples of the lack of humanity in the authorities’ approach, a report from Trouw revealed that the tax office had baselessly applied the mathematical Pareto principle to their punishments, assuming without evidence that 80 percent of the parents investigated for fraud were guilty and 20 percent were innocent. "
So this is actually exactly backwards. On top of being frighteningly amoral, I mean.
Yes, computers were used in the decision process and no I don't believe them to be algorithms.
It was specifically coordinated and covered up from the top. As your highlight shows, it was specifically designed this way. Which means it was a business process.
Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant. If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge. This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
The moral of the story is that stats describing a cohort cannot be projected onto individuals in that cohort. The data flows the other way; individuals comprise an average, no individual /conforms/ to an average. They may happen to coincidentally be average, but no causation has occurred.
Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"What would be wrong though is to blindly take that probability for the fact."
And it's right: your algorithms can use all sorts of indicators to flag suspicious cases, but in the end the evaluation needs to be done by a person on a case by case basis. As long as the investigation on the single case is performed "silently" (i.e. without causing any loss of time or money or any annoyance to the person who is investigated) I don't see the issue. Otherwise it's the same as saying you cannot sort rows in an excel sheet according to some criterion and start looking at the data from where it looks more promising.
As a concrete example of where this can go wrong, let's say that (hypothetically) 1% of people commit fraud, and that this prevalence is not correlated with race. But, due to past discriminatory practices, more people of one race than another are convicted of fraud. Then, this algorithm would perpetuate this discrimination, since, say, it would identify 80% of fraud in the discriminated group but only 50% of the non-discriminated group.
Also, dual citizenship is not a protected class last time I checked.
A model that does not incorporate all the relevant information will have more false positives. The practical consequence of doing what you suggest is, cetaris-paribus, that you will incorrectly penalize more people.
> There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts.
Definitely. Thats where courts and human review come in. The mistake the Dutch made here was not the variables in the model, but that they had limited human review of a model with (I'm assuming) low predictive power.
> Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"Guilty beyond a reasonable doubt" (or whatever the Dutch equivalent is) implies exactly the Bayesian calculation you are rejecting. That said, I agree that a machine learning model with limited inputs is not sufficient to determine guilt.
A lot immigrants from Marocco and Turkey own real estate in their country of origin. Their IRS doesn't readily share information with ours.
See: https://www.uwv.nl/zakelijk/images/handreiking-inkomen-en-ve...
That's insane. Men commit 99% of violent rape, imagine not being able to say that. And I say that as a man, innocent of rape.
Similarly, if people with dual-citizenship smuggle more you're wasting time by treating everyone equally when looking for illegal imports.
> If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge.
Yes, any black-box trial is crap. But a transparent algorithm based on real data that a certain subset of the population is more likely to commit a crime, which is used for proper planning and investigation, is not.
> Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
You have one proper point about terminology and then an appeal to rage.
You are right that nothing my demographic does (men, raping) makes me more likely to rape. But you're wrong later where you say "[claimed] /you personally/ were more likely to have" because, yes I (by the nature of being physically capable of rape) am more likely to have committed it than other people. To me I'm not a statistic, to you I literally am a population sampling.
> Would you accept someone saying ...
If they're right, yes. Otherwise you're just saying that my outrage should trump the truth.
If a woman wants to organize a women's only bus, or hotel room, because she fears the harm I as a man could commit I shouldn't have the right to force myself on her.
> This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
No, the problem with stop and frisk is that it was (allegedly, I'm not from there) used by racists to target black people. If it was used as commanded by an algorithm then it would only happen where data showed an actual correlation.
In this sense, dual citizenship is basically code for being Moroccan.
What if somehow we as a society magically get equality overnight and there's no more connection between race and wage? But you use historical data where they were connected, still excluding wage data, and so even now your model connects race and fraud.
In this hypothetical example, I'd argue it would be inethical to include race without wage. Hell, wage is only going to correlate with fraud via yet other variables. Maybe an ethical system needs to go out and gather those measurements too.
(side note, in an ML system with regularization, even if you do include wage in your data, the regularization might pin some fraud on race anyways.)
The solution is to acknowledge that there are always going to be omitted variables and either a) be extremely careful and rigorous in your data gathering, design, and roll out or b) don't try to automate this thing, leaving it the slow expensive way where people can gather facts as needed, case by case.
I really hope that at some point, the world starts talking about how Mohammed and Fatima ended up there. This is a part of what happens when you spend two decades bombing villages.
No one deserves to be the target of racism, no matter how bad or good their circumstances are. If someone has done something worthy of derision, then justly revile them for that, but leave their race, ethnicity, religion, sex, gender, orientation, disability, etc out of it.
[1]https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-... [2]https://www.statista.com/statistics/525804/netherlands-large... [3]https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-see...
Two centuries.
How petty to now attack the poor in such a cowardly manner...
For other examples: field drug test kits--they really only say "this might be drugs", but the prosecutors use jail to try to get people to plead guilty rather than promptly doing a proper test.
Also, the ubiquitous breathalyzer--it's actually coded into law as proof in many places but it's not. There is an inherent biological flaw that makes the readings inaccurate.
* A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6
* Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092
* When the dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abs...
* When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-rac...
* When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automatio...
* When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though:
* When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it...
* When the above is compounded for people of color according to studies which show that facial recognition systems misidentify dark-skinned women 40x more often than for light-skinned men. http://news.mit.edu/2018/study-finds-gender-skin-type-bias-a.... Another study showed false positives can be 10x to 100x more frequent for Asian and African American faces compared to Caucasian. https://www.nist.gov/news-events/news/2019/12/nist-study-eva...
* When an algorithm blocked kidney transplants for Black patients. https://www.wired.com/story/how-algorithm-blocked-kidney-tra...
* When clinical algorithms include “corrections” for race which directly raise the bar for the need for interventions in people of color, such that they then receive less clinical screening, less surveillance, less diagnoses, and less treatment for everything, including cancer, organ transplants, birth interventions, urinary and blood, bone, and heart disease. https://www.nejm.org/doi/10.1056/NEJMms2004740
For instance:
> * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males.
It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
> * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon.
A higher chance of any offense, or of being a burden to society? Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. If black kids were being enticed into crime we certainly want to know about it. If it's a subpopulation rather than an area the causes are likely different and good decisions only come from good data.
> It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
Not a single example I gave was the result of ill intent. That's literally the point of my list. There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
> A higher chance of any offense, or of being a burden to society?
I provided the link which answers your question. In this case, it predicted a higher likelihood of future crime for a kid who attempted to steal someone's bike and scooter, than for an adult who had shoplifted, been previously convicted of armed robbery, and served 5 years in prison already.
> Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon.
Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison. In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery. Even worse is that the algorithm was being used to determine the severity of their respective punishments. So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
Because the two things I pulled from your list aren't systematic. One was local to a specific health system and about a specific condition, and the other about kids from a certain area. Neither was just about black people in general. Your list is predominantly racial and I thought that was the point. This medical thing is really about any group not being recognized as different enough from the whole, women get this a lot, but even men do in areas where women are the primary sufferers of something. (Breast-cancer for instance.)
Also, to distinguish things that are intentional (redlining) vs absolutely unforeseen but still problematic. A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
> There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
No, they're an example of non-systemic bias. The aren't broadly across any system, or symptomatic of a general attitude. Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
> Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison.
It is a separate topic, but yes. Being put away unjustly, and in the wrong place or severity for your crime hurts. But not serving a sentence for crime is just as bad, if not worse. I'm not against jailing criminals, just doing it where it doesn't serve us or them.
> In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery.
Could be. Without knowing the severity of the crime predicted though it's kind of meaningless.
> Even worse is that the algorithm was being used to determine the severity of their respective punishments.
Well, without knowing the prediction you can't know if a stiffer sentence would have encouraged or deterred future crime. If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
> So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
Ah, no, that wasn't the point. They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in.
> A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
I honestly don't know how any of the examples posted are deniable, they're all well founded with research and evidence. But you're right that adding more to the list can only help.
> Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
That's not what systemic bias means.
"Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
https://en.wikipedia.org/wiki/Systemic_bias
Also, examples are evidence which support a theory (of systemic bias in this case). Any one data point is rarely intended to be absolute proof of that theory, much less a description of its scope in its entirety. All of these examples do indeed provide evidence toward the existence of systemic bias within systems without fully describing or proving the extent of the entire scope of systemic bias within those systems.
> But not serving a sentence for crime is just as bad, if not worse.
The article I linked explains that this is not what the algorithm was used for. It was not used to decide whether or not to sentence them, it was used to help decide the severity of their sentence.
> If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
You seem to be agreeing that this was in fact a badly biased algorithm, since it was recommending a stiffer punishment for a kid who attempted to steal a bike than for a repeat felon, previously convicted of armed robbery, who shoplifted.
> It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad; see the description above. Regardless of whether or not you think stealing scooters is a good prediction, the fact here is that it rated the likelihood of future crime as higher than for someone who was already a career criminal.
> Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
If only. Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues?
The medical one made me think about the similar case of differing heart-attack symptoms in men and women. In this case actually because doctors just focused on men.
> That's not what systemic bias means.
> "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
That just shifts the definitional question to the scale of the institution, structure, or process.
At one end, if everyone follows the same rules it's a system, but what about if one clinic got bad data, do we write that up as a systematic failure? That's okay if it's your definition, but cases at the other end seem a lot more important and a lot more amenable to systematic fixes.
> If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad
Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
So I was wondering if you'd focused on black issues to make the list.
> Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected?
I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror.
> You seem to be agreeing that this was in fact a badly biased algorithm
No. But not a good one either. More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell.
The 'good' case would be if someone built a criminality model and the city was trying to work with police and communities to intervene in a predicted pattern. It's not unreasonable that the societal harm from a non-criminal becoming criminal could be worse than an existing criminal remaining that way. So modeling and discussing this isn't bad, even if the data has a racial component and some of the questions are of bias.
But yeah, to recommend sentences, total crap.
When talking about systemic bias, I think it's less about intent, but that is one of the things that makes it so dangerous in terms of the level and longevity of impact; it's much more difficult to fix when there is no single bad actor at which to point the finger. This makes it easier for people to deny, either outright or at least in extent, and makes it harder for the systems to be fixed.
I don't know that I could place one type of demographic bias over another, but I think it's better for people to understand and acknowledge systemic bias of any form, if we have any hope to rectify any of them. I also think a lot of them are woefully entangled. For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference. I have plenty of examples of this, but don't want to dilute the impact of that last thought by going into detail.
> That just shifts the definitional question to the scale of the institution, structure, or process.
I agree. Systemic bias is everywhere, for almost all scales of systems.
> Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
Yes, you're right, I meant flexible. We can debate the severity of the impact of some form of systemic bias based on the size of the system (and the population or sub-population impacted), but to the individuals who are the victims of systemic bias, the result ends up being the same. At the end of the day, I don't think I'd fault any individual for taking up the cause of identifying and rectifying bias at any scale.
> I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Yeah, I didn't really provide much context for why I specifically keep a list. It's not intended to single out racism, and in fact, it is intended to focus on unintentional codification of bias, because I think there are more software developers out there who are at risk for unintentionally codifying bias into their algorithms than there are those who may intentionally encode it.
The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context. It's a reminder that what we do can have real impact on real people in ways we never imagined.
This is especially recurring as a theme for machine learning and AI systems which are taught based on limited datasets which often systemically under-represent those to whom it will end up being applied.
> Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
I think this is a good idea.
> So I was wondering if you'd focused on black issues to make the list.
I did not specifically focus on that. I started tracking the list I think 2 or 3 years ago, and I think issues of systemic racism have been especially visible in the mainstream in this time period for reasons I won't get into. So, it's largely a reflection of the systems I've been made aware of through research and publication.
> So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected?
I have no idea. I just know that I've not yet been able to find any real systemic bias, at least in the US, against rich, caucasian males. If the question is, which slice of demographics is the most biased against, I think there are some studies which have looked into that, but I couldn't say myself.
> I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror.
There is certainly a bright side in these things being identified as problems which need to be solved. However, even with the kidney example, I don't know that it's a success as much as further evidence of how difficult these things are to rectify at a systemic level, since the vast majority of doctors using the still-biased eGFR formula have no idea that it has this problem.
> More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell.
I mostly agree with this sentiment. The biggest issue I see is that most of the models weren't built with the intention of being biased or badly predictive, and those selling them often don't know until after they've already been sold and broadly adopted. Fraud generally requires mal-intent, which most of these models and products didn't have, at least if we're being generous and optimistic. Negligence is probably closer to it, but I think difficult to prove in such a new frontier.
I just finished reading a book called, Weapons of Math Destruction, which talks about the damage models can do. The author posits a set of tests to tell whether the model is beneficial or destructive. One of the hallmarks, the author argues, of a good predictive modelling system is one which includes feedback into the system as a result of its predictions. Many examples, especially those which alter the outcome by making the prediction in the first place (e.g. criminal sentencing based on predicted criminal activity, excluding candidates from the hiring process, firing teachers based on opaque models), have no feedback mechanism for directly determining how successful their predictions were.
> But yeah, to recommend sentences, total crap.
One of the other primary tests is dependent on the context of how the model is used, so it's not really reasonable to try to determine the benefits of a model in isolation with no consideration for the appropriateness of its implementation and utility. In this case, it's hard to defend a model with bad predictive data based on the presumption that it could have been more beneficial with another use, when the reality is that it was only sold with one use and that one use was harmful. Though, even this ignores the fact that it had no built-in feedback mechanism to determine how good a job it was even doing with its predictions in the first place.
Thanks for approaching it this way. I think we often (societally, looking for blood) want to find a bad actor and are intentionally blind to bad things that just sort of happen at the edges, but need oversight to find and fix.
> For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference.
It doesn't help the sufferer, at the time, to be told that someone else would be suffering equally, but it does help fix the problem I think, to realize that it's circular via poverty or whatever, not simply racial, because in some areas and at some times it has simply been racial and it hugely changes how you fix it. If it's overt you can't just offer change, you have to prevent further damage during the repair process.
> The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context.
Have you ever read comp.risks? I really like it as a source of Therac-25 type stories (across all fields) that engineering types should think about when building things.
> I just know that I've not yet been able to find any real systemic bias, at least in the US, against rich, caucasian males.
Is Twitter not a system? :D
It gets a bit fuzzy with bias against the majority. Every model that isn't right disadvantages everyone and the majority is part of everyone. So bad drug laws impact white people too. But because actual race is only encoded in one direction (affirmative action, "positive" directions) then anything that impacts white people also impacts everyone, whereas there are often specific laws (such as for constructing "The Projects" in the first place) that do directly exclude whites from the harm they caused. So subgroups definitely experience more exclusive problems, even aside from the amount of problem.
> how difficult these things are to rectify at a systemic level, since the vast majority of doctors using the still-biased eGFR formula have no idea that it has this problem.
I think it's just that the story is best told from that moment. That's the OMG. From there it improves, and I'm sure they sent a copy of the report worldwide asap. But no solution is ever 100% so there's no wrap-up party and it will never look done and solved. (Even one doctor who didn't check their email...)
> Fraud generally requires mal-intent, which most of these models and products didn't have, at least if we're being generous and optimistic.
Probably, but if they're saying "our product does X" maybe there's something to grab onto and investigate. Maybe they did misrepresent it.
> I just finished reading a book called, Weapons of Math Destruction, which talks about the damage models can do. The author posits a set of tests to tell whether the model is beneficial or destructive. One of the hallmarks, the author argues, of a good predictive modelling system is one which includes feedback into the system as a result of its predictions.
Good point, and thanks for the book recommendation.
A lot of things aren't amenable to that though, because the hypothetical city/community meeting can't take years to watch the outcomes and train continue to build a model, they've got to work from historical data up to that point and make policy decisions in the meeting.
> One of the other primary tests is dependent on the context of how the model is used, so it's not really reasonable to try to determine the benefits of a model in isolation with no consideration for the appropriateness of its implementation and utility.
I've been seeing that as the predictive vs directive use. City planning instead of sentencing guidelines.
It's a big job, but good work. Keep it up!
Younger people are more likely to re-offend than older people. Remove race from the situation entirely, and this is still the expected result. There's zero reason to think this algorithm was based with respect to race.
1. Even after accounting for criminal history, recidivism, age and gender, black defendants were still scored as much more likely to re-offend.
2. It incorrectly predicted that black defendants would re-offend much more frequently than white defendants.
3. It incorrectly predicted that white defendants would not re-offend much more frequently than black defendants.
Considering those three things and that they have refused to give any details about how the algorithm works, I see zero reason to give them the benefit of the doubt. Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be.
The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
* Out of 100 white people, 5 were labeled high risk.
* Out of 100 black people, 20 were labeled high risk.
* Out of the 5 white people labeled high risk, 4 re-offended.
* Out of 20 black people labeled high risk, 16 re-offended.
In either case, someone labeled high risk had the same likelihood to re-offend: 80%.
"It incorrectly predicted that black defendants would re-offend more frequently than white defendants." This is technically correct, but not because the algorithm was bad a predicting rates if re-offending. It's because there was higher rates of re-offending. The likelihood of re-offending among someone labeled high risk is the same.
This kind of objection seems like a blanket rejection of any system that produces an inequitable outcome. But the reality is that rates of re-offending is not equal. Even a perfectly accurate prediction of re-offense is going to predict higher rates of re-offending among men. Because men re-offend at higher rates. This isn't sexism.
That doesn't mean we shouldn't recognize the disparate impact of incarceration on underprivileged people. But simply concluding bias due to inequitable outcomes is simplistic.
I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
Again, not right at all. These things heavily correlate with crime. A broken home is the primary indicator of someone's future success, even over a 'better' but broken home.
I'm male, and not a rapist, but having a penis heavily correlates with rape. I suggest you do not pick me, or any male, to watch your children. Sure it's rude to the innocent, but oh well, a little rudeness vs potential harm.
> I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
Then nobody will ever follow the law. If it's illegal for me to reject a potentially bad babysitter for something I know about them that could risk my child's safety I'll happily lie and say I didn't like their haircut.
You'd get further if you tried to identify these people and treat them better - grants to move out of bad neighborhoods, to get educated, to get pardons for unrelated crimes, to get counseling for abuse, etc - than with this "wrong unless it's 100% identical, equity-of-outcome" thing.
It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
> It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
And that's not what actually happened. A guy got arrested on bad data and then released, we know some kids might be at risk because they're from broken homes, etc. Nobody was jailed for their parent's actions, and nobody even proposed it. Some trends are worrying, but you act like they're the intent of the entire system not just some scammy products that a company is pushing.
Voting machines, for instance, should all be burned. But I understand that most people don't know why and support them for convenience. I think they're anti-democratic but I don't think people are evil for using them. You should try to get a similar perspective.
> Nobody was jailed for their parent's actions
Not jailed, but they are being kept in jail because of them. That is the same thing.
I am not claiming that anyone is evil. Just that this is unjust and that they need to stop doing this. You are putting words in my mouth.
Thankfully, the article I linked to looked at both:
White African American
Labeled Higher Risk, But Didn’t Re-Offend 23.5% 44.9%
Labeled Lower Risk, Yet Did Re-Offend 47.7% 28.0%What do the non-black box findings from the area show? If blacks are a poor demographic in the area it might be true. (Crime tracks poverty, not race.)
But yeah, certainly don't pay anyone for, or use, a black-box algorithm.
> The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
Nope. That's perfectly fair to look at. For instance, a broken family is another predictor. It's not fair, but being a victim increases your chance to offend. Similarly, the number one predictor of child sex crimes is to have suffered them yourself. If you have a child and are picking a babysitter, skip the one who was molested.
fwiw, those things don't correlate with race, they correlate with poverty which correlates with race. Broken families are more likely to be poor and abusive.