Hacker News Reader

Comment by beat | Hacker News Reader

The money quote: "I don’t care what your politics are. If you’re building a data-driven system and you’re not actively seeking to combat prejudice, you’re building a discriminatory system."

This is extremely important. The weight of existing prejudice drives and reinforces future prejudice. Garbage in, garbage out. If you start with biased, flawed data, and you look for patterns like the biased, flawed data, you'll just add to the biases and flaws. You need to account for the poor data quality explicitly.

View on HN

This proposal shows more concern about being politically correct than actually correct. A truly unbiased system would just make the best analysis possible given the data available. What you propose is to make a biased system in order to give an advantage to one group of people rather than the other group. This type of political correctness fuels resentment.

What they are saying is that, if you know your data is biased, you have the option of designing your system to compensate for that bias.

Why do you seem to think that you have to just throw your hands up in the air and accept the bias? Wouldn't the 'best possible analysis given the data available' include compensating for known biases in the data? If I was designing a system for analyzing temperature data, for example, and I knew that every measurement was 10 degrees warmer than the actual temperature, shouldn't my analysis system compensate for that?

The issue here is how am I as a software developer supposed to combat that bias?

Given the contrived arrest rate example above, do I multiply every black arrest by .7 to compensate for bias?

How would a system built this way be useful in practice?

Why are these parallels always drawn against racial divides as well? How are we compensating for the poor white kid missing out or the rich black kid getting benefits he might not need?

It's such a messy problem but I don't think building bias into software is the solution.

I prefer factually correct, biased systems to incorrect, still-biased-just-in-another-way systems.

> The issue here is how am I as a software developer supposed to combat that bias?

Yes. And that is exactly the question you should ask yourself in work you do.

It's on all of us to unroot social ills. Even though it's thankless and difficult work.

Amorphously defined and not at all actable. Doesn't sound like hacking to me. Thanks but I'll take my code without the side of social justice warrior.

Well said, I am not a judge, nor do I plan to pour over millions of criminal arrests to make judgment calls (which would negate the purpose of automation)... All we can work with is the data given.

If the "data available" is biased, an unbiased system must compensate for the biases of the data it is using. Otherwise you are failing to be "actually correct".

If the data is truly biased in that it doesn't reflect reality then sure, go ahead and correct for this bias. But make sure you are not introducing your own biases which are wrong. "But is it OK that we’re using extraordinarily biased data about previous arrests to predict future arrests and determine where police are stationed?"

The truth is that men living in certain areas are much more likely to commit violent crime and therefore get arrested than the average person in the United States. This is the sad reality of the world we live in. It isn't wrong to deploy police resources where they are most needed. If the data shows that these areas are no longer hotspots of crime, then we can reroute resources elsewhere.

We want to know where crime rates are higher so we can target our limited enforcement budget to where it will have the greatest effect. The whole point of targeting enforcement at problem areas is that it introduces a bias to the rate at which committed crimes lead to arrests and convictions. This is a good thing.

The problem is that using arrest/conviction rates to predict crime rates will be inaccurate if you don't correct for the biases that exist in the ways in which crimes lead to arrests and convictions.

If the data itself is bad and known to be bad, the results will reflect the bad data.

In the case of policing, the arrest record is racially biased (the US releases data that shows this bias and it's well researched). If you train on arrest records, it won't predict actual crime because the results will be racially biased.

Being unbiased is actually a very difficult problem. Especially when we are predicting things rooted in social / cultural trends.

There's a difference between something being "offensive" because it's incorrect, and being "incorrect" because it's offensive.

Our job is to be correct. If we avoid offense in the process, great.

consider danah's example in which external studies show that blacks are convicted for drug crimes at a rate disproportionate to the actual crime incidence rate. the fact that the easily-available training data (court stats) is unrepresentative of the underlying process (crime incidence) is a problem that has to be corrected or predictive output will also over-represent them. the point is that this prior or correction is also part of the "data available" but you have to actually know something about the problem domain to know about it; you can't just grab a court data dump and pour it into a model and say, "welp, that was the data available to me, it's not biased."

No. I'm not obligated to subscribe to your politics, and I won't coerce or bully others, either.

Programmers are not the personal army of any social cause.

I work hard to accurately model engineering problems, and apply consistent ethics in doing so. If that aligns with your definition of "combating prejudice", great for you, but I need the latitude to critically analyze technical problems without the threat of bullying and shaming for wrong-think.

If I need to understand how surnames work across cultures, understand how time works across calendar systems, or understand what "marriage" means to the people getting married, then that's what I'll do.

If I need to support non-binary 'gender' selections because that's what users want, that's what I'll do. On the other hand, if I need to support binary gender for the purposes of simplified data analysis under the assumption of two biological genders with a low probability of intersex individuals, that's what I'll do, too -- politely, and without rancor, and with all the respect for others I can muster.

I'd go a step back from "flawed data". I'd call it a bad sample size.

See [Researchers have long known that whites are more likely to use and sell drugs. And yet, who is arrested for drugs? Blacks. 13% of the US population is black, but over 60% of those in prison are black.]

So how would this happen? One possibility: if a crime happens more police frequent there. So you find more crime. Because we've over-sampled black neighborhoods (including DNA testing making it more likely to find relatives of someone already in prison) the mistaken belief becomes confirmed, and reinforced by the continued arrests.

I'm unaware of how to call out bad sample sizes in data. However doing so may help highlight invalid assumptions.

You said "13% of the US population is black, but over 60% of those in prison are black."

Blacks have a six times greater homicide rate than the rest of the US population, according to FBI extended homicide data. Almost half of homicides in the US in the last year I checked were by African Americans.

With the long prison sentences given for violent crimes, violent crimes are disproportionately represented in prison populations. As a thought experiment, imagine that each year one person with a thirty year sentance was committed, and one person with a one year sentence. One the system stabled out, 97% of the prison population would be serving 30 year terms.

That's also very good to point out. Please note I quoted the article.

You point out a bias in the statistic. Of the imprisoned population. What are the percentage breakdowns for types of crime?

I did some research on this last year. Here's what I found:

https://news.ycombinator.com/item?id=10318406

You can see 4 categories here: http://felonvoting.procon.org/view.resource.php?resourceID=0...

Drugs and violent crime are the majority. So now we can get a breakdown of race. You can see a minor (and slightly misleading, I'll explain) breakdown by race here: https://www.fbi.gov/about-us/cjis/ucr/crime-in-the-u.s/2011/...

The reason the above table is "a bit misleading" is because FBI tracks "Hispanic" as "White". Most people tend to think of "Hispanic" being separate. You'll need to investigate a bit for accurate data/studies regarding this (because I'm not going to while at work), but if I recall "Hispanic" commit roughly equivalent amount of crime as "Black".

Note that this is arrests, not convictions, and it is also possible there is racial bias in courts where Hispanic/Blacks are more likely to be convicted for violent/drug-related crimes and thus over-represent themselves in prisons. Honestly, the data isn't useful because there are too many unknown variables to draw any meaningful conclusions but many people with racially motivated beliefs love to point at this data.

Homicide is a small percentage of violent crimes (close to 0.1%), so it isn't necessarily what skews the prison population. Thankfully, by your logic, violent crime in general would do that, but it becomes much less clear using that data.

The gap in violent crime is much smaller than 6 times according to this DOJ report:

http://www.bjs.gov/content/pub/pdf/cv14.pdf

summed up in: http://www.amren.com/news/2015/07/new-doj-statistics-on-race...

Comparing the violent crime rates to the % population in the US (63.7% white pop; 42.9% offenders | 12.6% black pop; 22.4% offenders), it's roughly 2.8 times as high for black as whites per person. Bad, but not quite 6 times as high by any measure.

Going back to the original number of total offenders, it would make sense (assuming violent crime is represented disproportionately), that roughly 42% of those imprisoned are white and 22.4% are black; instead blacks account for a whopping 37% of the prison population and whites only 22%.

It would seem something more is going on here.

Honestly I'm not surprised blacks dislike officers either; unarmed blacks are 3.49 time more likely to be shot by an officer according to a bayesian analysis[1]. They also found no relationship between this statistic and crime rate in the area.

[1] http://journals.plos.org/plosone/article?id=10.1371/journal....

Big props to you for digging into the raw stats.

To get estimates for prision population percentages you'll need to adjust the convictions for each category by the average number of years served for that crime category. As you said, homicides make up a tiny fraction of crimes and yet a double digit percentage of state prisoners are serving time for homicide.

Fair enough; homicides are indeed ~12% of the state prison population. I can't do it now, but I'm still very curious. There are dozens of other factors that could go into it as well; if one race has a higher average sentencing time or serving time; that would bring up more questions.

Yes, different sentencing would make an effect, though obviously you would need to control somehow for prior convictions and the probability of re-offending, which sounds difficult.

You'd also need to compare inside a state, since states can have quite different sentencing laws. For example, in California the sexual assault of an unconscious person has either a one year or two year max sentence. Whereas in South Carolina it has ten year max sentence.

i don't think you meant "bad sample size," i think you meant "non-representative sample" or just "bad sampling design."

the parent referred to bias. the example you're giving is one where the sample is biased because of a poor choice of sampling method. so, i think you're actually agreeing with the parent (as well as danah).

I don't follow you. When do you start with biased data? Furthermore, wouldn't a advertisement system that somehow progresses to an increasingly erroneous view be broken anyway?

If historic and contemporary data assumes American blacks live in the city and whites live in the suburbs, data alone will not "know" the histories of race-based discrimination. So if it advertises apartments to blacks and houses to whites, it does not "understand" that it is propagating a stereotype. A human has to intervene.

The point of the algorithm is simply to choose advertisements as successfully as possible.

Current distributions of property, wealth, access to services, etc. are all rooted in discriminatory laws and policies of the (not always distant) past. Which creates a risk that blindly pointing an algorithm at these data sets will result in an algorithm which assumes these are the expected/natural distributions and will optimize to perpetuate them, essentially becoming an algorithmic implementation of the prior discriminatory laws and policies.

Thus, unless the goal is to perpetuate the effects of discrimination, algorithms need to be carefully set up and trained to understand that "this isn't how things are supposed to be".

The algorithm just tries to model the world as it actually is, not as it is supposed to be. The algorithm cares nothing about uncomfortable truths. The algorithm should not be set up and trained to reflect a wrong model of reality, but rather you should consider if you want to use an algorithm based on reality or one based on how you want the world to be.

It really depends on what your algorithm is determining.

Fair enough. In many cases we would prefer to model it how we want the world to be in order to overcome human misconceptions that have caused the world to be as it is currently.

Society would have to compensate me for turning a blind eye to today's reality, because otherwise I'm giving myself a competitive disadvantage vs. people with accurate tools.

Yes, I'm glad we are in agreement. We shouldn't limit ourselves to current state of the world. We should design systems that help us get closer to an improved world.

This isn't hard to understand. Suppose crime is equal in two neighborhoods, but the police patrol one neighborhood 500% more than the other. What would the result of the algorithm be? Five times more arrests in one neighborhood than another.

Now, take that model and apply a really crude analysis. You would conclude that there is five times more crime in the more heavily patrolled neighborhood. Your algorithm, applied to policy, suggests that crime could be controlled better by patrolling the "bad" neighborhood ten times as much, rather than five times as much.

Now, consider the social result. The "high crime neighborhood" then sees its property values drop, and more homes sold into the rental market. This attracts poorer, rougher people who can't afford the "good" neighborhood. Now crime actually goes up, reinforcing the idea that the bad neighborhood is high-crime.

Now, imagine we've been doing this to those two neighborhoods for generations. What are the likely results of your algorithm? There's your uncomfortable truth.

> When do you start with biased data?

    In the days when Sussman was a novice, Minsky once
    came to him as he sat hacking at the PDP-6.

    "What are you doing?", asked Minsky.

    "I am training a randomly wired neural net to play Tic-Tac-Toe" Sussman replied.

    "Why is the net wired randomly?", asked Minsky.

    "I do not want it to have any preconceptions of how to play", Sussman said.

     Minsky then shut his eyes.

    "Why do you close your eyes?", Sussman asked his teacher.

    "So that the room will be empty."

    At that moment, Sussman was enlightened.

(from: http://catb.org/jargon/html/koans.html#id3141241 )

There is always bias (and other types of error) in data. There is even bias in the choice of which data to use and the type of analysis to perform.

If you think that this isn't a problem, you should really read about practices like "redlining"[1]. For many decades segregation was (and still is) enforced by opaque "loan approval" methods that just happened to always deny loans to blacks.

[1] http://www.theatlantic.com/magazine/archive/2014/06/the-case...

Except: "So long as man is approaching Ultimate Wisdom, he is still dwelling in the realm of consciousness. If he is to realise Nirvana, he must pass beyond the realm of consciousness. In highest Samadhi when consciousness has been transcended, he has passed beyond discrimination and knowledge, beyond any reach of change or fear. He is already enjoying Nirvana." - Sutra of Transcendental Wisdom

I love how we work in an industry where it's ok to manipulate massive numbers of people because as engineers we know what's best for them. \s.

Reply on news.ycombinator.com