How a Machine Learns Prejudice | Hacker News Reader

How a Machine Learns Prejudice

scientificamerican.com

37 pointsmatthberg27 comments

> In June 2015, for example, Google’s photo categorization system identified two African Americans as “gorillas.”

> Law enforcement officials have already been criticized, for example, for using computer algorithms that allegedly tag black defendants as more likely to commit a future crime, even though the program was not designed to explicitly consider race.

Consider these two examples from the article. One is when the machine failed in a particular undesirable fashion. The other is when the machine "worked^" but in an undesirable fashion. (^That is, overall it might have worked well, but the bias in errors wasn't sufficiently explored.)

Modelling "undesirable fashion", which is a dynamic, subjective social construct, is far more difficult than either of the tasks originally set in the examples.

In the first example, how many images of people were confused with non-humans? I can't believe this was the only failed result. The only reason this particular example was problematic was that these specific people were specifically identified as gorillas.

No problem if they were identified as a car, or a chess piece, or a satellite. Gorillas though, that's a special failure. We all know why. And we know why the error occurred. If the machine sees predominantly white people and gorillas, then "people" are white and "almost people" but with darker pixels is "gorilla". It is human prejudice in interpreting the results that made the error an issue.

There seems to be a bit of explaining away all bias as the result of human creators here. If an algorithm trained on real world data displayed some traditional bias, would we assume such a cause and dismiss it? One could just as easily make this argument to dismiss all correlation based evidence. Yes the code / methodology matters, but we'd best get ready for deep learning algorithms to confront us with some uncomfortable truths.

Let's assume you have red and green people who are otherwise identical, and that there was systemic discrimination against green people in the past that led them to have lower incomes and lower education levels and live in areas with lower property values and higher crime rates. A naive computer model trained on objective data might find correlations between green people and high default rates on loans, for example, which might be accurate, but which aren't inherent to green people but were forced on them by external circumstances. A reasonable bank might then conclude that they should discriminate against green people without any personal bias against green people at all.

The question you have to ask yourself is: is that fair? And what should the law say about it?

They aren't inherent in green people, but you shouldn't give them loans anyway. Loans should be based on reality, not an ideal world that we don't live in. And certainly fairness has no basis in reality. It's a human concept.

(quote by some guy named "oehead2000" on some other hacker news posting. It fit the context perfectly, so I am copying and pasting from his reply)

"Unfortunately whatever algorithm computers come up with to assess creditworthiness is going to be 'racist' by the broad disparate impact definition of racism that the government uses. Ironically, the algorithms will have to be made actually racist to correct for this."

Most significant variable. If green is a proxy for poor, which is in turn a proxy for risky mortgage loan, then the poor variable is the better point of discrimination.

Well, the proper response probably isn't to shoot yourself in the head and prevent yourself from knowing about the difference in default rates. It is entirely possible to figure out what is going on, to what extent, and then make an informed decision as to whether to continue. Avoiding discovering things, which is what all these asinine articles end up suggesting, is almost never a good idea for competent adults, and legally preventing people from discovering things is antithetical to a free society.

Forcing non-optimal decisions based on an arbitrary notion of "fairness" will only weaken our institutions and breed resentment: "group A gets loan approval with a lower credit score than members of my group, what gives!?"

It also increases the fragility of the group you're trying to help by acclimating them to lower standards.

The success of Asian and Jewish people suggests that things can correct themselves over time. The law should stay out of it.

http://www.pewresearch.org/daily-number/asian-americans-lead...

http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chine...

There is literally no support in the article for the contention that "artificial intelligence picks up bias from human creators" as opposed to making correct inferences from reality. All of the examples they provide, modulo blurry tank pix, are of the latter.

Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!

"The computer can't be wrong" is an equally dangerous attitude.

What makes me sad is that there are good reasons to throw out information which don't involve denying mathematical truth, yet people, for whatever reason, seem to prefer denial.

What should happen: "I understand that this person has black skin and therefore according to elementary statistics is more likely to be a criminal, but acting on the information would create an unfair double standard that punishes people for crimes they didn't commit, it would help perpetuate a cycle, and it would violate individual human dignity, so I won't."

What actually happens: exactly what parent said. "Oh no! the computer is making uncomfortable findings that we don't like. The computer must be wrong!"

facepalm.

It's not quite like that.

Though blaming algorithms for being "biased" is rather strange as well. It seems to me the algorithms do exactly what you'd expect them to: have predictive capabilities about reality.

Of course, there is a problem here. You don't train neural networks with data of people who've committed crimes. You don't have such data. So what data has been used by the engineers to feed to this machine? Data of people who have been convicted of committing a crime, mostly likely.

And now you've constructed a feedback loop. Those pesky blacks are more likely to commit crimes? Always knew it. A man has been murdered and there are 2 suspects, one white and one black? Why even bother investigating the white guy, we already knew it was the black one. Holy cow, 1 year with the new system and black crime has increased by 50%! Always knew it. Better keep the system in place to save everyone from those pesky blacks.

Oh, the irony.

I understand your/TFA's argument but I think falls dangerously wide of the mark. I'm sure that the feedback process you describe is real, but I'm also sure that judicial prejudice is far from the only thing feeding the cycle of poverty and crime that plagues black communities. Even if we were to succeed at eliminating judicial prejudice from our training data, the other factors maintaining the cycle would once again lead the ML algorithm right back to p(criminal|black)>p(criminal).

The way to fight the prejudice is not to pretend that math would work out differently if only we had better data / more sophisticated models / etc, it's is to realize that it's OK to discard the result of bayesian reasoning due to factors out of the calculation's scope. Courts discard evidence all the time for a million sundry reasons. Racism / double standards can be another.

What other data do we discard? Who gets to decide which data to discard? There is no point in doing data analysis if you then discard uncomfortable data anyways.

Discarding evidence isn't a new concept, it's a well established and very import part of our existing judicial system. Lady Justice doesn't wear a blindfold as a fashion accessory. Google "fruit of the poisonous tree" to learn about the specifics.

whats with shitty scientificamerican stories on HN today.

If you don't think the submission is appropriate for HN, flag it if appropriate and move on.

"racist computers" is the funniest trend since "sexist babies"

It should be obvious that if an ML algorithm is trained on sufficient real-world data, it won't be racist - it'll just not be politically correct

Why is that obvious? Take the case of criminal predictions. If you're using convictions as ground truth, then you're just mirroring existing societal biases. For instance, in a society with pervasive racism (like, say 1960s America), the more real-world data you give it, the more your model is likely to converge on racism.

Computers aren't racist but to characterise the article as referring to something like that is to fail to understand it.

> If you're using convictions as ground truth, then you're just mirroring existing societal biases.

And if you don't use real life data...? What do you get?

Not as bleak as you imply. Just don't give the model race or strong indicators of race as an input.

That's the problem. If there is ANY bias shown in your results, your algo (actually you) will be accused of prejudice. That is what happened with the Re-Offending Risk Assessment in the linked article. It didn't have race or anything like it. It had suburb.

Please. Zip code is a well known proxy for race given the history of racist housing practices. Don't believe it? Ask yourself: What does a person from Harlem look like?

Zip code is not a proxy for race and it is idiotic the say that it is. The zip code of Karl Malone is the same as Chuck Norris. Zip code is a proxy for COMMUNITY.

You're trying to shift the words because it's uncomfortable, but the underlying facts remain the same. Community is often dominated, and in fact is defined by, by race. You can cherry pick a millionaire here and there but those are exceptions but it's disingenuous and ahistorical to assert that this isn't true. Redlining existed to maintain racial divisions. Cities, counties, and even states explicitly maintained racial divisions, and even though these laws do not exist today, their effects are maintained.

Do you think it's a coincidence that the only community with a significant number of black people on the peninsula is East Palo Alto? It's not. ( https://techcrunch.com/2015/01/10/east-of-palo-altos-eden/ ) Do you think it's a coincidence that white people live north 8 Mile Road in Detroit? Shouldn't Marshall Mathers had grown up south of it? It's not. I suggest you read up on your history http://www.theatlantic.com/business/archive/2014/05/the-raci...

The interesting thing is that literally all of the inferences we are "worried" about computers making were social consensuses not too long ago, with ample first-order evidence to support them even in the absence of fancy machine learning algorithms.

Reply on news.ycombinator.com