How we use big data can reinforce our worst biases, or help fix them | Hacker News Reader

How we use big data can reinforce our worst biases, or help fix them

nautil.us

64 pointsdnetesn64 comments

> Instead of relying on algorithms, which we can be accused of manipulating for our benefit, we have turned to machine learning, an ingenious way of disclaiming responsibility for anything. Machine learning is like money laundering for bias. It's a clean, mathematical apparatus that gives the status quo the aura of logical inevitability. The numbers don't lie.

http://idlewords.com/talks/sase_panel.htm

>Machine learning is like money laundering for bias

A potent analogy.

The cycle is truly terrifying:

1. deep institutional racism compromises training data

2. data classified through that lens

3. ML insights and patterns inform institutional policy

4. goto 1

The book Weapons of Math Destruction raises similar questions - not just perpetuating racism, but also other discrimination (such as against the poor). There was a previous discussion on HN regarding an interview with the author: https://news.ycombinator.com/item?id=12642432

The writer of this book did a great 99 Percent Invisible interview, well worth the 15 minutes.

https://99percentinvisible.org/episode/the-age-of-the-algori...

Thank you for the link.

She also did over an hour on econtalk

http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

The problem is that algorithms are like a 5y old child. They know no such thing as politeness, political correctness or politically charged environment unless that is implemented with additional code. So they will do the equivalent of a 5y old child saying "you smell bad" to the old aunt, which might be true nevertheless. The question then becomes should we write politically correct code or bias data so that it doesn't offend our sensitivity? Or on the other hand should we not anthropomorphize a mere algorithm and not interpret as racist what is the mechanical reflection of our own problems (in the article example, poor neighborhoods in the US don't order much on amazon, poor neighborhoods tend to be non white, non white neighborhoods end up being less served by amazon's algorithms).

Also, just like a 5 year old child, the bias of the parents is coded into the algorithm. So the 'bad smell' of the old aunt may not be bad to many people's standard of smell but the algorithm still identified it as bad.

/offtopic

Are Algorithms Building the New Infrastructure of Racism? Possibly. But what I find even more worrying: One day algorithms may take so many factors into account that they can discriminate against minorities so small that they neither have a name nor a voice. If an algorithm like that affects you your life could turn quite Kafkaesque.

This is a crucial question, especially anywhere that algorithms will be applied to areas where there are laws against discrimination.

If you want to use algorithms in these areas you will eventually have to prove that they don't illegally discriminate.

It is impossible for an algorithm to "illegally" discriminate, as it's a machine devoid of the human concept of "intent"; It doesn't know what "race" even means.

So this narrows to these basic outcomes if we assume any given algorithm isn't purpose built to always have a discriminatory output:

1. The input data is junk data, so the output is junk data.

2. The input data is fine and humans don't like the output for human reasons.

The problem I'm hitting on here is that if we assume any sort of "discrimination" based on a data set is "bad", then we don't really care what the outcome of a given algo is, and we're just trying to massage data sets into an "acceptable" output. Seems like the opposite of good science to just throw away results that we don't like, because reasons.

This is nonsense.

It's quite possible for an algorithm that doesn't know what race is to be biased despite ignoring race if the real world is full of racism, and that racism manifests in dependent variables that the algorithm bases its decision on.

This concept isn't even a little novel. Google the phrase "previous servitude" for an example of just such a dependent variable in action.

In this case, you seem to be implying garbage in/ garbage out.

My point is that when you get a good input dataset and there is still some sort of bias in the output, you have to start asking questions on why the results are biased towards one group by examining the variables pertaining to that group.

The position you're starting with seems to be that the output should never be biased, which means you're not really approaching this scientifically and won't stop messing with input sets until you get the output you agree with; not the output which is necessarily true but you disagree with for personal reasons.

You can't just claim "bad data" every time you get a result that doesn't jive with your politics.

What I am saying is much simpler. You said that the machine can't discriminate because the machine doesn't know what race is.

I'm just pointing out that the machine doesn't know what anything at all is. All it can do is work with what humans are telling it. If the real world is biased it can then itself be biased.

There are many feedback loops in societies. The rich get richer and the poor get poorer and those subjected to discrimination often have objectively worse outcomes. An algorithm can very easily be susceptible to just recursively documenting actual bias that exists in the real world and giving it the veneer of objectivity.

It's an actual hard problem that is the topic of this thread, hand waving isn't going to fix it.

I think the problem is that you're looking at "bias" as a dirty word.

> If the real world is biased it can then itself be biased.

And if said algorithm corroborates that bias, then is reality somehow "bad" as a result, or do you just not like the outcome because it "feels bad"?

There is a difference between truth and an idealized outcome that you desperately want to be true.

None of that is relevant to what the law is.

Imagine a world ruled by blue people, who control all the roads. Every time a green person tries to drive a blue person will throw rocks at their car, pour oil in front of the wheels, etc.

Also for historical reasons all the green people live in certain neighborhoods.

Now imagine creating bureau that would review driving and accident data and attempt to answer the question "How good at driving is this person?" The idea being that we are going to try to keep the worst drivers off the road.

People scream hey wait a second that's not fair at all. So you take steps to address this whole system instead and solve the real problem. You try to crack down on all the rock throwing, and not prejudge someone without knowing the circumstances.

Until one day an algorithm comes along to answer the question. But wait, we thought of that, we make it so the algorithm can't take into account what color the driver is. It can only rely on other things.

See the problem yet?

> And if said algorithm corroborates that bias, then is reality somehow "bad" as a result, or do you just not like the outcome because it "feels bad"?

The algorithm won't corroborate the bias—it will include the bias, because the bias will be built into its data. It's not an independent variable.

> It is impossible for an algorithm to "illegally" discriminate

An algorithm is a set of steps given to it by a human. If the human can discriminate then so can the algorithm.

In the case of ML, let's say you are making lending decisions and your data set consists entirely of the race of the applicants. Obviously that ML algorithm would illegally discriminate.

Now, extrapolate from that extreme case and say that you feed your ML algorithm a whole bunch of random data points about your potential borrowers.

How do you prove that your ML algorithm is not making its decision based on data points that are proxies for race?

If you don't have an answer to that question good luck with the swarm of lawyers who will be suing you into oblivion.

This same article keeps popping up about once a week, with the words slightly switched around and in a different magazine. A couple of weeks ago it was the NYT (https://www.nytimes.com/2017/12/02/opinion/sunday/intelligen...), before that in the Atlantic, Politico, ProPublica, etc.

The error they make is always the same. I highly recommend this piece: AI 'Bias' Doesn't Mean What Journalists Say it Means https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...

Surprised at the downvotes you're getting. That's the first time I've ever seen that jacobitemag article and it was a very interesting read.

The majority of that article seems to report on the same stories as the Nautilus article. The meat of the disagreement is in the conclusion section.

> However, there is a significant cost to forcing algorithm outputs to reflect wishful thinking. If we are issuing loans, we will issue more loans to people who do not pay them back. If we are making parole/sentencing decisions for convicted criminals, we will release more dangerous criminals who go on to commit new crimes.

> This may not be a concern for the journalist, but it should be a concern for the rest of us.

I agree that this is the heart of the issue but by leaving this to the conclusion, I think the author didn't end up arguing for why we should optimize for objective cost instead of for our ideal reality. It might be financially profitable for a bank to use a racist lending policy but as a society we think that it is wrong so we passed regulations forbidding the banks to do so. Similarly, the criminal justice system is supposed to serve society and follow our ideal sense of justice and shouldnt be blindly optimizing recidivism rate. As the Nautilus article said, "What gets chosen is usually whatever is easiest to quantify, rather than the fairest".

This article is catnip for the HN community. Talk of algorithms and institutional bias? And a way to improve humanity using Silicon Valley machine learning technology? Many of you probably needed a change of pants.

This article is trying real hard to find victim's anywhere and everywhere. Not that long ago there were many people talking about how wonderful the future was going to be with algorithms and big data to make unbiased decisions. Now we are basically there and they still find a problem with it because of "calibration issues" which were mentioned to be due to mathematical incompatibility. You'll also note the wording about the improved algorithm being "independent" of race. Not that the results were more fair or accurate, just "independent".

Then there's the ever popular gender pronouns that were mentioned. Guess what. In nearly every country in the world doctors are majority male and nurses are majority female. Funny that we never hear these "gender biases" mentioned with things like "___ is a construction worker" or "___ is a bus driver", isn't it? Those are male dominated fields as well yet no one ever talks about gender imbalance or discrimination. It's only ever with the higher status/income jobs that you see this talk of imbalance. Many of these aren't "biases" but rather statistical probabilities.

I wish we stopped looking downstream, at this incident level data (arrests, recidivism, criminality in general) and more upstream. How does criminality play out as a function of the make up congress, historical generations, political leans of the administration, etc, etc?

Computers aren't racist, algorithms aren't racist, data aren't racist. Computers are inanimate objects, algorithms are instructions and data are bytes. None of those things can possibly be racist.

This comment is missing the point. The problem is that the algorithms can reinforce the disadvantaged situation of some groups of people, instead of help improve it. And the thin veneer of objectivity that the algorithms provide can legitimize something that might be effectively a racist policy.

Then they're shitty algorithms that don't work properly or your data is incomplete. If your data is showing biases it's because they exist in the world not because the data is racist.

Yes, the problem has always been about what we do with the data instead of the data itself. The issue now is that people currently put a lot of blind trust on machine learning, even if the input data is garbage or the output doesn't have the intended result.

Not just machine learning. I remember learning in ecology that if the computer models didn't match with reality. Reality was probably wrong the model just needed better variables. I remember thinking how incredibly stupid that was at the time. Now I realize that's the way it works with everything and has for a while.

The problem with modelling real world systems is there are too many variables and we have no way of knowing just how many their are and what effects they all actually have on a system. We can never make a complete model yet we act and make decisions as a society as though our models are complete and infallible.

I don't think its necessarily the job of data to improve a situation. Rather its the data that will define the problem.

"Racism/racist" in the last few years has become this all encompassing term to label bad anything that's inconvenient to otherwise interpret as a difference.

Its gotten to the point where I do not click media links where race is included in the title. Wild accusations of injustice get click$.

There are two ways to understand racism. One is to think of it as an attitude held by an individual that causes them to act in particular ways. The other way is to think of it as a system that perpetuates inequality, independently of the internal attitudes of individual actors. If you’re using the former definition, you’re possibly correct, if you believe that computers can and will never have attitudes toward things. If the latter definition, you most certainly can have computers perpetuating a system of racism, even if they are lacking internal attitudes.

That second definition of racism is idiotic. That has nothing to do with race. A system of inequality can be caused by lots of things. The entire fucking world is a system of inequality regardless of your race. The biggest thing separating people right now isn't even race it's money. I see people of all colours and races every day united by their poverty, I also see them united in their wealth.

Sorry, I was typing on a mobile device. I thought it was obvious that I was talking in both cases about a system of inequality based on some notion of "race".

Also, saying that there is one system of inequality doesn't mean there aren't others. Class is certainly another, related, system. But one doesn't reduce to another.

>But one doesn't reduce to another.

Yes it does. Every day I see people far more affected by class based inequality than race based inequality. Right now not having money would affect you far more than being a different race than you are.If you suddenly became a different race overnight chances are your life really wouldn't change that much, if you lost your job and all your money you'd be totally fucked. That means by definition one has a higher impact on the world.

To say that one problem reduces to another is not to say that one or the other is more important. It is, rather, to say that one is purely a function of the other, which implies that solving the latter necessarily solves the former. So, in this case, I took you to mean something like "Class is the fundamental problem; if we solve class-based inequality then race-based inequality will be solved as a side effect."

It sounds like you're saying that one is more important. I honestly don't know whether class or race-based inequality is more of a problem; I think it depends on context. Both of them have caused great problems. I will point out that many people have been as "fucked" as imaginable for being labelled as a particular race: murdered, enslaved, forcibly displaced from their homelands, etc. And this sort of thing continues to this day, e.g. in Myanmar right now the Rohingya are arguably being genocided. If one of those people could change their race, they would be decidedly unfucked in that context.

Wow, slick. Looks like a really well put together kit. Is he happy with the manufacturing quality? How much allowances is there for variations in router diameter?

I will not deny that Israel's violence. It seems you are ignoring Palestine's violence.

When is violence just violence. When is it racism?

Nautilus is a US website. At the same time, to imagine that every Jewish person is somehow required to criticise Israel with every second breath - no matter who or what else is actually being discussed - echoes the "Jew second" obsession of anti-semitic nationalists. Is it possible that the writer is Jewish and has no interest in writing about Israel in the article, accusingly or defensively?

Finally someone addressing real issues.

No, they aren't. They are the one chance we have to escape it.

You see humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker.

It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. Way back in the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists.

Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all.

But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". (http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...) The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them.

This is why they aren't everywhere already. The last thing we need is fear mongering articles like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not.

A big part of the case this article makes is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant, which is a pretty low bar to begin with. And yet it always comes up as the main reference in these discussions. It's the only piece of evidence they can find of this nonissue ever happening. Stop. You are just making the problems you claim to care about worse.

The bigger issue raised in the article is the idea that, even when algorithms are an "improvement" on human bias, the public is willing to accept that they express no bias at all. That level of trust in technology is unfair, and it doesn't exist for human judgement - if people are more confident in exercising oversight, that's at least one reason to not switch over to purely algorithmic decision-making.

No, they're not[0]. Correlation is not causation, and this article makes many such correlations. It also appeals heavily to fairness. Business is not about fairness, it is about profit. I don't see the potential profits in racism - if it is possible to provide services to people profitably, it will be done.

Arbitrary clusterings such as zip codes, while certainly statistically significant, do not translate to intent by algorithms or algorithm makers.

Fairness is somewhat subjective, and I think it's fair for companies to pursue profits such as "same day shipping", even if that cannot be simultaneously rolled out to all markets. If it is profitable, eventually, it will be.

[0] Invoking Betteridge's Law of Headlines ( https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline... )

"At the center of Northpointe’s argument was an essentialist fallacy: Because persons that police classified as Afro-Americans were re-arrested more often in the training dataset, they claimed, COMPAS is justified in predicting that other persons classified as Afro-Americans by police—even in a different city, state, and time period—are more likely to be re-arrested. The cycling of classification into data then back into classification echoes W.E.B. Dubois’ 1923 definition, “the black man is the man who has to ride Jim Crow in Georgia.”"

Sounds like the answer is yes.

Then I would not say they are building new infrastructure of racism. They are getting garbage input and returning garbage output.

The accusation here is that they're maintaining the status quo, in a way, instead of working on making things better.

True, but the research they are doing - "improving" language to reduce bias, reeks to me of 1984's Doublespeak.

What are the chances this won't be co-opted by the other side, to enhance divisions?

You're right, let's just do nothing and let the problem get even worse as we push blame onto an algorithm rather than deeply ingrained societal issues

I think you are misinterpreting my statement. I never said blame algorithm. I just said, what happens if algorithm (for eliminating some thought patterns using language inference) is used by the other side?

The problem which the quote is pointing to here, is that then people use the old garbage to justify the new reprocessed garbage and vice versa, and it becomes harder for people to realise that it's garbage.

How does that answer the question as an exclusive yes without any room for no?

> Correlation is not causation

That's particularly ironic, since a lot of ML can treat it that way.

"... do not translate to intent ..."

intention is orthogonal to structural bias

from the last section of the article: "infrastructure beats intent"

What a tired, lazy cliché...

Why don’t you engage with the content of the article: But when COMPAS’ prediction was inaccurate (either predicting re-arrest when none happened, or not predicting an actual re-arrest) it routinely underestimated the probability of white recidivism and over-estimated the probability of black recidivism. In other words, it contained a bias hidden from the perspective of one set of statistics, but plainly visible in another..

Edit, because you noticed how unhelpful “ha ha, Betteridge” by itself is: intent may not be necessary for discrimination, nor can you deny intent when you continue to use methods shown to produce discriminatory outcomes.

A profit motive is similarly useless as a defense of discrimination. Maybe young, white, heterosexual female employees are better for sales, because you operate in a xeno-, homo-, gymno-, gerontophobic neigbourhood. But it’s no excuse to limit your job ads to that demographic.

One reason is that such logic perpetuates its own causes: if the police is more likely to stop&frisk non-whites, you will have fewer whites in your arrest statistics, even if they commit the same number of crimes. To then turn around and use those statistics to justify your racist police procedures is both morally and scientifically bankrupt.

I feel like intent is necessary for racial discrimination. To discriminate between racial discrimination and mere ignorance or ineptitude one must look at the intent.

Why would that matter if the outcome is the same? People are being disadvantaged solely on the basis of race, whether this is being done intentionally or not it's harmful and needs to be rectified.

Why should intent matter? These are not criminal cases, but civil. If an employer is discriminating against a group because of unconscious biases, or using methods that lead to Discriminstory outcomes, there are real people who were harmed.

Civil law isn’t about morality. It’s about righting a wrong. If you die because a mechanic negligently forgot to reconnect your car’s brake lines, you (and your family) will be harmed. The question for the courts then becomes shifting that harm to the party responsible, as far as that is possible, and your family will likely be awarded damages.

The law doesn't agree with you.

https://en.wikipedia.org/wiki/Disparate_impact

If you are ignorant, and through your ignorance you help perpetuate and perpetrate racism, you personally may not be racist, but you are still perpetuating racism.

Somehow a tautology like this is controversial.

I don't blame people for pushing back. Nobody wants to wake up one morning and find out that they themslves have been the problem all along.

You use scientific buzzwords, but you’re coming into the conversation with unscientific assumptions.

> Business is not about fairness, it is about profit. I don't see the potential profits in racism - if it is possible to provide services to people profitably, it will be done.

That assumes companies are completely logical, singular entities which all follow the same motivation without deviation. As companies are the sum of the humans working there with all their personal problems, faults, opinions and attitudes (plus the organizational framework itself) that's rather unlikely.

It also assumes companies always know beforehand what the best strategy is to make more profit (just look at VW for a recent example where that didn't pan out).

Business is not about fairness, it is about profit.

Yet historically, a number of businesses in various parts chose not to serve certain sections of the population, and sometimes still do (off the top of my head, wedding cake companies choosing not to sell to gay couples).

Reply on news.ycombinator.com