Hacker News Reader

Comment by whatshisface | Hacker News Reader

Ok, help me reason though this. Let's say that native americans are poorer on average and therefore less likely to repay a $1k loan. Therefore, if you give out loans based on the probability that someone will be able to pay them back you will be discriminating against native americans (not the individual native americans with good credit scores, but you will be denying more native americans on average). So, you shouldn't try to compute how likely it is for someone to pay their loan back. I keep hearing things like, "you can't use zipcodes as input because zipcodes are correlated with race and it is illegal to discriminate on race." However, some races have better financial situations on average, which means that financial situations correlate with race, and by the same argument you shouldn't take finances into account when you review a loan application.

Obviously that is absurd, but where does it go wrong?

View on HN

Nope. You are exactly correct. The way to approach removing bias in all personal financial decision making, algorithmic or otherwise, is to only allow it to be based on previous individual financial behavior.

The solution is pretty much that simple. I would argue that all correlative models dealing with human beings will always be discriminatory.

Will loans become more expensive for many people? Sure. But that is the true solution. Sometimes the right decision is simple but unconfortable. In my opinion, this is one of those times.

>is to only allow it to be based on previous individual financial behavior

Your financial history includes your rent or mortgage payments, which are for living at your zip code. Restriction to financial history is not all that much of a restriction.

In principle I agree with what you are saying. We have to be a bit careful, however, since past financial history may have itself been biased by prejudice, e.g., if you were unfairly denied a job based on your race. Of course, we sometimes have to draw the line somewhere, and personally I agree that using previous financial behavior probably is fair.

Where things go wrong is when race comes into play to begin with. In your example it is clear that credit scores - or whatever equivalent is used in the country in question - are a factor in deciding whether an applicant should be allowed to take out a certain loan. That check should be done on base of that applicant's credit score irrespective of where he lives, what car he drives, what his name is, what colour his skin has or any other unrelated characteristic. While there might be relationships between average credit score and the aforementioned data points they are not in itself indicative of the likelihood of the applicant defaulting on the loan. The credit score is a better one, as is their income for the last X months and other similar data.

It might well be the case that a totally 'colour-blind' scoring algorithm still ends up sorting people by one of those mentioned, unrelated data points. This is not racism, it is just the consequence of the mentioned relationships. Calling it racism is a sign of ignorance and does a disservice to the effort to get rid of true racism.

The problem is, e.g. banks will want to use all the data they can to make a decision on what sort of loan to offer you. Employment history, travel history, family medical history, as much data on your spending habits as they can get, how much money your relatives have, which properties you own and are they likely to rise or drop in value, where, when, and how you drive (which affects likelihood of having a car accident -> unexpected expenses or death -> affects loan repayment), which websites you visit, how much time you spend on your phone, in bars, etc. etc..

Each has a plausible justification for inclusion. But the more data points you have, the better you're able to predict race from them (or anything else you're not supposed to).

So how do you decide if an algorithm is discriminatory? Maybe a whitelist or blacklist of what data it can use?

I think we can use dangerous data in safe ways. If you use how you drive only to predict likelihood of car accident that's ok. If you use it to predict future wage growth it's not ok. If you use it in a blackbox model it's not ok.

If you use time in bar to predict cirrhosis it's ok.

etc

>If you use how you drive only to predict likelihood of car accident that's ok.

We may say that now, but what if later some algorithm finds there's a hidden correlation between driving performance and race/gender? will it still be an ok metric then? where do we draw the line?

I said that expecting such a correlation to exist.

The line is between inferences of bad behavior -> bad result such as "you drive risky so you must be at risk of an accident" and pure correlation "you drive risky so you must be a man so you must be at increased risk of prostate cancer".

And conclusions should be assumed to be of the latter kind unless shown otherwise.

Your personal financial history is a lot more specific than your zip code. They may be on different ends of the same scale, but it seems far more defensible to make choices based on financial situation than it does zip code.

Setting aside for a moment the fact that your financial history includes your home loan which includes your home as collateral which is located in your zipcode, suppose that the model got more accuracy by looking at the zipcode. Isn't that all the defense that's needed? What is meant, exactly, by "defense?"

AFAIK the credit information in your file does not include zip code as part of the home loan accounting.

Defense means 'what are you going to say when the feds come knocking and want proof you are not discriminating against a protected class?'

As a practical matter it isn't too hard to come up with a legal defense for that, of course. Lots of plausible deniability here. IMO most of this effort is by people trying to help prevent computer nerds from becoming (or staying, as we may already be there) complicit in discrimination by race while hiding behind algorithms as if they are somehow infallible.

This part doesn't make sense to me:

want proof you are not discriminating against a protected class

If black people are poorer on average, then even if the system takes only takes personal history in account, black people will inevitably get worse credit rating on average. It just follows from the premise, unless banks participate in some form of race-based redistribution which I find distasteful. How's it supposed to work in your opinion?

This is what makes fairness so challenging. There isn't a clear place to draw the line, where certain covariates are permitted and others are not. Intuitively, what you are saying makes sense---your financial situation is a consequence of your choices, whereas things like zip codes are correlated both with your choices (i.e., I chose to live here) but also your race (e.g., due to lasting consequences of racial bias). Thus, we might decide that it's fine to use financial history, but not zip code.

Of course, things are not so clear cut. Your financial situation can also be a consequence of discrimination, e.g., if you were denied a job or unfairly arrested. At the end of the day, some human has to sit down and decide what is OK and what is not. Personally, I believe the goal of research on algorithmic fairness should be to give the people who will ultimately make these decisions (e.g., judges, politicians, etc.) the tools to understand both broadly, the kinds of things that can go wrong when using algorithms, and also to understand what might have gone wrong in a specific situation.

Reply on news.ycombinator.com