Hacker News Reader

Comment by rndgermandude | Hacker News Reader

Other than him being very outraged and claiming gender-based discrimination again and again, is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis?

Just to be clear: I am not dismissing the possibility that the algorithm (meaning the training data, really) is gender biased, it just isn't clear from what I have seen in the tweet storm that this is necessarily the case.

E.g. I had an ex girlfriend who had a slightly lower income and slightly worse credit score (well, German Schufa score) than me, and yet she got offered like twice as credit when she applied for a card. I am guessing (just guessing) that this was due to her having paid off e.g. a car loan in the past and being generally more consumerist than me, while I never had taken out any major loans.

View on HN

The whole thread is less about his wife having a lower credit limit, and more about Apple reps not understanding the fucking algorithm.

The axis doesn’t really matter. Maybe it’s gender, maybe it’s not. In any case, it should be about money, credit score, trustworthiness. Apparently, it’s not. Being a women could be one indicator. It might be "being black" for other people.

There seems to be something fishy going on and Apple doesn’t even know why. Hence, there is a bias.

>There seems to be something fishy going on and Apple doesn’t even know why

That seems a little silly, though, since Goldman Sachs is the bank in charge of the card and approval process. Just like Amazon doesn't know anything about the processes that are used to determine credit worthiness for their cards (that falls to Chase and Synchrony), Apple is merely licensing services from GS.

On top of that, GS representatives that work in a call center aren't going to know anything about the details or inner workings of the process either. With the amount of potential abuse for such information, it feels like it would be irresponsible for those people to have access to that info.

The bigger issue is that this couple had a valid complaint and had to turn to Twitter to resolve it instead of GS having some formal method of requesting the info.

Although after this, I would think that this is a good opportunity for Apple to work with Goldman to better understand the algorithm and strive to make the process become more fair and transparent. Just like Apple how deals with 3rd parties making their devices where they can help enforce work and environmental rules.

Definitely. And it sounds like this is exactly what's happening.

DHH posted about this exact thing today. Basically: who cares? It has the Apple logo on it. Apple says: "It was created by Apple, not a bank."

They are responsible and nobody else is to blame if the parts they outsource fail.

https://mobile.twitter.com/dhh/status/1193716881351294976

>who cares?

Apple said they weren't aware of the details surrounding the decision and had the customer (DHH) work with Goldman Sachs to determine the details of the decision. I don't think that's unreasonable and it's not like Apple just wiped their hands of it and said "We don't know, not our problem."

My response was directly responding to the person that said that Apple employees didn't know why the decision was made. Apple was attempting to get an answer for their customer. Especially with credit and finance information, this isn't something that would be acceptable for a low-ranking Apple employee to discuss with the customer and, if DHH is claiming that it should be because it's "Apple's card", then I think he's full of crap.

> is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis?

None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned.

When the process is set up like that, I think it's fair to assume the worst case scenarios and put the burden on the company to prove otherwise.

Adverse inference is a sensible way to combat secret decisions like this.

>None of us are allowed to know that. That's the big problem here. They delegate the decision to a black box that cannot be questioned.

The entire article is about questioning the black box, with regulatory force. Which is a good thing that is is going to be investigated.

>When the process is set up like that, I think it's fair to assume the worst case scenarios

Disagree. Consider it yes, but not assume it as a foregone conclusion.

> and put the burden on the company to prove otherwise.

Agree.

Black-box algorithms that affect your livelihood are bad, and I don't understand how anyone can be opposed to full transparency.

I don't care what the context is. Whether its credit ratings, job applications, or college admissions.

it’s notable that a lot of mid-20th-century fear surrounding the increased use of computers was centered around this exact scenario: a black box, the judgement of which cannot be questioned, deciding your fate, with no recourse.

We all scoffed at this for a long time. ML makes it real, apparently.

I haven't seen any actual evidence other than his single datapoint of him vs. his wife's experience. Of course its possible that the algorithm is designed to give lower credit limits to women, but I find it unlikely. My wife and I have applied for dozens of cards over the past 10+ years and I've consistently found that once you're score is above some reasonable threshold, the biggest predictor of what credit limit you will be offered are the credit limits on the OTHER credit cards reporting to your report. Maybe he already has much higher limits on other cards?

I started using credit cards a few years before my wife, and despite us both having excellent credit the first time she started applying for cards she was getting limits around $1K or $2K while at that point I was around 10x that. But after a few years of getting more cards and requesting limit increases, both of us now have roughly the same (fairly ridiculous) limits across all of our cards. Admittedly I don't have any cards issued by Goldman Sachs bank, but I can't imagine their algorithm would be much different than Amex, Citi, Chase, etc.

No, but neither is there anything that would justify giving a spouse such a very different limit especially considering that they file joint tax returns and her credit score is better than his (see article).

It would seem that his conclusion is warranted absent evidence to the contrary, the difference is too large to explain in ways that make any sense.

Maybe I am missing something, but wouldn't it make perfect sense if they have dramatically different income?

I think even if two people have their property in common (and if the algorithm even knows about that), it is still not unreasonable to believe that there is a higher probability of the one with higher income paying off his or her loans.

I don't think DHH or his wife are a risk with respect to being able to pay off their loans. Any reasonably adept credit scoring algorithm should be able to pick up on that.

> wouldn't it make perfect sense if they have dramatically different income?

The implication in citing their joint filing status is they submitted identical incomes. They have the same address, assets, and she had a better credit score. She is also a woman.

If she applied after him getting his application approved, this might very well explain her tiny line of credit: the algorithm didn't look at him and her alone in isolation, but at both at the same time, saw a certain combined income, saw he already has a large line of credit issued and concluded "I am not giving them another 3000 bucks or whatever, they already got that, so let it be 50 bucks or something, the minimum we can give to a customer, because really they together used all the credit we're willing to extend already when he got his card approved".

If she applied first, she might have gotten the big chunk, and he might have ended up with the tiny chunk.

Or it might be that the algorithm training data was just biased against women, which is entirely possible as well.

The black-box-ness, even to employees, of course, is a huge problem.

My wife and I fit that scenario. I was on the card beta. She applied (well, I applied for her) as soon as it went wide release. Exact same credit limit, she got a five percent lower interest rate. Applications were filled out identically, I did them both.

Credit card limits are usually based on household income.

i don't think that income is better then historical data of you paying on time.

i have 3x the pay of my wife but she pays always on time and i get overdue notices because i forget or don't care.

Income is generally supplied by the applicant at the time of application, and they presumably both used the same household income number.

DHH said his wife actually had a higher credit score than him.

So what? Credit score and income are two separate things. Credit applications look at both.

Doesn’t that concede the point that married couples can have different credit profiles? How much higher is her score? What is the max amount of credit GS will issue to a household? DHH has a history of acting righteously angry without having all the relevant facts.

> DHH has a history of acting righteously angry without having all the relevant facts.

Well, if you know a better way to get the relevant facts without throwing a fit on social media until hopefully a state AG takes note, then maybe we’ll all do that next time instead.

Well, it seems like both you and the twitter thread have anecdotal evidence. I'd say that the fact that an internal investigation is being launched leads credence to the belief that the discrimination is at least possible.

Saying you're going to launch an internal investigation is often a way of calming a PR fire. For a question like this, people can often get an answer with a phone call or email.

In this case, it’s a state government investigation.

In the thread, it was pointed out that her wife actually has a better credit score than he did, and judging from the photo, it looks like ( I may be wrong ) that she only gets like $50 of credit limit.

I had the same thought as you did when I first read it, for those of us not in US, it seems strange and we assume those Credit Score and Algorithm works as intended, like in your example of past loans and payment. And from experience they tend to be consistent and can be easily explained.

In this case however it seems something is very wrong.

I am still thinking and not sure if it is Apple to be blamed. It is easy to just point a finger to Apple, but in reality our financial and insurance system works pretty much the same way, changing these algorithm will require lots of work. Luckily GS is new to all these consumer business and changes are much easier as compared to other banks.

>it looks like ( I may be wrong ) that she only gets like $50 of credit limit.

She actually got a higher limit than that but spent the majority of it. The complaint was that she had paid off the entire balance that she had spent already but the limit wasn't reset and wouldn't reset upon payment but would reset at the end of the billing cycle so she only had a $50 limit until that point.

Reply on news.ycombinator.com