Hacker News Reader

Comment by Tade0 | Hacker News Reader

My favourite is still Art. 22, p. 1:

"The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her."

Which means that if e.g. a bank declines your mortgage application based on a decision from a fully automated system you have a right to have a human being review it.

Interestingly this made banks reluctant to use AI and helped with efforts in development of so-called "explainable AI".

View on HN

Just did exactly that with Avant. Could see where I was initially approved but then their manager said I was "still on contract" and quashed my application.

doesn't sound like this worked in your favor, or did i get that wrong?

I read this as you do, but it's still a big win.

If you make a machine learning model explain itself to humans, you open the possibility to challenge the model's conclusion. It's not just the question of whether the model does its math right - it also means we, as society, can say, "I see your math and it works out, but because $POLICY, you're not allowed to take these inputs into account". It means adding control points to a previously opaque process.

you are right, and i absolutely agree. basing any decisions that substantially affect someone on machine learning alone should be illegal (even if it is the right decision)

I was denied, but now I know why instead of just guessing. I could see that my "willingness to pay" (unsure what that means) etc and income were all fantastic, their manager just didn't like that I was on contract instead of a permanent employee.

so arguably, in this case the machine learning algorithm was possibly right, and the human was wrong. on the face of it, not using machine learning here would not have changed the outcome.

what i am wondering is though if this can't be used to everyones benefit.

apart from neding a human to verify a decision, how about, if the human and machine learning decision disagree, then the consumer gets the right to appeal for a second human review?

Funny thing is that I specifically opted out of automated decision making, having been declined before. It wasn't a big deal, really. EU lenders are far warier of contract jobs than American ones.

oh there was an opt-out. that's interesting. how does that work? you obviously opted out before, but could you also opt out after they tell you what the automated decision would be? if so that would effectively be that kind of appeal that i suggested.

I applied for a loan a year ago, didn't opt out got denied, and wondered why. Then I applied again ~4 months ago, opted out of automated decision making, and the subject access request showed the conversation.

What about a human reviewing the AI review and just nodding all along ? Does that fly ?

Yes, the Nodder has to have proforma authority to override the AIs decision.

In that case there is a human being who is legally responsible for the decision that is made. I have no problem with that. If the person is "just nodding", with possibly illegal consequences, that person can be sued.

Why not suing the bank? It always works, AI or no AI.

Well, if all the decisions are given to the AI, it's harder to establish mens rea. They can just claim they never intended to discriminate anyone but that it was only an "unfortunate glitch" in the system. Of course, this defense might not hold, but AI is a decent legal cover as it's more difficult point to any single person knowingly doing anything illegal.

Some countries have pushed it further : for administrative decisions where a program was used (whether alone or helping a decision), all the steps of the algorithm resulting in the decision must be provided in clear and simple language (if a request is made).

But also banks don't have to explain you why they have taken this or that decision.

It is about whose mistake or choice it is, if a bank denies loans to $ProtectedCategory because of a bug in its software it is one thing, if it is because employees have biases and discriminate applicants it is another.

Reply on news.ycombinator.com