Hacker News Reader

Comment by SkyBelow | Hacker News Reader

What I dislike far more than the idea of using such systems to predict social outcome is that the usage of such systems is done behind closed doors. I would be much more willing to accept such systems if the law required any system to be fully accessible online, including the current neural network, how it was trained, and training data used to train it (if the training data cannot be shared online, then the neural network trained from it cannot be used by the government).

Independent companies using AI is far less a concern for me. If they are snake oil, people will learn how to overcome them. Government (especially parts related to enforcement) is what I find scary.

View on HN

In my country, a relatively recent law added an obligation for the government to give on request a detailed and joe-six-pack-undersandable explanation for how an "algorithm" has reached a decision pertaining to that person.

I've therefore started stockpiling popcorn since this law was announced for the inevitable clusterfuck that was going to happen when this law would have to apply to a decision taken using machine learning.

(Which is pretty much impossible to explain the way the law requires to, because even those that made the neural network would be quite at loss to understand themselves how exactly the neural network came up with that decision, even less being able to explain it to your average person !)

Maybe they can use something like this?

https://blog.acolyer.org/2019/11/01/optimized-risk-scores/

They optimise a simple set of decision rules which has reasonable accuracy in their application, quite cool really

Using an association between features to make a prediction about something, rather than measuring the thing itself, is exactly what’s meant by “prejudice.” Even when the associations are real and the model is built with perfect mathematical rigor. ML is categorically unsuitable for government decisions affecting lives.

You seem to think this level of prejudice for prediction is wrong. Why?

If someone has killed 12 people, being prejudice about their chance of killing another and using that to determine the length of a sentence seems reasonable.

Even with something like a health inspection. Measuring how they store and cook raw chicken is about predicting the health risks to the public eating it, not about measuring the actual number of outbreaks of salmonella. And even if they were to measure the previous outbreaks of salmonella and use it to prediction the future outbreaks, that is still two different things.

I understand and agree with the outcry around using ML areas like criminal justice, but there are some really compelling examples of ML being used by governments to help citizens[1].

[1] https://www.kdd.org/kdd2018/accepted-papers/view/activeremed...

The problem with leaving this to independent companies is that some of the most natural application areas are dominated by independent companies operating as a cartel -- think about credit scoring. The fact that the data and models are a natural "moat" (in the sense of Warren Buffett) is all the more worrying.

The difference is the level of threat between a private company denying you a loan and the government deciding you should spend 10 years in prison. Please don't misunderstand, I'm not saying the former is good, only that it isn't nearly as bad as the latter.

The "independent company" part doesn't work though. If Facebook comes up with anything useful, the US government walks in, grabs the data, then issues a gag order so nobody knows. It simply wouldn't matter that the government was officially restricted to "open AI".

A judge using his experience and judgement to subjectively set a jail sentence is as opaque as a proprietary algorithm. He or she may cite reasons for the sentence, but nobody is verifying that judges' sentences are consistent with the criteria they cite.

You are right and I see that as a flaw in the current system that should be fixed.

Reply on news.ycombinator.com