The hidden reality of ridiculously complicated algorithms
the-tls.co.uk
the-tls.co.uk
Personally I don't see much difference between how opaque the world of automation is and how opaque law, or government bureaucracy, or media production, or any other human industry is from the point of view of a non-expert. But perhaps for many people, it is easier to trust long chains of human processing than long chains of machinery.
With humans one can't, and as a result there is blatant unfairness everywhere. Nepotism, racism, corruption, ... these algorithms can, if allowed, make it very hard for those to exist. Humans cannot be trusted to make decisions about other humans, not even with the best of intentions (because they still destroy people against the rules to protect the rules).
Needless to say a great many politicians are trying to stop the algorithms. Because they can't explain what they've done is the best they can come up with (which is stupid, the better argument, imho, is that one cannot with certainty predict what the algorithms will do in any given case. So if a neural network was the law, there would be no way to know what the edge of legality is. They might send Maria Theresa to jail, so to speak, for tying her shoelaces crossed or some other stupidly small unique thing)
Besides, do a bit of law, the reason humans can explain and algorithms can't be explained is as trivial as it is horrible: humans simply don't take more than 2 or 3 variables into account even for critical decisions. This is stupid, for obvious reasons, but it's also why you can explain. I assure you, for a 3 neuron neural network, just taking 3 factors into account, there is no explainability problem.
this is a fact often complained about in outcomes. Certain variables make seemingly malevolent decisions innocent. But those variables can't or aren't taken into account, so ... boom hammer comes down, life destroyed.
Why does your local paper run stories featuring some businesses but not others? Why do college admissions tend to overrepresent some populations and underrepresent others? Why are women paid less on average for the same titles? Why do prosecutors disproportionately bring charges against minority suspects with equivalent weight of evidence?
You can't get answers about those human decision processes just by asking - it requires research, competing theories, and sometimes legal action forcing people to share info.
Explainability of decisions isn't about how much tech is used... It's about how open a system's operators are to sharing info, and how much effort everyone is willing to put into interpreting it.
I question the word "intentionally". I would suggest that human systems evolve to be opaque, because that helps them be resilient and survive. If a human system is truly understandable by anyone, then it can be destroyed by some of its members/adversaries and sooner or later someone will.
it reminded me of that famous article about the strange ritual practices of the Nacirema people. (which, if you don't feel like reading it, is actually just an account of 20th-century American life, deliberately made to sound as weird as possible.)
it's part of a book, so hopefully the rest of the book doesn't take the same tone.
I think the warning the article wants to convey is that the current trends could - eventually - lead to a society in which personal freedom and even laws are undercut by completely intransparent algorithms. Intransparent both because companies are allowed to keep them under wraps (and will absolutely do so) and because even if they were made accessible, only a small group of experts would be able to understand them. Sometimes, not even then as the article shows. I think that is a valid concern.
The last point - that understandability is such a low priority in development that even frequently the designers themselves don't understand what exactly is happening - is a different problem, but maybe less important than the first one.
To overcome "intransparency" one may become an expert or trust expert(s) asserting that they did sufficient analysis of the thing in question (here an algorithm) and are confident.
In addition, with time, one sees that the thing does an apparently adequate job and doesn't produce so much patent problems. Such things are at first "feared" and rarely used, then gradually gain adoption/traction (benefiting from positive feedback).
No one can fully understand the whole technical and organisational context of jetliners, but most of us use them because apparently competent and objective experts continuously check all this, and because problems appear to be rare.
This approach may/will fail in catastrophic ways for various causes, either psy (one loses too much autonomy, and incentive) or systemic (the composition of various things may produce an inadequate system).
> The algorithm was also constantly changing. The data inputs were flowing into the algorithm in real time, but the actual weights, measures and trade-offs that the algorithm made weren’t static either. Some of the functions that the researcher had woven in used machine learning – techniques where the machine constantly learned and adapted to what the most important patterns, correlations and relationships were. It meant that the algorithm was constantly changing and moving as the world moved around it, and its diet of data changed to reflect that.
shows the reporter having only a superficial acquaintance, but wanting to sound authoritative, and engenders considerable distrust on my part.
For a much much more credible and vastly more entertaining take on the issue of opaque processes empowered with scary real world decision making abilities, see James Mickens' recent talk[1] on why we should be deeply worried by this, enough to update our traditional computer security model (discussed here[2] also.)
1: https://www.usenix.org/conference/usenixsecurity18/presentat...
If seeing a Jupyter notebook impresses the writer so much, it might blow their mind if they found out that the screen is made up of millions of dots. And that each dot is actually made up of three different-coloured dots.
> The reality is that if the algorithm looks like it’s doing the job that it’s supposed to do, and people aren’t complaining, then there isn’t much incentive to really comb through all those instructions and those layers of abstracted code to work out what is happening.”
If a judge is performing well are you going to run an expensive background audit on him, or open up his brain to try to see how he "really works"?
The difference is that algorithms today are practically run by centralised organisations that control it plus the data, rather than having lots of humans all slightly different making their own decisions in a semi-centralised[^] distributed fashion. But the opaqueness is nothing new.
[^] because most human organisations today are hierarchical, but at least most people have some sense of self-agency.
PredPol as mentioned in the article is closed source and considered a trade secret not viewable by the public. It's the largest supplier of these tools in the world.
Algorithms can blindly follow through into an obviously illogical decision in ways that a human brain doesn't (and when it does, we consider that person unfit for a job as well).
I maintain that the core difference is the way these technologies are being deployed in a centralised fashion by human organisations, but the technology itself is no less capable than actual human decision-makers.
In other words, if you do believe that humans can be illogical, then who told you the human thinking doesn't come from an algorithm itself (of e.g. NN nature)?
Not that this is not needed, of course, because what I wrote is much easier to achieve. I wrote that "Algorithms can blindly follow through into an obviously illogical decision". That doesn't even need the algorithm to be illogical itself, but just to arrive at an illogical (obviously for us, humans) decision. It can still arrive at that logically. E.g. if I program a simple algorithm that "if the car sees a zebra crossing, go full steam ahead" the algorithm will do just that, whereas most humans would question it.
> They often have no feedback on whether they made the right decision
Of course they do. They will see people stood in front of them that have violated their parole all of the time. They can read their case notes and use the information to adjust future decisions. This of course is after a career of standing the other side of the bar as an advocate/attorney of those people, because that is what you have to do to become a judge in the first place.
> and there is no knowledge-sharing between the judges.
Except the whole legal system in the US is built on common-law. Literally the law being applied in a common way to everyone. Literally by knowledge sharing. If the esteemed academic is suggesting that parole decisions are being shared less well than judgements the fix seems to be rather simpler than ML.
The idea that a ML algorithm is somehow less biased than a judge is of course just as absurd. A ML researcher is just as capable of using race, for instance, as a characteristic. They are just as capable of seeing a correlation without understanding the complex underlying causes.
I choose the human judge.
Before anyone comments that ML models are notoriously difficult to explain, the whole point of the regulation is that opaque models are intolerable. Also, https://github.com/slundberg/shap looks like it could be very useful.
As someone who works with machine learning, I find this statement rather misleading. The researcher quoted here has a very strong bias about what it means for an algorithm to be "optimal". A system cannot both "be optimal" and also "biased" in a way that the designers don't like: such a system is indeed not optimal. The dialog in the machine learning community has increasingly been about how we might structure these systems in a way that they are unbiased; it's a shame the researcher (and the author of this article) seem to think that "simplicity" is the only option.
(In case anyone is interested, I recent wrote a blog post to this effect: http://www.cachestocaches.com/2018/7/bias-and-ai/ )
It may comforting to think that all our biases are “baked out” of the models during the training process but if the researcher had been a black man do you believe that they would have arrived at identical models?
Even if the model designer found ways to compensate for his own bias the world that these models operate in are inherently biased. Training data may be subtly biased in ways that are difficult to detect. A model trained on recidivism is likely to be biased if, for example, a lower class black male or upper class white women have different risks of being arrested for the same crime.
Assuming the models themselves are unbiased as trained they may create bias with subtle errors when pushed into production or due to second order effects, feedback loops or errors or manipulation of data. For example during the credit bubble entire industries found ways to raise borrowes FICO scores.
There is also the inherent corporate bias that the researcher pointed out which the bias of the almighty dollar. Compensating for bias is expensive and may even impact salability if an unbiased model doesn’t “gel” with the expectation of customers who are biased. For example a judge may feel pressure of homogenize his decisions or abdicate his responsibility by rubber stamping the models decisions.
Currently we compensate for this in a democratic society with checks and balances. What are the checks and balances for a computer algorithm that is a trade secret for a for-profit corporate entity?
We've pretty much all read the article "Myths Programmers Believe about Names", and various snowclones. We still write dumb name validation code every day, because there actually isn't a perfect way to do that, and we have to ship something. Even when the concepts are perfectly executed, the fundamental assumptions of the model may change. Having separate classes for "tool" and "weapon" may make sense one day, and the next day you need to write the game Clue. Even if you're working with such boring concepts as filtration, you may have to adjust your model when you discover that water is compressible.
All models are wrong, in many ways, both subtle and overt, both currently and in the future. Some may be useful. I'm not sure that there's any solution to these problems except more and better models though.
Oh wait.. didn't that just happen again in Japan?
And here we have the tls discussing heuristics to sentencing which "out performs" the judge...