Trying to picture the "right to explanation" being applied to a Google car or an automated vacuum cleaner. Mm...
In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
In more seriousness, would it even be feasible for recent machine learning algorithms to explain _why_ they opted for a decision?
Why was my claim refused?
Because you have a large number of consonants in your surname which, when combined with the fact your phone number has a prime number of "4" digits, leads to an increase risk of fraud.
Why do those factors indicate fraud?
Because our training data indicates that they do.
Why does your training data indicate that?
...
A bigger problem would be when your features actually turn out to be proxies for 'undesired' features. For example, if you do put names in as features, then any credit risk system will learn that e.g. "Tyrone Washington" is a higher risk customer (assuming everything else is equal) than "Eric Shmidt", since those names are highly predictive of most socioeconomic factors; however, you'll risk a judge saying that this implements racial discrimination even if you exclude race directly, since those names are even more predictive of skin color than income.
However, if you do some reason analysis, then this is useful and gives a lot of interesting information.
Names will be correlated with socioeconomic status and ethnicity, and also with education level [of your parents], income, etc.
Addresses obviously are strong indicators especially if you combine them with data about that location or even particular building if that's available.
Even phone numbers can carry a lot of true information e.g. location, or for countries that handed out new ranges for mobile phones, it would be correlated with how long you've had your phone number/how often you change them, which is a proxy for stability.
I'd bet that you could even build a somewhat predictive model of credit risk based on character-by-character analysis of email adresses - even excluding domain names, the fact if someone chose fluffybunny420 or jonathan.cumbersnatch carries some signal; not very much, but if you don't have better data then it will serve as a vague proxy for age, lifestyle and frivolousness (and in case of domain names, possibly employer) which all are important influencers of risk.
"Redlining" practices aren't bad because they don't economically work. They do economically work; if one demographic is societally disadvantaged, you don't do your firm any short-run favors by catering to them, because they aren't where the money is. They're bad because they work by unfairly shifting the pie away from people who don't actually deserve to have the pie shifted away from them---it's a variant on "sins of the fathers" reasoning to bias on categories like surname, or race, or demographics that correlate strongly with those demographics (like geography, in cities in the United States). There are moral reasons to not allow those practices to get codified into algorithms so the people using them can excuse bad behavior as "just following the computer's orders."
I think name-based decisions are pretty obviously racist. I'm wondering about the less correlated variables such as addresses.
For reference see an older thread on this very topic: https://news.ycombinator.com/item?id=10388795
Generating human-readable explanations from a machine learning algorithm or NN is probably an impossible problem to solve in full generality.
However, for many particular systems, doing so should be feasible.
Also it should be possible to train the NN model to explain its decision, even in plain language.
I think the "right to explanation" law is pretty sensible, people should have a right to question the algorithmic judgment that impacts their life. It will be somewhat inconvenient for big companies applying ML to make these decisions, but it won't ruin their business.