Hacker News Reader

> An especially interesting thing about machine learning algorithms is that even understanding the algorithm itself will often tell you nothing about its biases

A related problem is that many (most?) machine learning algorithms cannot tell you WHY they make the decisions or classifications that they do. Suppose you are a bank, using a neural net to decide on credit applications, and the net turns down an applicant. You are required by law to tell them why they were denied credit.

How do you do that for a neural net?

I can think of one way: try tweaking their application until it passes, and then derive an explanation from that. For instance, if you find that they would pass if they made $20k/year more income, you tell them they were rejected because their income was too low. I have no idea if that would be sufficient to satisfy the legal requirements, though.

Offering them a clear separating criteria in the passing and non-passing portions of the space of all properties considered is definitely a good idea. And, as an explanation / description, it's independent in form of the algorithm used.

Well you don't train the algorithm to just give you Yes/No. You tell it to give you

a) Yes, SUPER green

b) Yes, Green

c) mmmmmmmmmmmmmmmmyeah, ok, but watch this guy

d) please review manually

e) no, too many other loans

f) no, but maybe yes, with more collateral

g) no, convicted felon

h) no, unemployed

i) no, your wife has a billion loans

j) no, insufficient money down

k) ...

(there's like 300 or so different reasons in practice)

So the inputs to the neural net would be all the information you know about the applicant, appropriately encoded. Then you sprinkle hidden layers to personal taste, and the output layer would be a->i binary neurons.

This seems to miss the whole point of ML: you want to consider all the features holistically. As your GP said, you could tweak one parameter until it passes. But which parameter? If you have 300 features about an application, but they have an income of 10 billion dollars, of course they're accepted. If they have 0 income, they're rejected. You don't need a fancy algorithm to reject people for the obvious reasons you've described. The problem is the fuzzy gray area where I make 100K a year, but I seem to have too much debt. You don't want to kick this back to a human and say 'do it manually', you want to say 'Based on all his features, his risk of default is x%. This is (higher|lower) than our threshold'.

edit: Just to clarify, for banks it doesn't make sense (in my mind) to train a classifier based on a bank manager's past decisions. I would train a regression based on the historical default rates, so the algorithm would predict the probability of the client defaulting on the loan.

No you don't. As the parent poster says, if you screw up, you want to be able to figure out why. Unless you enjoy digging for months for biases in large incomplete datasets with a panicked guy screaming at you, you'll have at least a couple dozen output neurons (you can always also have a yes/no one if you like).

In practice all the parameters will fire to some extent, right. You need some function to resolve it. Generally people start out by giving the "no's" preference, then you switch to something more complex, then some product manager suddenly declares the law: give "yes" preference.

So the output of that network will be a) 15.15% b) 74.3% c) 55% d) ... and so on. The highest values are considered and some function chooses from that.

It's how it works in the human brain as well, but the solution that wins is the one that gets control of the brainstem. Generally being the first to respond is pretty much guaranteed to get you control, but something else may override you later. The human mind, in reality, is not a single whole, but is ~300 separate minds that could, if it were necessary, operate alone. Each of those is constantly coming up with responses to inputs, and they're all linked through the brainstem, which is also the link to the rest of the body (except the eyes, which are only input to ~6 of them, and the ears, which also are connected directly to ~6 of them. That seems to be to enable brain-eyes bandwidth to be ~100000 times more than all the bandwidth it has to the body, and brain-ear bandwidth ~100 more).

True, but I believe the argument still stands for other ML algorithms, such as support vector machines. There is no way to query directly why the SVM maps an instance to a label.

It seems like in SVM the best way to find out 'how to pass' is to find the shortest path from your point in the feature space to the separating hyperplane. The length of that vector in each axis explains how much you need to change to cross the threshold.

How about you run 20 vector machines ? Each only identifies how likely one reason is (using distance from separating plane as a "how sure is this SVM" measure). Besides, SVM do not have a particularly good reputation due to the need for good kernels, which requires either massive computing power (just have 10 million kernels) or people with very good domain knowledge. And if you need people with very good domain knowledge, why bother with AI ?

AI is hard and people who know both a lot about statistics, AI and programming AND a lot about, say insurance claims, well good look finding them. People with good knowledge about insurance claims and good knowledge of excel, they're, well, not a dime a dozen, but certainly much easier to find.

I think that this is more a criticism of the way that machine learning is commonly done, rather than a valid criticism of machine learning per se.

I have certainly (sometimes) been able to build insights using (some) machine learning approaches, but it depends a lot on having the time, the resources, and the right organizational culture in place. Unfortunately, a lot of people approach machine learning algorithms expecting a magic bullet; attracted by the hyperbole, so the self-deception starts even before a line of code is written.

Doubly-unfortunately, these people are usually the ones offering the highest salaries for "data scientists" and "big data engineers". Like most work of consequence, the mundane outweighs the exciting by a ratio of at least 10:1. Obtaining and cleaning the data, understanding what is missing, the sampling biases that are present, testing and training the algorithms, all of these things are deeply important and seemingly mundane; yet it is these very activities that help to build deep insight and experience.

Perhaps we should pay our data scientists a bit (a lot) less, and we would get fewer flashy presentations and more substance.

This all depends on the algorithm and whether the algorithm is discriminative or generative.

Regarding the cat identification task: If the goal is to identify cats from pictures, then what would you expect to learn about cats? The high-level features of a cat that we combine in a hierarchical fashion to arrive at 'cat'? After all, if we are just looking at pictures of cats, I think this is the best a human would be able to ascertain given numerous examples. If so, then algorithms like Convolutional Neural Nets do in fact provide such information - one just needs to look at the convolutional kernels to see these features emerge over the hidden layers.

A simpler example is the task of identifying handwritten digits - and this is the example that is almost always referenced when discussing CNNs. A well-trained CNN on pictures of handwritten digits will extract things like diagonal and straight edges as high-level features and then combine those in appropriate spatial orderings to identify the digits 0-9...which is likely similar to how we discriminate between such digits.

There are musical analogs as well of using NNs that extract out from a time-frequency representation, for example, individual instrument timbres (or at least decent approximations) in a mixture of instruments sounding simultaneously.

How is this really any different than solving problems utilizing the raw speed of computers? That is, simply enumerating all possibilities of a problem might not really tell you much more than whether there is an answer, either. However, armed with that some folks may be able to find more.

To take your cat example. Now hook up and improve this algorithm in cameras out in the wild and it will become much easier to track where cats are found. Want a rough count of the unique stray cats in an area? Is there a camera that covers it? :)

That is to say, sometimes we simply need the data that these algorithms can provide in order to feed other decisions. This is really no different than any other situation where you have to infer a wider truth from limited data. We are just expanding our ability to do so, now. Not necessarily by finding the fundamental equation to something, but by applying simpler equations to more and more data. Right?

You do feel empty, but there are plenty of other things that need fixing :)Or understanding.

And often there is no perfect solution, and the solution changes over time. In these cases machine learning is very handy.

it sounds like there's no problem with machine learning itself, it's how you are using it.

if you're not learning about cats then you need to try a different strategy that deals more directly with them.