A New Approach to Understanding How Machines Think

I found it easier to understand the idea by skimming the introduction in the paper [1]:

> TCAV uses directional derivatives to quantify the model prediction’s sensitivity to an underlying high-level concept [...] For instance, given an ML image model recognizing zebras, and a new, user-defined set of examples defining ‘striped’, TCAV can quantify the influence of striped concept to the ‘zebra’ prediction as a single number. In addition, we conduct statistical tests where CAVs are randomly re-learned and rejected unless they show a significant and stable correlation with a model output class or state value.

[1]: https://arxiv.org/abs/1711.11279

> “AI is in this critical moment where humankind is trying to decide whether this technology is good for us or not,” Kim says. “If we don’t solve this problem of interpretability, I don’t think we’re going to move forward with this technology. We might just drop it.”

AI interpretability is a very important and exciting field, and I don't mean to detract from the rest of this article, or from the speaker's work. However:

1) Technology is neither good nor bad in and of itself. It is a tool that can be used for one or the other.

2) If it's useful for something, some people will use it, despite the protests of others. Have we ever collectively decided to "drop it" in the past, where "it" is a powerful technology?

Imagine making a linear regression to predict runners’ speed using data from two individuals timing their speeds. Not a very interesting problem. Assuming they’re both competent measurers, you would probably be best off taking half of the first measure and half of the second measure (aka the mean). But the clever linear solver might notice that you can technically get a better answer by weighting the first measurer by 3.001 and the second measurer by -2.002 due to natural variance and the way things happened to land. We can adjust the solver to not do this by punishing it for large coefficients (ridge regression) and that’ll mostly settle that.

But for a neural network with hundreds of mini logistic regressions built in... seems tough. Interpretability isn’t just knowing how decisions are made but also how much irrationality is being built into it. If one factor is being marked as a huge negative influence, what does that actually mean about the problem space? Maybe nothing. If you actually want an interpretable neural net model, you should probably hand craft layers of domain specific ensembles that you can individually verify and are closer to being self evident. Maybe you won’t know how it determines stripes or horselike, but if you feel good about those two models individually then it’s a much easier task to follow the last step which is: it’s a zebra if and only if it’s horselike and has stripes.

Very interesting although I thought the chainsaw analogy was poor, after all while the user may not understand how it works the person who built it certainly does.

The question of explaining how a neural network made a decision also comes up with us humans of course and is often not easy. Don't know why I didn't call... https://www.youtube.com/watch?v=tO4dxvguQDk

Human rationalisation has it's issues eg “So convenient a thing to be a reasonable creature, since it enables one to find or make a reason for every thing one has a mind to do.”

Wonder if AI will do better.

Just finished reading the paper. TLDR; a linear classifier is trained on the activation values of an already trained network when it is fed examples of a specific high level feature. Assuming the feature is isolated to a certain linear subset of the activation matrix, a vector along the decision boundary can be computed. When feeding examples, the magnitude of the directional derivative in this vectors direction can be computed on the activation matrix, yielding a measure of how much the feature is responsible for the overall derivative..ie. the total activation.

Seems like an obvious technique. A few months back, someone posted a similar linear reverse engineering technique to make a customizable face generator from already trained GANs.

Any ever created decision making algorithm is nothing more than programs. A machine is not thinking. It’s executing code.

"The question of whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim."

- Edsger Dijkstra (EWD898).