Hacker News Reader

Comment by robius | Hacker News Reader

We're not going to get very far letting machines guess what we want by making models we can't interpret.

Explainability is important, and critical in applications where lives are on the line.

Explainability where we're guessing what's happening in a black box model won't do either. Nothing but complete transparency of the model and why it's doing what it's doing. Its source code, that makes sense to humans, is needed. Full on model audit. No guessing.

I can think of only one company that's attempting to do this, and it's not anyone you hear working on explainability, including DARPA.

View on HN

1. There are a lot of researchers who are working on interpretability (check on arxiv).

2. This rant has very little to do with the article, and feels like a borderline meme that some people post on all that is ML-related.

If the visual cortex were a machine model, people would be complaining about how we can't explain it and how it's a dangerous black box. They'd probably tout the many optical illusions as demonstrations of this danger.

Yet we don't demand that other humans explain how their visual cortex work. There is a double standard here.

I do not approve of the red herring. Either it is visual cortex, or the intelligence and decision making/planning. One works very differently from the other.

What's the difference between the output of a network and what an expert says? Unless you can probe mathematically why networks or human mind works, there is no explanation for any of both methods. I can argue that humans learnt from examples the same ways neural networks do, you can say the opposite. But we have no way to say that that any claim is false or true.

You can't make a full on model audit on a neural net, but there are architectures such as the Transformer (an attention scheme) that can give a lot of insight into what the neural net thinks. We can also visualise what inputs maximally activate a deep neuron in a CNN. Not all DL models are truly black boxes.

The problem is the models have no reflection capabilities unlike people. The explanation is always done by a really different external system and sometimes by an actual intelligence.

State models (Markovian) are sometimes able to explain things but not always really, especially in complex cases.

On the other hand, humans tend first to take a decision and later retrofit an explanation to it, even if it is completely wrong, and we assume the explanation is the reason and not the effect.

But when you’re in advertising all that matters are the results.

Tommi Jaakkola at CSAIL, in particular, comes to mind. [0] I think a number of others are moving towards it.

[0] https://www.csail.mit.edu/research/interpretability-complex-...

Reply on news.ycombinator.com