Hacker News Reader

Comment by notelonmusk | Hacker News Reader

Exciting research. Not my area of expertise. An opinion:

> without any knowledge of the game rules

I'd prefer 1000 times an AI that can explain to me why an opposite-colored bishop ending is drawish or why knights are more valuable near the center, and can come up with those and more new concepts/relationships/models on its own (regardless of whether we have given it the game rules or not), than a black box that is excellent at beating you at chess but you can't understand or trust. Adversarial examples (false positives) are supporters for this preference.

View on HN

There are ways to reverse engineer neural nets to try and understand the reasoning but AFAIK none of them are very good.

For example, I went to a SAS/STAT course a few years ago and one of the exercises was to train a simple neural network and then use it as input to generate a decision tree, that would (with pretty good accuracy, around 95% or so) explain the choices made by the neural network. I think the given scenario was that the neural network was used to decide who to send marketing emails to, and the head of marketing wanted to know why it was choosing certain customers.

The problem with this was that it didn't offer insight into why decisions were made, it could only show what variable was being branched on at a given point in the tree. Also while it worked with this simple example, more complex NNs generated huge decision trees that were not usefull in practice.

Generally, it's a superhuman task. No one yet came up with an explanation of how to tell apart a picture of a kitten and a picture of a puppy that doesn't rely on our common low-level visual processing.

That is the explanation mechanism should take into account what kind of explanations humans consider understandable. For tasks that can be represented as mathematical structures (like two bishop ending), it's probably simple enough. For tasks that we don't really know how we do them (vision, hearing, locomotion planning), the explanation mechanism will have to learn somehow what we consider a good explanation and somehow translate internal workings of decision network and its own network (we'll want to know why it explains it like it does, right?) into them.

Reply on news.ycombinator.com