Hacker News Reader

> Obviously we know they were the result of a deep minimax-style parallel search with a certain evaluation function, and we could simulate a similar search by hand if we wanted to.

You don't really need to though. The computer can justify its evaluation with a principal variation (best play by both sides) leading to a leaf position of the same value. And if a human were to wonder why at any point in this variation, some other move is inferior, the computer can again easily show a principal variation branching off from that other move, leading to a worse (or equal) result for the deviating player. In this way, minimax results are FAR better justifiable than neural net results.

Exactly, the Deep Blue's move explanations are boring - they all boil down to "based on the inputs and rules programmed, this line of moves has the best overall outcome to a depth of X" where X is however deep it goes.

You can try to translate that to human methods of understanding, but that's not how the computer "thinks", and attempting to do that translation leads to misunderstanding. Kasparov may make moves because he wants to get a better board position or knows his opponent is weak to certain positions (I have no idea how experts explain moves) but the computer isn't programmed that way.

I don’t think that’s weird at all, I think humans can intuitively grasp “I simulated 100,000 games starting from the current board state and going at most 50 moves ahead and in games where you did X you reached a strong position most often.”

Is that a useful explanation for a human who is training a wetware ML model, not really. But it’s really understandable compared to trying to explain neural nets.

>in games where you did X you reached a strong position most often.

FYI in standard minimax, the search is for the move that leads to the strongest outcome - the "most often" bit doesn't factor into the decision making process.

That's punting the problem of explaining down the line. All the PV tells you is the computer is making this move because it thinks that 20 moves down the line, you'll end up in some desirable position.

It doesn't abstract or summarize any understanding. If you say, "well, what if they do Kf3 instead", all it can show is another PV. It can't tell you "knights on the rim are dim", or "this pawn needs to move now to prevent the bishop entering the position in 5 moves".

You can try to build an explanation on top of a chess engine, but it's very difficult to come up with generalizable or clear explanations beyond very broad heuristics (like "knights on the rim are dim").

It doesn't provide a clear explanation in those terms, and in some fraction of positions, there is no simple such explanation. But I believe that in a decent fraction of cases, a human chess player probing PVs will gain a reasonable understanding of the winning or drawing strategy involved, and that such interactive PV probing is often a better form of explanation than a concise non-interactive textual one could offer.

There is no equivalent to that in neural networks.

Note that PVs explain the connection between the root position of minimax search and a leaf positions. Evaluation at the latter is easy to explain in terms of features like material, mobility, pawn structure, king safety, etcetera.

I agree that a human can use the PV to get insight into why a move might be good. I also agree that non-interactive explanations provided by a chess engine will often be inferior. I think one way to break it down might be that the PV (or sets of PVs) can provide a useful explanation of tactical value (Kf3 leads to mate in 6), but struggles to explain strategic value (Kf3 leads to the positioning opening up, which broadly makes it harder for you to press your advantage).

> There is no equivalent to that in neural networks.

The first example that comes to mind is using backpropagation to determine how much influence each value had on the output result, and then rendering it in a human-readable way. Stuff like this:

https://miro.medium.com/max/4800/0*Y3Yi7cEueF0XLZP-

https://miro.medium.com/max/1100/1*IPhQ12OKnxH31AQNja6FnQ.pn...

Interactive PV probing is just playing chess against the AI to better understand the position. That is literally the interface for which people built chess AIs.

Unfortunately that misses a lot of what makes grandmasters worth listening to. GMs will say things like "Kh6 almost works, but fails to ..." which is not something you can get out of a computer currently.

Computers currently fail to explain any moves which are good ideas, but are not the best move now. Humans normally describe these moves as "resources" which are good to keep in mind for the future or to play later when the board cools down.

> anything new

It is nothing new. It is called (at least) "the problem of transparency".

The chess context is probably not the best, because many systems allow a lengthy complex explanation of the response (I cannot remember now the exact workings of Deep Blue - it has been a while last time I met the full info).

It is a real problem in general, because we may not just want responses but we may want "to learn something", and specifically, because ANNs applied to e.g. Decision Making, or ANN based Decision Support Systems, pretty much require that natural intelligences assess the "oracular" proposals.

And of course, "understanding" the engine is required to advance it.

It is a key problem.

In 1963 Marvin Minsky called it “The Credit Assignment Problem” — which, among a multitude of variables, were most important in solving an AI task?

I suppose this is why it is called "machine learning" and not "human learning"!

We deal with intelligence to either solve problems, or to benefit from intelligence to advance our own.

If you valued technical advance with a reduced concern towards civilization, ie. power over wisdom, ie. more "how" and less "why", you would get - well, for example, the present situation, that some consider "strongly suboptimal".

Some said that there have been two big branches of science fiction, that revolving around power (spaceships etc.) and that revolving around information, and that we have been just lucky to have reached the latter.

But some forms of machine learning actually are interpretable, which makes them useful in scientific investigation, which doesn't just want a black box. A good example of interpretation machine learning is decision trees -- given a series of variables and outcomes you can find out which variables are important to the outcomes and in which combinations.

> And of course, "understanding" the engine is required to advance it.

I don't think so. People are clearly making huge advances in machine learning even though they can't explain the systems made 10 years ago, let alone more recent ones. There's a lot of trial and error / cargo-culting involved. Some researchers go as far as comparing machine learning research to Alchemy:

https://www.science.org/content/article/ai-researchers-alleg...

> many systems allow a lengthy complex explanation of the response

All computations have this property: you can execute the code by hand for as long as you have time to. But if you do this for long enough you become a mere observer of the computer's steps, which won't give you any intuition as to why the final output is the way it is (unless the algorithm being run is very simple).

> "understanding" the engine is required to advance it

What I meant was more on the lines of "LeCun proposed Convolutional NN recognizing a promising paradigm", or "our prespective on Artificial Vision changed as we realized that the ANNs recognized textures instead of figures".

> All computations

And what I meant there was that in the chess systems of topic the calculations performed have been "as if symbolic", ie. reasoning on branches of consequences of the type "what could happen if I move this there" - even when adopting ANNs in some parts of the architecture -, as opposed to, say, "computing weights". That is the level "of machine code", not that "of electronics".

But an AI must “understand its reasoning” in order to revise it efficiently to learn from its mistakes. If revision is impossible, the only way to correct its reasoning is to re-train the system from scratch. But in doing that, you can only hope the problem is fixed, and worse, that new problems were not just created.

It is very rare, but I must congratulate.

An aim is "non-deterministic virtual self-modifying code".

There is also the problem of label noise. A label might not be necessarily correct, especially when the dataset is large. Nobody got time to check them all, and even if you put two people they might disagree.

Two things are new:

1. Machine learning is increasingly being involved in important decisions.

2. Deep neural networks have become a popular technique, and their results are particularly difficult to explain.

If you're a bank and your AI says somebody is likely to default on the mortgage they're applying for and you can't demonstrate that their race or a proxy therefor wasn't a factor in that calculation, they'll have a settlement and won't need a mortgage.

Older ML techniques are explainable in nice ways. With something like logistic regression you get a very nice "this feature contributed this much to the final classification" breakdown. It just turns out that logistic regression is considerably less effective than RNNs, which have all these synthetic features within the inner layers that aren't easily meaningful to humans.

I implemented an interpretable LSTM from this paper https://arxiv.org/pdf/1905.12034.pdf. It works by segmenting the context for each input variable, and then applying an attention layer to interpret each variables effect.

I applied it to mortality prediction based on ICU data, the prediction/recall was pretty good considering the small size of the dataset used.

How would you debug such a system?

E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.

If you have no idea how the program arrives at its results or even if the results are correct or not, how would you be able to find that bug?

Similarly, how do you make sure the program doesn't learn garbage features or overfits on your testing environment? There are enough war stories of image classifiers that just learned some subtle lighting differences in the trainset photos of something that actually distinguished the object in question.

Or how do you prevent models learning certain features that you specifically don't want to learn, such as skin colour?

I think developing those models without trying to understand what they do has a high risk of leading to magical thinking.

> E.g. imagine somewhere in the code for your chess supercomputer is a genuine bug - some dumb typo, I don't know - which leads it to make the wrong decisions in some situations.

If the evaluation function executed at the leaf nodes of the search tree contains a bug that massively over-evaluates a position, you might be able to find this by having the chess computer play against a different computer that exploits this bug. Then you'd observe the positions at the end of the principal variation as some other comments mentioned.

Even if you can do all that, that still leaves the main problem unanswered which is to explain the correct moves (or rather, moves "assumed to be correct").

I think these kinds of articles are geared toward people who do not know (even superficially) about the models. Perhaps the sentiment they want to elicit is "If the person who created these models cannot explain how they work, then he is not so different from me."

> Whenever you perform a massive amount of computation, you can get results that are nearly impossible to explain.

My thought on reading your comment was Douglas Adams and Deep Though’s 42.

The explanations used to explain or justify human behaviour are just as speculative though, really. We can try and hypothesize, but that's it - there is no engineering design specification for the brain, nor even a debugger.

We practically invent our explanations post facto, first decide, then verbalise.

Side question, what's better at chess today - neutral network approach or deep blue approach?