Same with drawing images and understanding language: this is not solved yet.
This is like showing an answer on an exam but failing to explain how you got it. I doubt you can get away with this.
> But humans failed to find an algorithmic solution for Go.
sure we have algorithmic solutions for go, they're just not very good.
> All they could do is to throw a lot of data and get a bunch of coefficients without discovering underlying rules.
that's not completely true either, the special thing about ~alphago~ alphazero* was that it learned by playing itself instead of learning from a pre-recorded catalog of human games (which is the reason for its - for humans - peculiar playstyle).
now i'm not sure how you're arguing a neural network trained to play go doesn't understand the "underlying rules" of the game. to the contrary, it doesn't understand ANYTHING BUT the underlying rules.
explaining why you did something isn't always easy for a human either. most times they couldn't say anything more concrete than "well it's obviously the best move according to my experience" without just making stuff up.
*edit: mixed up alphago and alphazero
In other words, the algorithm is very long for a relatively reduced programming language.
We don't understand the explanation, though it is correct. Not sure if the problem here is with the capabilities of the examinee or with the examiner.
The explanation is perfectly sensical, just too complex for humans to understand as the model scales up.
The thing you're looking for - a reductive explanation of the weights of a ANN that's easy to fit in your head, does not exist. If it were simple enough to satisfy your demands, it wouldn't work at all.
Meanwhile things like stock markets attempt to with things like partial future prediction, which means all possible outcomes are not calculable in finite time, hence they use things like ML/AI.