The Dark Secret at the Heart of AI

This is not a secret and is a well known fact of deep learning.

It's also a misnomer to think that we don't know how they work. We know exactly how they work. What is difficult and arguably will become more difficult as we go along, is investigating the exact parameter pathway that resulted in a specific output of a DNN.

Ok, well that's actually way easier than understanding how a person came to their decision/conclusion because we can actually evaluate the string of inputs and the training history of a net. If you ask someone how they made a decision, it will be mostly FUD because we don't understand granularly how humans reason.

Looking at outputs and moving the baseline as they improve on a tangible, quantifiable task is really the only way to do this in the long run IMO.

I think people want a decision tree visualization that includes biases from previous datasets etc... and I doubt if that is tractable as we inch closer to general AI.

I'd go further to say, that even if we did have that, it would still be intractable to evaluate. We'd need an AI to verify AI explanations - so at some point you just have to trust it.

Perhaps the solution in this case is to put the AI in sort of a sandbox, where it has control up to a point. But there are independent anti-collision systems in place with override power.