Similar to the human brain in that respect. Ironic that the human brain used itself without understanding how it works to eventually create something it uses without understanding how it works.
-college degree (Doctor)
-employee referral (engineer)
-FizzBuzz (engineer again)
-licensing exam score (driver)
-popular election (supreme commander of the armed forces)
E.g., AlphaGo is clearly good at Go. Yet it makes moves that can appear inscrutable to a human expert.
(NB: I'm not arguing that inscrutability is or is not an issue for AI systems, just suggesting that third-party validations don't meaningfully address the issue one way or the other.)
It's like trying to create an AI to play chess but the creators focus so hard on keeping it from making moves that result in its losing the queen that they forget that the objective of chess is to win.