I admit that it's unfortunate that AI can't write out their decision logic but I would argue that is because there hasn't been enough resources put into explainable AI. Considering the increasing use of these algorithms, I don't know if that is even a high priority.

I tend to think that people are not as logical as they like to think they are, myself included. Not to say there isn't good reasoning, just that much of our decision making is emotional and habitual over some pure sense of logic.

Systems like BERT seem perfectly rational to me. Are they not just following a set of rules on a given input to modify a state?(In the most simplistic sense of computation). I think the confusion is more over what the goal of these programs are and how do we encode that. This reminds me of the ai system that would pause the game of tetris so that it could never lose. Not we it's programmers intended but still accomplished it's "goal".

Humans make decisions based on arbitrary and capricious preferences, and are good at coming up with logical-sounding reasons to justify those decisions (eg: rejecting a candidate because she is a black woman, but saying it was because she didn't communicate well). A neural net makes a decision based on a perfectly reproducible sequence of calculations. There has been a lot of work published about how we can explain the output of neural nets, and IMO this has much better prospects than trying to explain a human’s decisions by scanning the brain or whatnot.

Is explaining the decision process a requirement? It can still solve extremely complex problems, problems humans couldn't dream of ever solving on their own (e.g. Deepfake).