Split brain patients will often have one half of their brain invent explanations for why the other half did something. The explaining half believes these explanations to be true. Which suggests pretty heavily that even the decisions that humans make aren't as explainable as they seem.

Imo, this whole explainability thing is an incorrect human intuition that we're imposing on AI. If you want unbiased AI, explainability isn't the way to do it. Instead judge its performance against explicit criteria for what counts as bias. If those criteria don't exist or can't be agreed on then the problem is hopeless from the outset.