If a person and not an algorithm decides,there is no full and verifiable explanation. If we want that we need to move to fully algorithmic decisions to start with.
But a person can (and will, sometimes) be held accountable.
Nothing stops us from holding algorithm users accountable.
It certainly would help then, if the algorithm was scrutable.
It could be judged on outcome alone.
If you already know what the outcome is, why bother with the model in the first place?
Sorry, not the outcome of the model, but the outcome of the use of the model, i.e., the ex-post effect.
Who is holding loan officers accountable?
I agree. Also it’s worth noting that there are multiple, reasonable definitions of “fair” and separately following these will not result in equivalent outcomes. So no matter what is done, there will be room for criticism.