I think this is a major challenge too, although it depends. Some models are easy to reason about, e.g. Bayesian graphical models, while black-box approaches like deep neural networks are not.

One especially problematic issue is: if a model is too complex for humans to reason about, then a business could encode any kind of illicit behavior in the form of model parameters they like. Even if someone could prove that the model is biased one way or another, there is complete plausible deniability for the business, i.e. "I didn't make that choice, the learning algorithm did". We're in for some very interesting legal battles related to this, I think.