But wouldn't that cause problems with disclosure of patient data?

I.e., if you want to explain ML decisions, you'd have to provide the training data, which is sensitive data.