It is easy, but it gets political. The sketch of the argument is "Banks should not be allowed to make judgments which negatively affect certain groups of people disproportionately even if those judgments are accurate, because it is contrary to social justice. It would be better if substantially everyone paid more and/or if banks were less profitable if it meant that access to homes/credit/etc was improved for protected groups at the margins. We have a special obligation to do this because of our collective responsibility for historical and continuing inequities which directly result in the fact pattern which tends to cut against the interests of protected groups."

n.b. Not me talking there, just summarizing a view which is very popular.