The Secret Bias Hidden in Mortgage-Approval Algorithms

As the article said, "this is structural racism".

Interestingly, the models did not do anything wrong. They are only trying to predict the future based on the past. The colored groups have been discriminated and tend to have less salary/less stable jobs etc, which causes the model to characterize them as high risk for mortgages. This is technically correct though clearly unethical and promotes continuing structural racism.

Unfortunately, not all technical engineers who trains the models may realize this. In fact, a few years ago there's news about Facebook use AI to pre-filter applicant resumes, and it results in applicants being penalized for being female.

This is another reason I believe "explainable AI" is important. Humans can easily identify the "correct but unreasonable" parts. Machines probably can never learn the human society's ethics by themselves.