The problem is, e.g. banks will want to use all the data they can to make a decision on what sort of loan to offer you. Employment history, travel history, family medical history, as much data on your spending habits as they can get, how much money your relatives have, which properties you own and are they likely to rise or drop in value, where, when, and how you drive (which affects likelihood of having a car accident -> unexpected expenses or death -> affects loan repayment), which websites you visit, how much time you spend on your phone, in bars, etc. etc..

Each has a plausible justification for inclusion. But the more data points you have, the better you're able to predict race from them (or anything else you're not supposed to).

So how do you decide if an algorithm is discriminatory? Maybe a whitelist or blacklist of what data it can use?