This is kind of an ironic claim on a community that believes so strongly in "black boxes" like machine learning. Golly, if their heuristics point in a particular direction isn't that a ~signal~ with predictive power and market value?
Take heed of this the next time you extoll the virtues of blind inference - whether human or machine. Maybe the next time it'll be _your_ soap dispenser that doesn't recognize black people. Or redlines black people and keeps them from getting a credit card, or mortgage, etc.
Because the reality is - there are all kinds of signals like these that are real. People in certain areas are more likely to be poor, to default on their loans, etc - that's a real signal, just like these high-value financial transactions. People doing big transactions in cash are way more likely to be involved in crime - just like people in certain areas are way more likely to default or whatever.
But relative risk says nothing about the false positive rate. People do cash transactions all the time, legitimately. Poor people manage to pay their mortgage without defaulting all the time.
Also - the reason people are trapped in that area in the first place is social, and economic. The decision to value a particular signal - or trust a model that blindly ignores confounding factors - ultimately lies with you, the human creating the model. Never hide behind it.
The machine is just a machine, it just does math for you. It only calculates the model you tell it to. If you let your model be racist, that's on you. You designed it that way. You designed it to be racist.
The creation of the "black box" model is one of the scariest things we've done in the last couple decades. Because now, nobody can tell you why it happened, it just was a bunch of neurons that attributed various weights to factors in your profile. If you're lucky they'll even be able to tell you what some of the weighting factors might be (but usually not). And now it's nobody's fault, it's just ~the model~. It's a blatant abdication of any responsibility for the faults or consequences of your work.