I agree with you in general. My issue with this line of thinking for policy decisions, is that I personally believe relying solely on objective probability theory to associate people with _______, without looking at the larger picture, and then building policies from that, could exacerbate the reason the association is being payed attention to in the first place.
Let's say theoretically, that people who wear red shirts are objectively more likely to commit a crime. Policies are made such that people with red shirts are targeted for extra policing. People with red shirts who are not criminals feel discriminated and overly watched, and because of that, generate a resentment against society and (I admit this is a jump), ultimately end up with a higher disposition towards committing crimes. And then it just goes on ad infinitum.
So in some situations, I would argue that society benefits from ignoring these objective probabilities entirely, if there's evidence that taking action based on them could cause downstream negative consequences.