I think you've just explained why stereotypes, prejudice, and racism aren't issues.

People notice patterns that are true often enough that they decide to avoid certain people. Saving time on customer support, reducing odds of being the victim of a crime, etc.

It's only an issue when people use too low of a threshold for numbers of experiences or percent of experiences.