Imagine a scenario that a black family moved into your neighborhood and started causing a bunch of problems. There are no other black people in the neighborhood.
It should take you a picosecond to figure out which one of these is obviously racist and which isn't:
"We need to remove the black people from our neighborhood"
"We need to remove the new family from our neighborhood"
People like "well he has a good point they are causing problems" so technically the set of "black people" and "people who are currently causing problems in our neighborhood" are the same set. This is irrelevant.