Concerns about crime, undesirables, or culture are real, and taxes. They are neither inherently racial nor dog whistles, but are only so because of how American history shook out along racial lines. You are inverting cause and effect here.
Your comment is contradictory. On the one hand you say that crime, who is undesirable, and culture ate incidental and have nothing to do with race and then in this sentence you acknowledge that American history was largely determined by race, although you use very passive language as though that "shaking out" was some benign, incidental process instead of the cruel and brutal system it was.
But there is a difference between saying people are primarily motivated by race vs. primarily motivated by class but using race as a proxy vs. primarily motivated by class only. Given that these are distinct things, it's not helpful to insist on identifying everything as being secretly racist, not because it conveniently absolves anyone of responsibility but because it muddies the discussion and denies possible solutions.