There's a correlation between crime / neighborhood blight and low income residents.
That correlation may be because of historic and current structural racism, lack of access to mental health services and a social safety net, or a myriad of other socio-economic issues, but it doesn't change the fact that it exists.
Insinuating that anyone who brings up those issues is racist isn't helpful.
It would be more productive to say "Yes, I understand why neighborhoods might be resistant to denser, low-income tolerant development, and here are some ways we could incorporate that fact into a productive plan..." E.g. building more mixed-income developments.