Decades of redlining have conditioned people to associate minorities moving into a neighborhood with the decline of property values. Black and brown people appearing on sidewalks is a signal to sell and move to a nicer neighborhood. If you don’t, your other white neighbors will and you’ll be the sucker left with a house you can’t sell as its value plummets. It’s called “white flight”.
That’s what this this issue is really about. Poor people = minorities. NIMBYs want to keep the minorities out of their neighborhood because they’ll trigger white flight.