As somebody who has spent a lot of money to live in a high-value area, not because of an investment in the property, but to avoid having my family exposed to inappropriate behaviors, yes let's keep out of the poors.
Affordable housing doesn't mean it needs to be in the same gated community as expensive housing, or in the same condo building. Economic segregation is not a net societal ill, in fact it's the primary driver for many people to increase their fortunes (myself included) so they can escape the crime and blight that comes from lower class people.
My reply might seem incredibly classist, because it is. I don't want my children exposed to public drug usage, petty crime, violence, and other ills that perpetuate any place where the people reside who lack the mental stability, intelligence, or self-control to hold down a professional job. I don't want them seeing this behavior, and I don't want them being pulled into dangerous situations because of the people who are in the community around them. I pay a lot to ensure that doesn't happen, including the choice of school and where we live.
I say all this as someone who grew up in a poor family in a bad area and spent my entire life escaping and staying out so I can raise a family in a better circumstance.