Why do you assume only poor people are annoying neighbors? Rich people can afford bigger speakers, more booze, and to not have a job so they can stay up all night. I think wealth is positively correlated with the problems you describe.
Expensive areas have people who are busy working jobs to pay for their expensive housing and go to sleep at night.
Ergo, people who value peace and quiet gravitate towards expensive areas.
This is kind of like blaming environmentalists when NIMBYs use environmental reviews to prevent new construction. They don't give a shit about the environment, really, but they know how to misuse the system to get what they want.