This is a real problem. For a while I was running my own company and making more than I could spend - so I bought a house in a nice older neighborhood - walking distance to the beach, restaurants, and a pier, and very pleasant. But I soon discovered that most of my neighbors were never there - perhaps 10% of the houses were owned by wealthy people who would visit once or twice a year. So most residents of my town can't afford to live there - while the wealthy can afford to buy those properties and leave them empty.
Something is very wrong with the US economy and needs to be fixed. I don't want things to be "fair" only for the rich - I want things to be "fair" for our entire society.