This is why you should ensure that the medium of exchange does not get stuck with people who don't need it. They block the economy for everyone else, preventing us from helping ourselves.
There's "rich people" buying houses, which are investment funds that buy property to rent - that is still availalbe on the market and isn't decreasing the supply of housing.
Rich people buying homes for personal use can't have an appreciable effect on pricing. You couldn't afford a billionaire's eighth home in whatever city even if it were available for sale anyway.