Them owning real estate keeps the real estate off the market, thereby increasing the price for everyone else.
There's "rich people" buying houses, which are investment funds that buy property to rent - that is still availalbe on the market and isn't decreasing the supply of housing.
Rich people buying homes for personal use can't have an appreciable effect on pricing. You couldn't afford a billionaire's eighth home in whatever city even if it were available for sale anyway.