I doubt there is any significant amount of housing that is just bought and left vacant. Why would you do that? It’s a stupid and expensive place to park money. You have to pay taxes, possibly condo/co-op/HOA fees, pay someone to do maintenance and keep out squatters, pay transaction costs both buying and selling, it can’t be sold quickly.
Why would you do all that instead of just buying treasuries or some other liquid instrument that actually earns money?
There may be some of this from foreign investors that don’t have easy access to financial markets, but again, some Russian oligarch buying a $50M penthouse in Manhattan is not the reason an average 3 bed 2 bath suburban home is $700k. They are not remotely in the same market.