This problem exists regardless of who does the buying. Where I live the locals got into the market first. Still, it's a zero-sum game, every short-term rental is a house a family cannot live in and probably cannot afford to buy.
On my street 30% of the houses are short-term rentals. Some rent out for $10k/week just 8 weeks a year and are closed up the rest of the time. My daughter is currently the only kid on the street, which has over 100 houses.
Not only are all the houses now priced as income-producing investments, they are killing the community that used to exist here.