I think it's a NIMBY vs YIMBY thing.
Established players in any community have little desire for expansion.
Non-established players in any community have greater incentive to want to see expansion. For the most part, the non established are seeking to become established by filling the niches created by expansion.
Expansion will always have a cost, as well as a benefit. These two are un-evenly distributed and cause tension.
Overall, at least where I live, I know that removing vacation rentals would decrease tourism. As we have seen with COVID, less tourism in a tourist town and city budgets are restricted, services cut and business close. I like to see my little city vibrant and economically healthy, so I support YIMBY projects, even with their necessary negatives. That means I have to put up with unruly-annoying people.
We can't eat our pie and have it at the same time.