Restaurants are a tough business, so towns with a high vacancy rate end up being unable to support very many restaurants. Forget the quality of the restaurant, lots of these places barely have any restaurants (or they're very seasonal)
Honestly, I wonder if one could do a "benefit local business tax" where they tax the non primary residents and then just subsidize the types of local businesses they want to keep open (it's hard to design good systems for this though!)
There's a third option you're ignoring: The rich people from elsewhere don't own those houses. They're priced more affordably and owned by locals who genuinely care about the area.
Everyone wants to live in the goldilocks situation of a desirable area with low prices, good services, low taxes, good commutes (low traffic and-or good transit), and plentiful jobs. I've lived in various areas from rural, suburban to Manhattan and.. you can usually have 2.5 out of 5 at best.
In touristy Montana cities the amenities are nice sure, but people aren't there for that. They're there to ski or hike or bike.
Regarding your last point - Manhattan can hit 3/5 if you can coordinate your apt and work locations.
Propery taxes are the same in NH whether someone lives there full time or not, so it doesn’t matter that they aren’t there and would be equally valuable if they did live full time. The traffic argument is the only one I can see being true, except that traffic in the seacoast is awful already anyway with tourists driving back and forth to Maine.
Personally, I think having permanent residents is far better for a town than a graveyard of empty soulless homes, but I’m just a regular human being and I’m sure its just a matter of opinion at that point.
Property taxes pay for municipal services like parks, schools, and police. The rest of the town's residents get better versions of those than they could afford otherwise. This isn't a very complex idea, regardless of whether you think the town would be better off with more people living within it full-time.
We're certainly a year-round town. But it's a reality of a lot of coastal communities as you get up the coast further that, if they're not commercial fishing ports, they do largely close down in the winter. If there weren't summer homes and a tourist industry, not many people would live there.
Unless you are suggesting replacing such a home with higher numbers of lower value homes. Obviously that would be a different story.