Property taxes pay for municipal services like parks, schools, and police. The rest of the town's residents get better versions of those than they could afford otherwise. This isn't a very complex idea, regardless of whether you think the town would be better off with more people living within it full-time.