To some extent this already happening. Many people moved out of the city during the pandemic. The people in desirable rural areas aren't necessarily happy about it. They are often upset about all the outsiders with more money moving in.
They probably wouldn't be very happy with their property taxes going up either? When there's increased demand for land in a certain town from newcomers, there should be higher land taxes for everyone according to Georgist theory, to keep the price of land zero.