Or they care too much, or society has driven them away, or they just don't like the population density, or it's cheaper, or they like the countryside better. See: the increasing number of people moving to the country and the popularity of Musk's Starlink project, indicating that remote people are invested in society.
I think that making the assumption that the people who live closer together have a better handle on how to run the country (society != country) needs a lot of justification. The large fraction of the population that lives less densely should not be disregarded as somehow 'less valuable'.
As I referenced above, the last time we tried to weight the value of different people it didn't turn out to be such a great idea. Skin color (the 3/5ths compromise, if you didn't get the reference) or geographic location, the value of a person or their vote shouldn't be tampered with.