Nowadays though, it's common to own that second dwelling in a city or other built up area.
Just last week I was talking with a family member (mid-60s, about to retire). They are planning on buying a small apartment in the city their son and grandchildren live in. They are not selling their existing house in another city to do this. They plan to use it only while visiting and have explicitly stated that they see it as an investment. They are by no means 'rich'. 30 years ago this would have been unheard of for people in their position.
Reductio ad absurdum... if rich Chinese businesspeople buy every home in Vancouver (strictly to park money, not live or rent), nobody who works in Vancouver can live there and everybody has to commute from the suburbs.
Edit - at an indivual level, you're right, who cares. At a policy level, ensuring reasonable housing for the general public is a good thing.
Manhattan only exists due to Manhattan schist, exceptional bedrock due to old volcanic flow.
Many locations have poor subsoil for tall buildings, even with the best foundations there are other limits to building height.
There's a lot of potential space, sure, but not infinite space - it's finite on planet and bottlenecked by transport and terraforming off planet.
I do agree, there is really something weird about owning 'extra' houses that you may only use a few weeks a year.