There are 130 million more people in the US today than there were 50 years. What changed? 130 million people is what changed.
E.g. look at Austin TX, it's housing prices have skyrocketed in the past 5-10 years. It's not particularly dense, but the vast majority of the existing housing stock is typical single family homes - there are very few open tracks of land where you can just build dense housing. So to increase density, you have to change existing neighborhoods to be more dense, and I'm not sure I've ever seen folks that I previously thought of as rational, compassionate individuals fight tooth and nail over even modest increases in density.
They might have overdone it, but still. They did exactly build entirely new cities.
What makes people think housing should be affordable in the "most wanted" locations? You can't physically fit unlimited people in a small area.
Although before WFH (and maybe even afterwards), the world's top billionaires were insisting engineers live within 20km of them so they can work at their companies. Industries like tech (and finance, and defence) heavily cluster, leading to certain cities/regions getting horrendously expensive.
You can certainly fit a lot more people in most major Western cities, without even much difference in lifestyle. Simply allow people to build what they want on their land, within reason. We've seen this in New Zealand, where recent law changes have allowed townhouses (terraced housing, with shared walls but otherwise freehold) and many many townhouses are being built everywhere.
Most houses are still standalone houses, no one is forcing you to develop your property, but if every 20th section is turned from a single house into 6 townhouses, a lot more people are able to fit into a smaller space while still having a comfortable, quiet home in a good location.
Our intertenancy walls have good soundproofing, enforced by regulations, so you don't really hear your neighbours. You can pay more for a standalone house, or less for a house with shared walls.