The headline about all American cities seems like it doesn't match the rest of the article, which is really just about San Francisco.
More conservative metros will at least build outward to deal with housing supply problems. Yeah, that's not really good for the environment, but it does help with prices. But not liberal cities: they don't want to build out OR up.
Instead, you get a plethora of excuses as to how they really shouldn't be expected to host any more human beings: why can't another city in the metro build more? Why can't those people move to another state? Why can't those companies hire more in another state? What about my god-given right to free, abundant car parking? Shouldn't we build more infrastructure before more housing (also let's not build that infrastructure please)? Why can't we freeze my neighborhood in amber? Etc.
SF, NYC, Seattle, and Boston don't have more land to open up for development by virtue of their geography. Of course places like Dallas, Oklahoma city, etc. are able to build outward -- they aren't surrounded on multiple sides by large bodies of water. To build outward in SF, NYC, Seattle, and Boston they would have to recreate Waterworld.
https://ss2.climatecentral.org/#12/37.7341/-122.4540?show=sa...
(The East Bay and Palo Alo are screwed, but that's not the subject here.)
Also, expensive high rises benefit the working poor by giving rich yuppies a place to live where they don't bid up the same housing that working-class people have been living in for decades. If you put up luxury condo highrises in SOMA where tech workers could walk to their offices, for example, then maybe they wouldn't be bidding up the price of 1BRs in the Mission to $4k/month. Landlords are generally unwilling to let housing go vacant, because even a 2-month vacancy can destroy much of their profit for the year. So one way or another, if you build enough housing the price will adjust itself to the point where everybody gets housing.
It has to happen eventually. As long as there is economic activity, there will be jobs hiring the best talent from around the world, spurs more economic activity, etc. That's just not going to change, demand will continue, so cities really need to get rid of zoning for height so supply could be built near the job centers. The alternative is sprawl, which means people need to be transported, consuming resources, etc. Infrastructure is also all the more difficult to build out over sprawl. In particular for California, adding to the sprawl could also mean building into wildfire prone areas.
California really does have a major problem now with wildfires and encroachment. I was amazed how fast the Camp Fire moved (an acre a second burnt at one point). There's just no way to really escape something like that. This is why I think building down as well as up is a good idea for many cities, Californian cities may have dinner different issues due to their tectonic activity but it should at least be explored.
I agree with your point regarding density in general. I always find it silly when people say America doesn't have space for more people. We have entire states with less population than some cities. I think improving infrastructure between the "flyover" states and coastal states might be one improvement that's overdue, but I don't know what the best solution would be off the top of my head.
Why?
Property is an investment (and that investment creates wealth for CA via Prop 13). It's rational to oppose anything which would decrease the value of your investment. For many home owners, their home is one of their biggest wealth sinks.
That said, we should have never treated housing as an investment without some sort of control.
In the grand scheme of things, screwing over one person (or one family) could be considered to be OK, given the benefit to the city as a whole. But, having gone through this with my parents in a stupidly small town (40k), I can understand why anybody doesn't want it in their back yard.
And ultimately, they're the tax payers and voters for that city. What they want is valued more than what outsiders want for their city.
Yup, this is a big part of the problem. Potential residents can't vote, so you end up with a tragedy of the commons kind of situation: everyone agrees that someone ought to make more housing, but they all want someone else to handle it, and leave them alone.
It's not entirely unlike homeless shelters. People agree that they should exist, just never in their own neighborhood. Because, y'know, eww.
The solution is to have policy dictated at a higher level of authority that can avoid regulatory capture by hyperlocal interests. A zoning board that operated at the regional level would have fewer problems than cities or neighborhoods making these decisions.
I thought the article made a good point, that tribalism and xenophobia is common to most humans and that wealthy & powerful people have the means to enforce their exclusivity on less wealthy & powerful people, and that different parties just manifest it through different policies operating at different levels. Then they dress it up in outrage politics because that generates pageviews.