It's almost as if excessive regulation and NIMBY-ism in San Francisco has created an artifical bubble of hyper inflated housing prices that causes the most harm to the people who are in most need of help. Damn.
Here's where I'm coming from - SF is small, old, and largely built. It's in the middle of a very large metropolitan area, but it includes many of the old parts that have had high population densities for a long time. Contrast that with the population densities of most American cities, and the general land area. With the exception of New York, SF much smaller and more densely populated than almost all of the regions used in these comparisons.
Now, when someone builds a large housing development far from the urban center of, say, Houston, that counts as development for Houston. When this happens in Walnut Creek, it goes down as not developing in SF.
So the way I'd answer this question - and I haven't yet - is to look at subregions in major met regions in the US that have had a population density of above 17,000/sq mi for more than 50 years, and try to measure how much development has occurred within those areas in the last 20 years, or what kind of opposition to new development happens in those regions now.
My refutable hypothesis is that SF will no longer show a dramatically different pattern of NIMBYism from those sub-regions outside San Francisco. That doesn't mean it won't show NIMBYism, or that it won't show more NIMBYism than other old, relatively dense regions - I just think that it may reveal that "population density over 17000/sq mi for more than several decades" correlates with general opposition to new construction regardless of whether it happens in SF.
Like I said, I haven't checked, it's just an idea. This would actually take some work to investigate.
The assumption being that I can buy the city block & it's in near other high rises.
It's less dense than LA. If you want to compare it to NYC, it's less dense than the NYC suburbs in NJ and Queens.
Rebuilding at higher densities is definitely practical, they just choose not to do so.
http://en.wikipedia.org/wiki/San_francisco http://en.wikipedia.org/wiki/Los_angeles
The Bay Area as a whole is less dense than the LA Metropolitan area.
It's true that the Bay Area as a whole needs to get denser, but SF proper needs to lead the way.
I guess another phrase this thesis is that SF's apparant NIMBYism is a misleading artifact of what you describe as a "historical accident of municipal lines."
This distortion, of course, can make SF look good as well as bad. You can pump SF up by talking about the ratio of <desirableculturalfactor>:Resident compared to Houston, or Chicago, or LA, conveniently ignoring the fact that in Chicago, the residential suburbs are counted, whereas in SF, they aren't. My guess is that this ratio in LA or Chicago's densest 48 square miles would look quite similar to SF. Or, alternatively, you can make SF look bad by excluding the sort of developments that favor the burbs (such as percentage of households with children - when a family moves from SF to Redwood City, SF "loses" a family - in many areas, this would just be considered a move within the region).
Good or bad, my thesis is that these differences are largely an artifact of how the borders are drawn, rather than innate cultural differences between SF and other metropolitan areas. I don't doubt that these cultural difference exist (and they may in fact be driven by the fact that the electorate in SF isn't diluted by the votes in the surrounding areas), but I think they may be greatly overstated.
Property owners sell off places they would otherwise have rented out or just leave them empty because it just isn't worth the risk. That is what constrains the supply. You'd be crazy to go into the rental business unless you had a lot of units to balance out the Inges of the world, particularly since once you finally evict someone like that you have to tear out the walls to get rid of the stench.