Best read about this: http://smile.amazon.com/The-Rent-Too-Damn-High-ebook/dp/B007...
Employment is "sticky" in that personal mobility can't move as fast as capital mobility and that seems to have led to concentrations of wealth.
EDIT: Heck, industrial network effects of the sort you saw in Pittsburgh and Detroit and now see in Shenzhen can't be spread around either.
Think about it: if the homeowners in SF relent, approve massive housing projects, and decrease the average housing price of SF, then more people will now move into SF and its total real estate price (and hence the size of SF in their visualiztion) will increase.
Suppose you met the demand by doubling the supply. Suppose that takes you out of the exponential region, and housing costs drop fourfold. Overall, the market shrinks by half.
You might argue that when I buy a house for $500k, the seller gets $500k, so where's the problem? But the seller needs somewhere to live too- they are probably putting that $500k into another house, commonly an even more expensive house.
Typically, I'm only putting 20% of that $500k down on the property. Now, to be fair, that $100k is effectively mothballed unless I pull it out with an equity loan or somesuch, but that's a far cry from freezing $500k.