In many areas (including the Bay Area) the government won't allow private developers to build new housing (via zoning laws etc.)
Local governments are small and arguably inefficient, it is easy for special interests such as homeowners to influence them. But there is a long tradition of local government in US and they have real power. The solution here would be for higher-level (state) government to intervene, keeping in mind interests of all state residents and not only homeowners.
What we need is a tax on new houses that covers local infrastructure needs. We don't do this because developers have long fed on subsidies of there external costs by local governments.
Consider you add a new subway stop in an area and suddenly people want to replace single family dwellings near it with apartment buildings. The region paid for that subway expansion, but a few well connected developers stand to make a killing. When you get down to it Zoning is really an ideal way for local politicians to shift the benefits of public infrastructure into well connected private hands.
Subway extensions typically distribute costs over a wider population (the regional/state/federal government paying for large chunks of it). On some level this makes sense. A special assessment on the areas within X feet of a subway station would likely be prohibitively costly for property owners, and the entire region benefits from better transportation infrastructure with more benefits. On the other hand, the person who owns the property across the street from the new subway station is benefiting much more than someone who lives on the other side of town. Perhaps a combination of special assessments and general revenue funding is more appropriate for this scenario? You're certainly correct that the current system transfers a lot of public value into private hands. The Central Subway in SF is going to make some property owners in Chinatown very wealthy.