Part of this is likely due to building restrictions. For example, Palo Alto has a 50 foot building limit [1]. If developers were allowed to build 20-story apartment buildings, it's very likely that housing prices would plummet.
[1] http://www.paloaltoonline.com/news/show_story.php?id=27136