Everytime this issue comes up, this post deserves a link:

https://techcrunch.com/2014/04/14/sf-housing/

Probably the best written summary of the SF housing shortage.