I'd be willing to bet NIMBY antics are a large cause of this problem. SF metro area has the 4th highest rate of homeless people per capita, I have to imagine absurdly high rents is a part of it.
SF politicians don't want to admit this because of the donations/support they receive from NIMBY groups, but they are very clearly the cause of many problems in the bay area, whether its housing or slow transit expansion. It's especially ironic that such a "progressive" area would be so rife with elitism and gate keeping.