I don't think they're prohibiting dense housing per se, just not approving/building enough of it. There have been several 50-story condos opened in the past few years, and several more currently under construction, but they're not enough to keep pace with growth in demand. It didn't help that there was a ~2-year lull in construction when a bunch of developers got into financial trouble during the financial crisis and had to put projects on hold. It's not as if SF isn't dense; it's the 2nd-densest city in the country, and way-way-way ahead of local peers San Diego and Los Angeles.

I agree on public transit, but it's not clear what to do about it. An east-west Geary subway line has been proposed since the 1930s, but it's really expensive to build in an earthquake zone, and the city / taxpayers just haven't been willing to pay what it'd cost. A cheaper alternative that leverages the already-built infrastructure might be to convince people to live in under-developed areas that do have good transit already, such as Oakland, which is actually closer to downtown SF by transit-minutes than much of SF proper is.

One thing that would help on the latter issue is if more of the suburban cities on transit lines also dropped their anti-development attitudes. For example, there is enough demand that there should be condos/apartments near the Palo Alto Caltrain station. But Palo Alto won't allow it, because they have some idea of an upper-class suburb full of single-family homes that they'd like to protect.