Why is SF (I assume you mean local government?) anti-height? Are there downsides to building upwards (environmental? taxation?) or is it just a refusal to allow the landscape to change?
The Bay Area's problem is that nobody wants the high-rises: putting them in the center or putting them near transit a bit outside the center would both work, but everyone, except to some extent Oakland, is anti-development, so they go nowhere. Heck, Palo Alto won't even allow mid-rise apartments near the Palo Alto Caltrain, so you actually have people "reverse commuting" from SF down to Stanford, because as crazy as living in SF is, trying to live in downtown Palo Alto is even crazier. A few livable, urbanized downtowns on the peninsula might siphon off some of the demand in SF, not solving the problem, but at least reducing it.
Other problems that seem like they exacerbate SF's problems: 1) poor intracity transit, e.g. lack of an east-west subway, concentrates demand in a handful of eastern districts; and 2) large areas of the city are not even mid-rise, but full of two-story houses (partly historical, partly anti-development, partly related to #1).
"f you look at San Francisco’s zoning map, you’ll see height and density restrictions everywhere. There are also citywide building caps — restrictions on the number of new buildings that can be started every year. And finally there are labyrinthine regulatory procedures. Here, for your amusement, is a flowchart put together by the San Francisco Planning Department that outlines the steps a developer needs to go through to obtain a building permit. Despite the department’s use of Comic Sans, this is not meant to be a joke."
Allowing building upwards would be seen as 'pandering' to 'special interests' by lots of locals --specially people who like to call themselves Xth-generation Franciscan.
SF is always about the short-term but made to look long-term so as to seem forward looking and progressive. The only thing SF is progressive about two things, sex, and some minority causes (both of which are good things but hardly encompassing of the term 'progressive'.)
SoMa probably ends up being the big-city, tall building part of SF in the future. It would take a big mind-set change for many other neighborhoods to build up anytime soon.
I presume that's mostly the reason.