Vertical growth is the logical choice for SF, but it's complicated by two huge factors.
The first, as others have pointed out, is NIMBYism. SF has the most stringent zoning laws I've ever seen. Furthermore, lots of the deep-pocketed residents and real estate developers have been successful in fighting off verticalization.
The second major factor is instability. SF is famously overdue for "The Big One" and is known to be seismically unstable. But what a lot of people don't realize is precisely why it's so unstable: because a large portion of it was basically built on wet sand and landfill. If you look at topographic or seismic maps of SF, you'll see that much (most?) is built on areas of liquefaction. It can be unsafe to build highrises on shaky ground.
The first problem can be solved, albeit not easily. The second problem is a time bomb waiting to go off. Perhaps in our lifetimes, or perhaps in the next generation's.