edit: an example: https://www.google.com/maps/@41.8970249,-87.6709707,3a,75y,9...
IMO, in order to accomodate more people, and deal with the insane rents, the bay area needs to start growing vertically. Every other urban area I've been to has towers, apartment complexes. Here everything is flat. Most apartment buildings are only two floors high, rarely more than three.
I won't get into the politics of all of this, but politics are the real problem. Many of the suburbanites around here want their suburbs to remain suburbs. The law of offer and demand should mean that apartment towers are getting built and rents go down as competition increase, but the regulations in place prevent this. Many swanky new condos are being built, but these are still flat buildings, not making efficient use of the available space.
Obviously, San Francisco is hemmed in by geography, but even if you compare the metro areas, Chicago is better suited to hosting large-scale companies than SFBA.