I agree with your comment. Granted I only visit SF now and then, but it seems very strange that things don't just go vertical. That's the obvious solution, isn't it? Build huge retail+office+residential towers like those proposed over half a century (or more?) ago.
At first evaluation, it seems that people who own space must be extremely opposed to this and that works its way into local politics. I can't imagine people are voluntarily height-capping these little buildings I see going up.