There is a disproportionate amount of companies starting or moving into SF over even just the past 5 years. Before, if you're starting a tech company, it could be equally common to start it in the peninsula (maybe even more so), say Palo Alto or Mountain View and the like. Nowadays, the default choice is SF and that's majority of the cases by far.
I think collectively, the whole environment is pushing towards building SF into the New York of the west, except instead of banks and the financial industry filling up the town, it's tech companies. I'm not saying it's the agenda of anyone in specific (in government or otherwise) to build out SF that way, I'm saying it's what we're all collectively doing, whether intentional for some of us or simply riding the waves for others.
SF is a city that used to support many industries. Tech is obviously the one industry that is almost the top earner overall (or might even be the absolute top earner by industry). It seems like an obvious factor (and a big one) that's driving anyone not in tech away. In some cases, at the lowest end of the income spectrum, they might not have anywhere to be driven away to other than to the streets. Another anecdotal point is that recent popular article about how <$105k is considered low income in SF.
If this is indeed a factor, one of the things we collectively as part of the tech industry can do, is to intentionally spread ourselves away from SF, into other areas of the bay area. I get it that people want to live in the bay area for its climate and all the conveniences. There's no reason why there can't be tech companies HQ'ing in East Bay, for example. (and I'm not talking about just Oakland, but everywhere from San Leandro down to Fremont, or further east like Pleaston, Livermore, Walnut Creek, etc.)