Or are there other political/zoning reasons that make SF more attractive? I believe SF is giving some rich companies discounts to keep them around and reap (I assume) local taxes?
It didn't emerge in SF. Quite the contrary, virtually all tech startups for most of the Bay Area's history were no farther north than Menlo Park, and often much farther south (Netflix was founded in Scotts Valley). That's why YC was headquartered in Mountain View until recently.
SF only became a startup hub within the last decade, IMHO in response to the desire of Millenials to live in urban environments.
It it only during the present decade that startup activity has appeared in force in SF, away from Silicon Valley. Due largely (I believe) to the tax breaks SF created in 2011 to encourage that.
I knew a number of people who moved to the SV area, got FAANG type jobs way down in the Valley, and yet still chose to live in SF enduring hellish commutes. I never could quite understand why.
SJ is part of the Bay Area and absolutely has skyrocketing costs; it usually tops the lists of cities with the fastest growing housing prices.
When I got a job in Redwood City (in between the two), I thought about moving there from SF, but found the costs to be about the same to live there versus SF.
So I stayed in SF; it's definitely the anchor of culture of the Bay Area. For everything outside of work it's a much better place to be (for me, a single queer person, especially). I since got a job in SF and have no intention to move or work south again.
In fact, the general trend has been for companies to move towards SF away from the peninsula, but mostly because it's where workers want to be. It's much more easier to get by without a car in the city than in the peninsula, it has much better clubs/nightlife, etc. For a young person, the city (meaning SF) is often much more appealing, though of course different people have different preferences.
Of course, late at night I once did SF-SJ in 45 minutes. Having tried every solution at all times of day (e.g., biking to the caltrain, carpooling, etc) the "big bike" always wins; truly off peak the car is just as fast, in theory (and, hey, safer and more comfortable... but less fun).
But, let's be honest all the solutions are stressful and sucky. You can trade time for stress to a certain extent, but you're spending a lot of both no matter what.
But far from theoretical.
I'd not recommend it.
NYC is way more dense than the Bay Area. It’s not exactly a cheap place to live.
I grew up in the South Bay. I can answer that question.
The South Bay is eye bleedingly BORING by design. It's an absolutely terrible place for a teenager or young adult. Most of the fun stuff in the city is illegal in the south bay.
Most of the young people I know who work in tech want to live in or near San Francisco -- it has nightlife, restaurants, cultural events, etc. This is especially the case for people who are not from the Bay Area originally. When they start looking into buying property or having children a lot of them leave San Francisco and move closer to the suburbs.
(This comparison is unfair to both Queens and the Bronx, mind you.)
Because VCs are mostly investing in software startups with twentysomething who are stupid enough to believe the hype. Those who believe hype want to live in the "cool" area, and that's San Francisco or Berkeley--not San Jose.
Those people building real businesses don't care, and probably aren't even located in the valley anymore.
Population of San Jose: 1.0 million.
Population of San Francisco: 880k.