Why is it popular? Serious question.
Working in a startup is also chaotic, unpredictable, and fast-paced, so it seems natural that people drawn to startups would also be drawn to cities.
This isn't new either: there were plenty of SF startups during the old dot-com boom as well. I'd also predict that if there is a further growth of tech startups we'll see startups spread to other areas now considered "less desirable" -- in dot-com days there were plenty of companies in Santa Clara, Milpitas, Downtown San Jose, Fremont, and even exurbs like Pleasanton. When the boom ended, CA-237 was locally called the "dot com graveyard".
As you mention there is plenty of cheap and large offices as you go south along the peninsula, but startups seem to consistently choose more expensive SOMA offices over these. Only startups that need a warehouse or other specialized facilities start their search in the south bay.
I think the reality is that its a combination of factors driving people to SF, from 'cool factor'/perception to quality of life issues (employee proximity) to 'cheaper-than-palo-alto' benchmarking.
By the way, if you're reading this thread and looking for office space, the south bay is a perfectly good place to look. Yes, it's not as glamorous, but it's both functional, frugal, and actually quite nice. The graveyard doesn't consist of dumpy factories, but pleasantly maintained offices with good facilities. You can even live nearby, as long as calm neighborhood is an acceptable condition for you.
I would say significant amount of startups would still prefer Mountain View to SOMA, but they'd probably prefer SOMA to Santa Clara or San Jose.
There are many positives you've described to places further South, but there are also issues: fewer other startups, less third places in walking distances, being rather far from any kind of public transportation. In terms of employees it's somewhat of a balancing act: there are older employees with family in South Bay, more infrastructure folks (ex-Google and VMW folks, as well as various infra startups), and there are also younger employees, front-end developers, designers, and people with finance/consulting who live in SF. It would be more far more difficult for SF set to commute to, e.g., an office park in Santa Clara than for folks in South Bay to commute to SOMA (fairly easy trip by either car or Caltrain).
That said, Peninsula is exactly what its name implies, and SF is at very end (with hills in between). There will be greater pressure for startups to go further South again, as well as East (if Berkeley/Oakland adopt more startup friendly laws), elsewhere in SF (already happening), and so on.
This is often not true around SF. Up and down the peninsula, you'll find continuous cities with strong tech and other industries. Would we consider Redwood City a stereotypical suburb, for example? It becomes more murky, and many of the stereotypes fall apart.
That said, your perception is held by many people and is accurate for many parts of the US. As a result, people moving to the Bay Area choose where to live with a similar bias.
You should visit Sinagpore once.
I have one, but I avoid driving like the plague.
For me, though obviously it doesn't necessarily apply to anyone else, I miss singing in the car. :/
Also:
Cities aggregate enough people that you won't be the only person like you.
As a sub-point to the above, if you're looking to meet any other people, for whatever reason, you'd want to be located in a city.
Things that are for sale, are generally for sale in cities. Want to buy something? Your best bet is a city.
Huh? In the US, median rural income was $40,135, compared to $51,522 in metropolitan counties. [1]
Things that are for sale, are generally for sale in cities. Want to buy something? Your best bet is a city.
Please tell me how long the drive from SF to the nearest Walmart is.
Cities aggregate enough people that you won't be the only person like you.
This is a very good point. Lots of interest-based communities form in cities. In suburbs, there would not be a sufficient "critical mass" for this kind of community to exist. It drives business too - there are, for example, restaurants in cities serving niche foods. They would never have a large enough customer based outside of a highly-concentrated urban area.
For comparison, my family lives in a "suburb" 20 minutes or more away from everything (and 25 miles / 40 minutes from the nearest Walmart). I wouldn't have thought much of driving 30 minutes to Walmart, except that SF doesn't exactly encourage, or even tolerate, car ownership.
Then again, I could walk 10-15 minutes to Macy's, Nordstrom, or Target.
Now I live in Saratoga where I've a 20 minute commute to Palo Alto -- and again, am in walking distance of grocery stores, coffee shops, and biking distance of downtown.
---
For comparison, my family lives in a "suburb" 20 minutes or more away from everything (and 25 miles / 40 minutes from the nearest Walmart). When I was in San Francisco, I wouldn't have thought much of driving 30 minutes to Walmart (because of my background), except that SF doesn't exactly encourage, or even tolerate, car ownership.
Then again, I could (when I was in San Francisco) walk 10-15 minutes to Macy's, Nordstrom, or Target.
---
I'm not sure why you bring up Mountain View, as I didn't mention it and in fact was not referring to it in any capacity. My family lives in a "beach community" with one store, of which I do not know the name. Besides the store, there's a church, a school of some primary-or-kindergarten level, an outpost of the public library, and a fire department. I have also heard the location described as a "suburb". Going anywhere at all is a hassle; there is nothing nearby. It is possible, given the right wind, to smell the fertilizer that the local farms apply to their fields.
I was just calling attention to your example (25 minutes to get anywhere) as it's a better example of an average suburb than Mountain View.