The decreasing relevance of suburbia is an important story. Urban living is popular for very good reasons.
A huge amount of the growth of SF's (the city proper) tech scene seems to be driven by companies staffed primarily by very young (straight out of college, <30yrs) people. Similarly, much of the migration to the city seems to be driven by young folks (under 30/without family).
Back in the old days (when I was growing up on the peninsula), one of the big reasons people wanted to live in Palo Alto/Mountain View/Cupertino was that they were family-friendly towns with good schools and communities. Companies sprung up on the peninsula for many reasons, but people at or near child-rearing ages (~30+) had their priorities set.
I'm unconvinced that this rationale has changed in the intervening years. Gentrification and high demand have made living in the 'ideal' parts of the peninsula more difficult, but people still choose Palo Alto to start a family. Similarly, tech companies that tend to have slightly older workforces still prize their peninsula locations.
Perhaps I'm wrong, but I think studying the demographics of SF growth may tell us more. That said, I don't think that the suburbs[1] are being marginalized at all.
--
[1] While cities on the peninsula might be suburban in a way, they are quite different from suburbs in the rest of the country. It's a bit strange calling Mountain View a suburb when it is so different from a city like Needham (suburb of Boston) or the various suburbs of Seattle or Houston for example.
Working in a startup is also chaotic, unpredictable, and fast-paced, so it seems natural that people drawn to startups would also be drawn to cities.
This isn't new either: there were plenty of SF startups during the old dot-com boom as well. I'd also predict that if there is a further growth of tech startups we'll see startups spread to other areas now considered "less desirable" -- in dot-com days there were plenty of companies in Santa Clara, Milpitas, Downtown San Jose, Fremont, and even exurbs like Pleasanton. When the boom ended, CA-237 was locally called the "dot com graveyard".
As you mention there is plenty of cheap and large offices as you go south along the peninsula, but startups seem to consistently choose more expensive SOMA offices over these. Only startups that need a warehouse or other specialized facilities start their search in the south bay.
I think the reality is that its a combination of factors driving people to SF, from 'cool factor'/perception to quality of life issues (employee proximity) to 'cheaper-than-palo-alto' benchmarking.
By the way, if you're reading this thread and looking for office space, the south bay is a perfectly good place to look. Yes, it's not as glamorous, but it's both functional, frugal, and actually quite nice. The graveyard doesn't consist of dumpy factories, but pleasantly maintained offices with good facilities. You can even live nearby, as long as calm neighborhood is an acceptable condition for you.
I would say significant amount of startups would still prefer Mountain View to SOMA, but they'd probably prefer SOMA to Santa Clara or San Jose.
There are many positives you've described to places further South, but there are also issues: fewer other startups, less third places in walking distances, being rather far from any kind of public transportation. In terms of employees it's somewhat of a balancing act: there are older employees with family in South Bay, more infrastructure folks (ex-Google and VMW folks, as well as various infra startups), and there are also younger employees, front-end developers, designers, and people with finance/consulting who live in SF. It would be more far more difficult for SF set to commute to, e.g., an office park in Santa Clara than for folks in South Bay to commute to SOMA (fairly easy trip by either car or Caltrain).
That said, Peninsula is exactly what its name implies, and SF is at very end (with hills in between). There will be greater pressure for startups to go further South again, as well as East (if Berkeley/Oakland adopt more startup friendly laws), elsewhere in SF (already happening), and so on.
You should visit Sinagpore once.
This is often not true around SF. Up and down the peninsula, you'll find continuous cities with strong tech and other industries. Would we consider Redwood City a stereotypical suburb, for example? It becomes more murky, and many of the stereotypes fall apart.
That said, your perception is held by many people and is accurate for many parts of the US. As a result, people moving to the Bay Area choose where to live with a similar bias.
For me, though obviously it doesn't necessarily apply to anyone else, I miss singing in the car. :/
I have one, but I avoid driving like the plague.
Also:
Cities aggregate enough people that you won't be the only person like you.
As a sub-point to the above, if you're looking to meet any other people, for whatever reason, you'd want to be located in a city.
Things that are for sale, are generally for sale in cities. Want to buy something? Your best bet is a city.
Huh? In the US, median rural income was $40,135, compared to $51,522 in metropolitan counties. [1]
Things that are for sale, are generally for sale in cities. Want to buy something? Your best bet is a city.
Please tell me how long the drive from SF to the nearest Walmart is.
Cities aggregate enough people that you won't be the only person like you.
This is a very good point. Lots of interest-based communities form in cities. In suburbs, there would not be a sufficient "critical mass" for this kind of community to exist. It drives business too - there are, for example, restaurants in cities serving niche foods. They would never have a large enough customer based outside of a highly-concentrated urban area.
For comparison, my family lives in a "suburb" 20 minutes or more away from everything (and 25 miles / 40 minutes from the nearest Walmart). I wouldn't have thought much of driving 30 minutes to Walmart, except that SF doesn't exactly encourage, or even tolerate, car ownership.
Then again, I could walk 10-15 minutes to Macy's, Nordstrom, or Target.
Now I live in Saratoga where I've a 20 minute commute to Palo Alto -- and again, am in walking distance of grocery stores, coffee shops, and biking distance of downtown.
---
For comparison, my family lives in a "suburb" 20 minutes or more away from everything (and 25 miles / 40 minutes from the nearest Walmart). When I was in San Francisco, I wouldn't have thought much of driving 30 minutes to Walmart (because of my background), except that SF doesn't exactly encourage, or even tolerate, car ownership.
Then again, I could (when I was in San Francisco) walk 10-15 minutes to Macy's, Nordstrom, or Target.
---
I'm not sure why you bring up Mountain View, as I didn't mention it and in fact was not referring to it in any capacity. My family lives in a "beach community" with one store, of which I do not know the name. Besides the store, there's a church, a school of some primary-or-kindergarten level, an outpost of the public library, and a fire department. I have also heard the location described as a "suburb". Going anywhere at all is a hassle; there is nothing nearby. It is possible, given the right wind, to smell the fertilizer that the local farms apply to their fields.
I was just calling attention to your example (25 minutes to get anywhere) as it's a better example of an average suburb than Mountain View.
I worked for three years at Stanford. Once, after I'd been living in Menlo Park for about a year, I flew back into SFO from a business trip, and took the train away from all the skyscrapers and nightlife and excitement -- for me, this was the wrong direction.
I found an apartment the city, started taking the train down, and didn't look back.
Peninsula/South Bay isn't a cultural desert -- there are amazing ethnic restaurants, used book stores, concert venues, etc... -- but they generally aren't all within walking distance, nor is it likely that a great selection of such places will be equidistant from both you and your friends. On the other hand, if (like me!) you prefer to read/code/etc... most evenings and are more deliberate in regards to outings, it isn't really a problem.
Cities also allow you to save money by foregoing having a car, but it's less true for SF: public transport in Bay Area -- even SF proper -- is quite horrid compared to NYC, Berlin, etc...
Not everyone lives in SF by choice: while SOMA is easily accessible from Peninsula by multiple modes of travel, other areas (FiDi, mid-market) are fairly far from Caltrain, expensive as far as parking goes. BART is great when it's not having a strike, but the only locations in Peninsula with BART stops are Millbrae, Dalty City, and Colma.
What most people raving about "end of suburbia" are missing is that high density (SF, NYC) living is rather expensive and this isn't likely to change (SF is a Peninsula, Manhattan is an Island).
What's also missing is that cities vs. suburbia is a false dilemma: many neighbourhoods in Portland and Seattle, parts of Peninsula (e.g., Burlingame/San Mateo area), East Bay (e.g., Berkeley), and even Downtown Mountain View/Palo Alto are far cry from the usual suburban cliches (there is abudance of independently owned restaurants and stores, relative dearth of strip malls, rail transportation, etc...) while still having lots of green areas, being amenable to driving, good school systems all available to middle class (St. Francis Wood in SF might have all of this features, but the middle class is priced out of it.)
If you want to stay out late drinking and whatnot, public transportation isn't really an option because it doesn't run very late or very frequent. And if you're drinking, you shouldn't be driving so... you live in SF.
To answer the grandfather post, if bars/clubs aren't your thing, no, you're not missing anything.
That said Bay Area in general is hard to beat when it comes to food and drink.
I'm not completely familiar with it's characterization, but it seems like uniquely 'urban suburban' in a way, no?
If that's true, I wonder what that uniquely middle-of-the-road quality gives it, in the context of the shifting relevance.