It's very hard to find the kind of space they need anywhere else. And while working at Google I get the sense that they would prefer more mixed use, higher-density development near their headquarters, but local city planning officials make that almost impossible.

I think you have a somewhat skewed sense of what the "rest of the country" is doing. There is an overall urbanization trend in the US but I'm not at all sure that companies are especially shifting their offices from office parks to downtown locations. Furthermore, the vast majority of US cities are not particularly amenable to a walking lifestyle. Many are sprawling and have fairly limited public transportation options.

Few things:

Google and Facebook have offices in New York (in the case of Facebook, there's a team available for most any engineering role in the NY office; I can't speak for Google, but I believe Spanner is out of the NY office), so for folks who really want a city lifestyle, this is always an option.

As for Bay Area, the big issue are schools: San Francisco schools operate on a lottery system (with the exception of a single "test school", Lowell), whereas South Bay has excellent public schools available to all those who live within their district boundaries (Cupertino School District, Los-Gatos/Saratoga School District, Los Altos SD, several schools in Fremont area, etc...). So for many, a South Bay location is actually a perk: commuting from South Bay to SF on Caltrain is a major hassle (that is assuming the offices of the company are located to close to Caltrain -- public transportation within SF proper is a nightmare unless you work and live close to BART station) and leaves no time to pick kids up from school/drop them off at school/curricular activities.

I should add that at least Google and Facebook run buses within suburban South Bay as well -- so the worst part of the commute (going North on 101 in the morning, South on 101 in the evening) can be avoided.

I should add there are also people living further in various bedroom communities in East Bay like Albany or Walnut Creek, or San Ramon (which also have good schools) and most of those folks either work for SF-based companies (keep in mind Google does have an SF office as well), or make use of company busses (Google runs busses to SF), or work from home some days a week.

This would also be a huge generalization, but one can also think of the Bay Area as the OSI stack: application/presentation in SF, mid-tier (databases at Oracle, distributed infrastructure at Google and FB) in Peninsula/Northern parts of South Bay, and physical layer (Apple, Cisco, Juniper, Intel, etc...) in Sunnyvale and further south. So for companies that do more systems-level development, there's both network effects (they're hiring people coming from other companies in the area), and the need to somewhat more experienced engineers (who are more likely to have families and to own vs. rent).

(background: lived in South Bay Area since I was 13, worked as an engineer in various South Bay Area software/Internet companies -- both in office parks as well as offices in more walkable downtown areas -- since late teens, including at Facebook in the Menlo Park campus).

I remember taking the light rail everyday in North Sunnyvale on Java Drive a few years ago, and wondered why companies like Google, Facebook, etc didn't move over there-there were so many vacant buildings, all next to VTA light rail. Apple's campus has mediocre bus service (VTA 55), Google's has practically none I can think of (piss poor location for transit) and FB is banking on the unicorn-like hope of revitalizing Redwood Junction and building a Caltrain spur.

I bet dollars to doughnuts most people who work at big name tech companies in Santa Clara County, and who actually live in Santa Clara County, drive to work.

Yea I've been thinking that was weird too.

I suspect its for economical/space reason and dealing with expensive real estate. Still though it feels opposite to the their "image".