Hacker News Reader

Comment by TeMPOraL | Hacker News Reader

Yes, this point was also raised by Maciej Ceglowski (hn: idlewords) in the talk that keeps circulating in HN comments:

"And speaking of California, let me try another tack.

This festive map shows seismic hazard in Northern California, where pretty much all the large Internet companies are based, along with a zillion startups. The ones that aren't here have their headquarters in an even deadlier zone up in Cascadia.

Now of course, each company has four zillion datacenters, backed up across the world. But how much will that matter when there's a major quake, and Silicon Valley can't get to work for a month? All of these headquarters are going to be shut down for a long time when the Big One comes. You're going to notice it.

So even if you don't agree with my politics, maybe you'll agree with my geology. Let's not build a vast, distributed global network only to put everything in one place!"

View on HN

Pretty confident if there were a 1989-level earthquake, most startups and tech companies would just decamp to somewhere else for the recovery period. The easiest is when you already have an office somewhere else (we have London and soon Asia, although SF is still far larger); I'd probably pick Hawaii or Seattle personally.

It's easy enough to find an off-season vacation destination and move everyone there; essentially anywhere is cheaper than SF, so that's easy. Not every company has to move to the same place.

It'd suck to be a service-industry worker in SFBA after an earthquake, but tech employers can just move, and hire service-industry people to fix up their offices for a few months before returning (or not).

(this might not be necessary after a 7, but after something insane like a 9, would definitely happen)

I've noticed a lack of of human planning in most DR plans. In some industries it's called business continuance. Our company has focused mainly on the technical side of the business: backups, DR, offsite access.

Depending on the situation, earthquakes can bring fires and tsunamis. So what do you tell the network engineer that had their house burn to the ground? "Sorry for your lose; we're decamping for Las Vegas. Wheels up in 8 hours"?

If the big one does hit there is a lot more at stake than just your precious data.

Google has a program called DiRT (Disaster Recovery Testing)[0], when in a certain week of the year, some major exercises that simulating the loss of headquarters are carried out, including the (simulated) loss of some teams, some network connections in the headquarter and/or the physical access to the headquarters buildings. That makes a lot of sense when you place your headquarters at a place that can be demolished by an earthquake.

[0]: http://queue.acm.org/detail.cfm?id=2371516

That is actually an excellent idea. Loss of manpower is something that is overlooked in a lot of companies. I believe part of it is job security (make yourself invaluable). And part of it is HR optimisation. Why pay for 2 people to do the same job when 1 will do? The same reason you pay for 2 servers, 2 backup processes, etc.

I imagine for a large tech company, it would go something like this:

1. Immediately call extra hands in other locations (datacenters, customer service, etc); begin over-time scheduling to alleviate need for workers in disaster zone. Doesn't require any coordination of disaster zone, just remaining executives located outside.

2. Offer two special payments to workers in the impacted zone: 1. Disaster recovery assistance, to help them cushion emergency needs and get their family/belongings/etc recovered faster; 2. Relocation bonus, to create an incentive for quick relocation to temporary (or new) office.

3. Spend as much as you to have to get extra emergency services in to the area, and generally assist in government/regional recovery area. This will free up the last of your staff for temporary relocation.

I doubt that a large company (we're talking Google, Microsoft, Facebook, etc) would take more than a couple days to have other sites working overtime to cover the load, less than a week to have already transported some of their staff to a new site, and less than a month to have brought almost all of the office back online.

This timeline, of course, assumes that the problem wasn't that their office fell on all their employees, killing them or something of that nature.

Likely, we'd see a hiccup in the development cycle, but in terms of keeping infrastructure online, emergency bug fixes, etc, we'd probably see relatively little damage.

Disasters tend to have a knock on effect that can ripple for years. 2 that come to mind were the floods in Thailand and the 9-11 terrorist act. The floods temporarily knocked out the supply chain of HDD components, which had the knock on of system builders not having enough components. And eventually, technicians not having enough spare replacements for current systems.

1. In most businesses these other locations are supplementary to HQ. Typically there is no one at the datacenter and CS centers rely on guidance from HQ. They are not in the position to make decisions and drive the company.

2. Growing up and working in Florida you natural disasters are like clockwork. I can say for sure that 1 is covered by insurance and the gov't and 2 is going to be hard to pull off without looking like a complete dick. Also, you have other constraints. Airports may be closed and driving can be dangerous. Plus cars need fuel which is typically in short supply and expensive.

3. If you have a company of a few thousand I would say a few hundred are essential. Where are you going to find a place to house a few hundred people at once? Remember that other companies are thinking the same as well.

I also believe the most essential employees are that way for a reason. If they have the same preparedness in the company they also have the same mentality in their community. It would not be illogical to imagine the person in charge of DR for the company is also a member of the volunteer fire department.

Microsoft would be able to recover, then they find out the caterers don't have food to serve the cafeteria, employees can't come to work because there is no fuel, etc. These are the soft issues that most DR plans don't account for.

Used to work in Florida, and it's a pretty noticeable difference in DR/Continuity strategy to what I work with now out here (SF). For the first 24 hours after Hurricane Ivan it was completely focused on: Are our employees safe; do they have water/food; what medical aid is needed; how can we get blankets and cots into our office so our employees have a place to sleep tonight? All the roads are washed out, there was no real in/out of town for a week, no power to our main DC for 3. Having redundancy in Chicago and Virginia didn't really mean shit when we couldn't get key people to those data centers to run them full steam.

The end result was a high reliance on contract employees at the fail-overs, and a real appreciation for the human element.

Not a case of "if" but when.

In 1989, most of SF was back up and working by the next Monday. Of course, that was a different era without much telecommuting, and many people suffered tragic losses, but it wasn't bad enough to "decamp" anywhere. The oldtimers seem to complain more about the world series than the quake.

1989 was also only a 7.1. Go up almost an order of magnitude to a 7.9 or 8.0 and it might look a bit different.

That wasn't a particularly large earthquake.

Yea. In real life it doesn't quite work that way. You would be pulling families apart ("Honey, you stay at home in Palo Alto, fix the house, make sure everything is okay with the kids while I go to Hawaii for 2 months") isn't gonna fly.

But then again, I keep forgetting that the 'real' valley is just for early 20s-fresh-out-of-school-startup-single-guy-whizz-kids apparently.

Do you run periodic disaster recovery tests including simulating one of your location being offline (including employees being unreachable) for e.g. one day? This is a good test to check whether the know how is well balanced geographically.

All of the companies that don't already have away-from-the-Bay data centers will be begging for hardware. And if your name isn't Microsoft or Oracle or some other big ams, you'll be on the end of a product delivery pipeline measured in months. Hardware is going to be fought over.

If all your data is in your own building, you might be okay (assuming you can get at it, stuff it into a truck and move it out). Otherwise you're going to be waiting for a while.

No one really has servers In the Bay Area anymore, in my experience. A few startups with legitimately special security needs and competence (Stripe), but mostly people either use the cloud (not in the bay) or colo/managed hosting (also not in the bay, usually).

There might be a run on laptops, monitors, chairs, etc which overwhelm the Hawaii and Las Vegas retailers for a while, but basically California's high energy costs, low power circuit availability, high sales/use tax on servers, etc have solved this problem already.

There are plenty of servers in the Bay Area. There are several large colos in the South Bay (Equinix has several facilities in San Jose), San Francisco has a large colo at 200 Paul Ave, there's a bunch of stuff in the East Bay as well. Amazon has a nothern california region, not sure exactly where those servers are, etc.

Sure, I have equipment in several of those, but it isn't a sole hosting location for most large tech companies. Really big companies tend to be multi site. Smaller ones use cheaper clouds or managed hosting generally outside the area. A lot of the bay area gear is enterprise for local companies which would already be screwed by an earthquake, or network, etc to support people who live in the Bay Area.

SFBA is critical for personnel, not so much manufacturing or hosting.

There is about 3mm square feet of datacenter space here. There are very few large companies which have servers only here and not somewhere outside the area but which would be reasonably expected to continue operating if their servers were somehow unaffected.

Okay, but you started by saying "no one really has datacenters in the Bay Area anymore", and now you're talking about sole hosting, which is a completely different assertion.

True. (You may also note the time of my initial post; I was still awake at 0320, and this was shortly after the quake...); I wasn't being particularly precise.

However, SFBA isn't even the first location for most companies I see. They go either into the cheaper AWS cloud regions (us east or the Oregon), or managed hosting somewhere (rarely SFBA).

Bay Area companies which directly get colo early on are fairly rare: stripe, square, etc. I fully support it as a strategy, but it is statistically insignificant.

Aside from price, east coast single location also gives you a lot better latency to Europe. Asia is usually screwed anyway, but the extra 80ms makes a big difference.

It wouldn't be Hawai'i (as much as I'd love that). It'd almost certainly be LA or Vegas. Maybe Phoenix.

It need not be a 1989 level earthquake. That's the biggest worry here.

> But how much will that matter when there's a major quake, and Silicon Valley can't get to work for a month?

Well, putting everything in the Australian outback doesn't help that much, because there are good reasons why thousands of tech workers can't work there. This is essentially an argument against having any population centers in earthquake zones.

> This is essentially an argument against having any population centers in earthquake zones.

Which is an interesting point in and of itself.

Is it? What American city is not in an earthquake zone, or a tornado zone, or a hurricane zone, or a blizzard zone?

Indeed. In fact, it seems like the Pacific Northwest is a pretty solid choice, as is the Bay Area, although maybe the NY Times isn't properly weighting the relative risk of very bad earthquakes:

http://www.nytimes.com/interactive/2011/05/01/weekinreview/0...

While I don't remember the relative probabilities, the next big earthquake in the Pacific Northwest and Northern California region is expected to be much more intense: https://en.wikipedia.org/wiki/Megathrust_earthquake#The_Big_...

In the list of worst predictable natural disasters in the Continental US, it rates one above the expected earthquakes to the south, the result of two plates mostly sliding past each other: https://en.wikipedia.org/wiki/San_Andreas_fault

Worse, in part because we aren't at all prepared for it (aside from I hope more serious buildings) would be a return of major earthquakes in mid-West: https://en.wikipedia.org/wiki/New_Madrid_Seismic_Zone (something I pay attention to since that's on the other side of Missouri from me).

I forget one of them, and the worst, which few in the US would survive, and where being in Australia might not be a bad idea at all, would be a return of the Yellowstone Supervolcano: https://en.wikipedia.org/wiki/Yellowstone_Caldera

Supposedly that is why AWS located their infrastructure in Oregon, after starting in NoVa. They optimize for "lowest probability of disaster" before considering other options like cost (primarily power) and connectivity.

Phoenix. Sure they have drought, but not much other weather or natural catastrophes to speak of.

The problem with Phoenix is that it already has a larger population than its water supply can support.

Okay, that's one. Where do we put the other 300 million people?

http://www.reddit.com/r/dataisbeautiful/comments/29gw4s/worl...

Atlanta is right on the edge of all of these, but it's rare that they come this far with enough force to matter.

> You're going to notice it.

If the whole of the Americas was wiped off the Internet tomorrow then, for those of us in The Rest of The World (TM), life would go on. I think the only web property that would be annoying to not have would be Wikipedia. Second to that is the Google search engine but I take it as a given that they could cope just fine if America just mysteriously vanished one day.

I think that there are some that think that Silicon Valley is the centre of the universe, some indispensable magic place where all of the innovation happens. If there was The Big One (for the valley) we would soon find out exactly how important things like social networks and iphone apps really are. Life would go on. If anything, a big earthquake in Caliphonea is far preferable to a big eathquake in Japan. In Japan they make stuff the world needs.

  If the whole of the Americas was wiped off the Internet 
  tomorrow then, for those of us in The Rest of The World 
  (TM), life would go on.

????

What about the DNS roots, all the sites running on AWS, any site using Cloudflare or the JQuery/code.google.com CDN, etc

can't tell if sarcasm or not, so here you go.

>DNS roots

http://www.iana.org/domains/root/servers

>AWS

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-reg...

>Cloudflare

https://www.cloudflare.com/network-map

>code.google.com CDN

https://developers.google.com/speed/libraries/devguide#Intro... - no location list though, only "globally available"

We would certainly not cope "just fine". Sure we'd get over it, but there's plenty of stuff depending on American services. The whole .com/.net/.org registry is just a small example.

... seismic hazard in Northern California, where pretty much all the large Internet companies are based, along with a zillion startups. [..] All of these headquarters are going to be shut down for a long time when the Big One comes. You're going to notice it.

What we'll notice is not having filters for the pictures of our Starbucks cups. The demographic of teenage girls in malls will be devastated.

Silicon Valley is no longer the epicenter of technology; it's the epicenter of the fashion of technology.

Could you explain the phrase "fashion of technology" more?

It sounds like you're making an over-generalization based on the last few years of "social" while overlooking many of the largest tech companies in the world. Apple, Oracle, HP, Cisco, Google, Intel, the list goes on, are all headquartered here. http://www.siliconvalley.com/SV150/ci_25548370/

What made SV great was that it was, up to the 80's, the place where the foundations of technology were created. We've been building on those foundations since then and we've come to the point now where it's clear how frivolous it's become. We can't take it any further so we're adding pastel colors and glitter. It's become a series of fads. The 'great' tech companies of today's SV are no different than the 'great' lines coming out of Milan this season. Both will be gone when winter arrives having added nothing.

It was a poor place to make the comment; it wasn't on topic. My apologies. But when the comment was made that zillions of startups would be wiped out, and that we would notice, I wanted to clarify that we might notice but only in the most superficial way possible. Yes, it's a generalization -- there are always exceptions, and there are large tech companies there, but there are large companies in every major metropolitan area. If HP can't brand laptops for a year, I think we'll somehow manage to all survive.

I made the comment because I wanted to correct what I saw as an antiquated notion: that SV was different from Houston or Singapore. It used to be. It's not anymore.

Your comment is both incorrect and childish and this is actually an important and interesting topic, please don't participate on HN discussions with comments like this.

Comparing the startup scene in silicon valley to fashion is actually very apt. The loss of life and pain would be devastating, but lets not think that a few billion dollars worth of hardware would be anything other than a tax write-off and some middle managers pain to deal with for the next year.

If you want to look at devastating datacenter loss, if anything happened to Northern NJ the markets would go into a tailspin.

your comment is no better than op's. you could provide some examples why his opinion is wrong, because i believe quite a lot of people (myself included) think that most of companies that only exist in silicon valley are about extracting money from markets that are essentially driven by fashion.

You are right, I should have made a better comment or no comment at all.

I'm just tired of the obviously incorrect proposition that Bay Area is producing just frivolous photo sharing and social communication tools (which both has a huge value for humanity, but that's a different story) and I seriously find it an interesting thought experiment what would happen on a global scale if the Bay Area would be devastated by a big earthquake.

The obvious tech giants like Google, Apple, Oracle and Cisco are arguably supporting a huge number of businesses and organizations around the world. But a glance through a list of just publicly traded US SaaS companies reveals that about 40% are from the Bay Area. These offer SaaS-based solutions ranging from HR tools (Workday), IT service monitoring (Splunk), ITMS (ServiceNow), CRMs (SalesForce), ERPs (NetSuite), Insurance backends (Guidewire) to supply chain management (e2open), to just name a few.

If Bay Area suffered anything resembling 2008 Sichuan earthquake, it would halt a good part of the economy for a while.

Reply on news.ycombinator.com