Hacker News Reader

Comment by macros | Hacker News Reader

The perils of only having one datacenter :(

I was reading some item (on HN probably) about how they have a big one in Tampa, Florida, a small one in Amsterdam and are planning on building another big one in Virginia.

That said, the article mentioned a time when a failure in Amsterdam lead to a failure to fall over to Florida which took out the whole service for a good while.

I live in Tampa. We are the Lightning Capitol of North America {insert Wikipedia link here}. Right now is the rainy season here, and generous amounts of lightning. It's basically the monsoon season. So the power sucks here, and even if you have excellent generators, as Home Shopping Network does, the network connectivity is still bound by local limitations. Tampa is a terrible place to put a data center.

I think it's there for historical reasons, because Jimmy Wales used to live there, so his company was there, and now it's a significant project to move everything.

It's well worth doing. Putting a datacenter in Tampa is like putting a lighter factory in the middle of the fireworks district.

According to this, it looks like all the db servers are in the Tampa datacenter, with Amsterdam (and formerly Seoul) just having miscellaneous utility servers, backups, and a bunch of Squid proxies: http://wikitech.wikimedia.org/view/Server_roles

Looks like way too few servers in the Amsterdam cluster to fail over to anyway, even if they were set up for it: the Tampa datacenter has over 400 servers, versus Amsterdam's 44.

If you are building a startup, I seriously recommend starting out in two data centers, even if you just have a vps or two in one or both of them. While operating at crazy scale may or may not be a Maserati problem, it will pay dividends early for your code quality, and site stability.

Reply on news.ycombinator.com