One big cluster: How CloudFlare launched 10 data centers in 30 days | Hacker News Reader

One big cluster: How CloudFlare launched 10 data centers in 30 days

arstechnica.com

67 pointsjgrahamc14 comments

This was an interesting article. Mostly because of how it took words that mean one thing to people and then tries to define them as something else.

When we set up our "data center" in Santa Clara (nearly 1600 machines in 100 cabinets) racking and stacking and booting didn't take long at all, what took forever was getting various network providers plumbed.

Co-location facilities range from extremely 'swank' like the Switch [1] facility in Las Vegas, to the extremely 'bare' like some mid-peninsula facilities I looked at. They can be run by a 'team' of folks with on site security, network staff, mechanical teams, and HVAC teams, or one contract security guy in a bullet proof glass booth noting who goes in and out.

A 'swank' co-location facility can sell you floor space at $800/kw per month. A 'bargain' co-location facility might be as little as $150/kw per month. A 'cabinet' is generally between 4 - 10kW so $3,200 - $8,000 per month per cabinet high end, and as little as $600 - $1500 per month per cabinet at the bargain side. If you negotiate your IP transit (networking) costs you pay between $3 and $6 per mbit or $3000 to $6000 for a gigabit line. For swank co-lo centers like the ones mentioned you're also going to pay $150/hr for "smart hands" services where someone on staff in the colo will go out and swap a drive for you or replace a bad Ethernet cable.

The ownership of these places ranges from real estate investment trusts (REITs) to businesses dedicated to internet services.

It is an amazingly diverse set of things for something as simple as chilled room with power and security and network connectivity. The first time I had to go up and do some work in Google's Oregon data center it became really clear to me why someone like Google would build their own data centers. The amount of costs you could shave off your computing budget were enormous. With that much profit just laying there on the table it becomes clear how folks can compete in that space.

With regards to the article, if I want to drop 3 or 4 cabinets of 'stuff' into 50 data centers around the world using the local datacenter IP connection its pretty straight forward to do. If you need a megawatt of space (I know megawatts aren't normally a measure of volume but in Data centers they are) and guaranteed redundant gigabit IP that can burst to 10Gbits, that is a bit harder to co-ordinate in 30 day bursts unless you do most of the work yourself.

[1] http://www.switchnap.com/

I think it would be more accurate to say "the install took less than 30 days." If you get all your circuits dropped in and then start counting days, it becomes much more realistic.

A bunch of machines in someone elses DC isn't launching a data center...

This article basically amounted to "we set up machines with some provisioning tools".

Looks like their trying to launch some new staff as well: https://www.cloudflare.com/join-our-team

Having done lots of this type of work I see it more as a triumph of paperwork and scheduling skills than techincal acumen.

It's nice to see in the age of the cloud that co-lo can be "cool" again.

Ironically used by a so-called "cloud" company.

well I wouldn't call a few racks in some one elses co-lo a data center.

From the article, it appears that they are the sole resident in the data center; "[Cloudflare] owned every router and every server in their racks".

The word "their" refers to Cloudflare's racks, not the datacenter's racks. Cloudflare might have two racks out of a thousand, but all the equipment in those two racks were supplied by Cloudflare, not rented/leased from the datacenter.

They mention that they use anycast rather than geoip-located DNS. (which itself would probably use anycast) I was under the impression that with anycast it was possible for routes to change mid-connection and you could end up with packets being routed to different datacenters as a result. Is there something they do to prevent this, or is it not a problem in practice?

That doesn't really happen in reality. Convergence is fast and as long as your datacenters are topologically disparate enough, you won't have random route selection changes outside of failures, in which case, the connection would have failed anyways. :-)

And if you're really tricky, you can sync state between POPs and keep a connection moving if your application is aware of the state change. :-)

Yup. David knows what he's talking about.

Routes change all the time. What you probably meant to say was destination or endpoint. If the destination of a route changes during a TCP session, there are two possible ways to handle it; shared state and retrying the connection.

In a shared state setup, the old and new endpoints are communicating the 'state' of connections, so as long as the state is in sync when the old endpoint goes down, the new endpoint can pick up the connection without delays. This is the basic way arrays of stateful firewalls work together to provide redundancy. When a shared state system fails to keep state in sync, the application can fall back to retrying the connection.

Retrying the connection incurs a larger delay than shared state systems, but it's handled transparently, so it's often not a big deal.

10 datacenters in 30 days really isn't anything terribly amazing. We're doing about double that in half the time with about half the staff ;)

Reply on news.ycombinator.com