Major data center power failure (again): Cloudflare Code Orange tested

I feel there’s a sweet spot where if you do it too quickly, it’s a bad sign. And if it takes years, risk just keeps going up and up until it becomes basically impossible to do smoothly.

> quite impressive to make major infrastructure and architecture changes in a few months

And have it work first time round.

There is not much impressive here. Their architecture seems to be relying on a couple of datacenters, and if somebody have turned on or not the right switches in the right places. This means you will continue to hear from regular outages and maybe nice postmortem blogs. I don't see a fundamental approach to reliability based on concepts around blast radius, expecting anything to fail anytime, and fundamental principles of bulkhead patterns.

Here is a free hint: By talking so much about where their data centers are located, on my view, they already failed item one on my check list. Principle number one of Physical Security is, you don't say where your Data Centers are, except of course to very restricted number of "need to know group".

As predictions have no value, unless they are made prior to events, I predict the next outage will be some common core component, with some on/off type of config, with some common core configuration, that "could not be foreseen". Then to the next blog...On to the next outage....

The blog post contain no details at all about how they achieved high availability. I'm disappointed.

Cloudflare is in EdgeConneX Portland, you can try poking around but I haven't seen a RCA of what happened. Details are usually only shared with direct customers because it is bad for the brand.

https://www.edgeconnex.com/wp-content/uploads/2018/10/ECX-22...

> there was no way to fail back power without an outage, then them going off and building their own equipment.

Anywhere I can read about that?

I would guess the devices in question are some sort of electric trip unit, along these lines:

https://www.eaton.com/content/dam/eaton/products/electrical-...

I don’t see anything in the manual about needing to power it down for reconfiguration. All the relevant buttons, set screws, etc should be accessible without removing any dead fronts, so they can be safely accessed while the system is live.

I guess if you have really fast fibre connections between them then the separation isn’t all that separate even with distance

At its core Kubernetes is mostly a HTTP API being used to sync state between nodes. I see no reason that part shouldn't work even across the world, albeit at the cost of slower syncing of state (eg. pods taking longer to be created after a scale-up).

That HTTP API is backed by etcd which uses Raft and that is where running over a large area is more likely to cause problems. One approach would be to keep the etcd instances in one region (and probably also co-locate the scheduler, controller manager there), while having far-flung worker nodes. This creates a risk of losing the control plane but in most cases services would keep running (but would be unable to react to any further issues until the control plane recovers).

You would also want to carefully design your workloads with topology constraints and region-specific services to avoid high application layer latencies though.

Overall it's a fun thought experiment. In practical terms I think cross-datacenter in a small geographic area would work fine but I probably wouldn't want to run a single worldwide cluster, both for the reasons above and for other scaling reasons.

It’s no different than running EKS in Multi-AZ configuration, except for the cross-az egress fees…

From what I understand of what I read, it was a single point of failure for one product instead of 15

It’s complicated and it’s not supposed to be. PDX is where the config plane for the edge network and services is stored. Some of this information is transported to the edge via QuickSilver [1] (e.g. auth tokens). This information is replicated to Europe and fail over is possible. The challenge with the previous outage was a combination of things (as it always is) whereby certain services had a dependency that relied on PDX as a single point of failure (if I recall correctly). That underpinned enough services that a good chunk of Cloudflare’s config plane went down. Additionally, the cutover to Europe didn’t go smoothly because it was instantaneous instead of gradual which resulted in traffic amplification as retries for previous requests & current requests were shuttled into the online data center resulting in a thundering herd.

What this blog post talks about it is how this time nothing went down (or at least cut over within minutes due to presumably automated systems noticing & doing it) with the exception of analytics data which does have a single dependency and that’s determined to be “ok” (i.e. it’s an acceptable failure mode for that product & that product only).

There are additional failure complications required because PDX (& Europe) is composed of several independent data centers which aren’t supposed to all fail simultaneously. What’s pretty clear from the implied “we’re not happy either” is that Cloudflare isn’t pleased with their vendor’s separately located data centers still having correlated failures.

[1] https://blog.cloudflare.com/introducing-quicksilver-configur...

Same. My employer was highly impacted by the November 2nd outage but this latest one didn't appear to have affected us at all.