Hacker News Reader

Comment by rkwasny | Hacker News Reader

Update from CEO https://twitter.com/eastdakota/status/1250479501726760960

How on earth can rented "remote hands" at a datacenter take down the ENTIRE Cloudflare management layer?

I don't know how big Cloudflare's management layer is, but I'm assuming it's relatively small (maybe dozens of racks at most), hosted in some sort of datacenter (Equinix, Digital Realty, Coresite, etc) that provides the remote hands.

Maybe some piece of core network equipment.

A small colo may have a single pair of routers, switches, or firewalls at its edge. If one had failed for some reason, and the remote hands removed the wrong one, it is possible you could knock the entire colo offline.

There's a bunch of other possible components: Storage platforms, power, maybe something like an HSM storing secrets, or even just a key database server.

Their failover to their backup facility may be impaired by the fact that well, their management plane is down. They probably rely on their own services. Avoiding chicken-and-egg issues can require careful ahead-of-time planning.

and for over 3 hours? What could they do accidentally that would cause an outage for this long?

The tweet says they're failing over to their backup facility. I would've expected that fail over to happen much faster.

Seems like they have two issues going on. First, the remote hands could take down the datacenter. Second, their fail over is taking this long to come online.

I also wonder how much Covid impacted the process, if at all.

I'm looking forward to the details after they get things back online.

Good luck CF engineers!

Three hours for a physical problem like this is very little time. Once you discover your equipment is not there, you still have to put it on its place, power up, reconnect al cabling, and check networking.

From this kind of mess to a fully functional infrastructure you need at least 12h-16h to function minimally. Probably takes 2-3 days to have that node working as before.

When I was on call, I always jocked with my colleages the worst incident you could have in a datatenter was someone swapping two cables.

Probably nuked the main DB.

"If things are painful you should do them more often," isn't supposed to be about callouses, but a lot of people take it that way. People with callouses do not experience the activity the way everybody else does. Sounds like a bit of that happened here.

I want my most of my mid-level people (and all of the ones bucking for a promotion) to be comfortable running 80-90% of routine maintenance operations and have pretty decent guesses on what should be done for the next 5-10%. I don't often get my way.

From your link:

>"During planned maintenance remote hands decommissioned some equipment that they shouldn’t have. We’re failing over to a backup facility and working to get the equipment back online."

Some questions:

When the gear was first powered off why wasn't it just powered back up again as soon as the on-call person's phone started blowing up? Why does this require a "failover" to another datacenter?

Why are remote hands decommissioning Cloudflare's gear in the first place? Isn't this supposed to be a security focused company? For context "remote hands" is the term for people that work for the colocation facility such as Eqiunix, Telehouse, etc. They are not employees of the tenant(Cloudflare.) Remote-hands are great resources for things like checking a cable or running new cross connects etc but certainly not decommissioning gear without some form of tenant supervision.

Why is this a single point of failure?

Just FYI -- Most colo facilities are prohibiting customer access during the COVID-19 lockdowns and have gone to 'smart hands' only for health and safety reasons.

Sure but that would seem reason enough to postpone the scheduled maintenance then no? I would think trusting "remote hands" to decommission production gear would carry some serious weight in the risk/reward analysis of keeping the maintenance window. At any rate I would think that those same remote hands should have been able to immediately powered that gear back up as soon as they were made aware of the error.

If the risk of the change was minimal, why would they not proceed?

How can you plan for things occurring out of your control? CF engineers are people as well. Things like this happen and there will be learnings to take out of it (like how to fail over faster)

What you mean "what if"? Clearly the risk was not minimal if it resulted in a 3 hour outage and affected customers. Someone at Cloudflare should have been able to identify that the services that were colocated at this facility had no redundancy prior to asking a third party to power down gear. Foregoing the maintenance until proper redundancy was in place was not out of their control.

Circulating a MOP(method of procedure) for data center maintenance among all stakeholder is pretty standard. The purpose of the MOPS is so that everyone can vet the plan(roll forward and roll back)and identify the risks.

It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen.

As Matthew said on Twitter, this isn't the kind of mistake that happens twice. But for those of us who do operational work, it's easy to see how this happens once. Bit rot in a cab, work orders from someone who didn't know about the legacy patch panel or assumed too much in their instructions and a catastrophe. From now on, photos for every smart hands will probably be a part of the prep-procedure.

>"It's confusing to me to read this -- you write like you know what you're talking about, but that can't be the case because if you knew what you were talking about you'd understand how the kind of mistake that happened can happen."

That's a pretty odd and flimsy argument that I can't know what I'm talking about if I hold an opinion that differs from your own. I have years of experience with this particular space. Had a MOP been circulated and the maintenance plan properly vetted then the need for visual confirmation because of "legacy patch panel" and "bit rot in cab" would have been identified. Further you never unplug a cable unless you first verify what the ends of the cable are connected to. As I mentioned this is "datacenter operations 101" stuff. This is not some new startup, this a publicly traded company who has been in the game for a decade now.

It's not the your opinion differs, is that's you are presenting a cognitive dissonance. If you have years of experience then you would understand exactly how this happens, and yet, you think it was avoidable, which is a misunderstanding of operational failures.

If this failure was so easy to avoid, it would have been avoided. It was the result of a combination of failures, as all failures in highly-available systems are. It was a failure in bitrot, in planning, in execution, in communication, in reluctance to failover due to a concern about failing back, etc.

That said, like Matthew said, this is the kind of failure that happens once.

Reply on news.ycombinator.com