Hacker News Reader

Comment by savanaly | Hacker News Reader

Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something.

It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level."

At the organizational level, I guess it wasn't rated as all that likely that someone would try to remove capacity that would take a subsystem below its minimum. Building in a safeguard now makes sense as this new data point probably indicates that the likelihood of accidental deletion is higher than they had estimated.

View on HN

I'm drawing conclusions based on my time at AWS, but I believe this is due to the service discovery mechanism most of AWS uses. It's a gossip protocol, with a daemon running on each service host. There are very valid reasons you would be manipulating the set of hosts currently being gossiped- to remove a host for maintenance for example. In this use case I think they wanted to take out an entire service, so ensuring a majority is still alive isn't necessarily a solution.

There are two distinct failures here, the tool being too liberal and then the failure mode not being well understood for this index subsystem.

If I remember correctly, all of our processes were organized through a weekly change management process that went through reviews of the exact commands to be run. Being oncall was a bit more liberal, you would typically execute commands on production as needed based on your experience and with others over your shoulder if you had any doubt. Interacting with the gossip protocol was a pretty common thing to do when you were triaging issues.

Unrelated, I was briefly a SME on the EBS billing system and probably interacted with the poor guy whom executed this command.

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

then you need to "sudo take them all offline now!"

"We can't because we put a safeguard in for that last week!"

"Just pull the power!"

"We've got redundant power - you can't!"

"So take the emergency chainsaw, and go cut the Internet cables."

We have physically redundant cables and microwave backup.

Which is why there needs to be an override switch, but it needs to be very very explicit that you are going past the safeguards. And only a limited number of people who can use that override.

By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just trying to edit a character it's impossible to accidentally hit that delete key. An analogous system for Amazon that would have prevented this outage: delete 10 nodes, ok. Delete 100 nodes, box pops up saying 'To delete this many nodes you must type the following in to a message box: "I want to take down a dangerously large amount of nodes."'

It occurs to me that having to type the English version of the numbers would probably work in this scenario.

   s3-shutdown -c "one hundred fifty"

But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.

Maybe more like --force-with-lease

I was in fact thinking of the --force problem which is why I went another way.

Facebook uses a `--clowntown` flag to let people do things without normal checks. [0]

[0] https://www.quora.com/When-people-who-work-at-Facebook-say-c...

I also rather like their intentionally verbose and scary parameter for generating arbitrary HTML in React: https://facebook.github.io/react/docs/dom-elements.html#dang...

I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations.

I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the command, or there's an approval step, or whatever.

The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input.

I've always found that explanation a little threadbare.

I think the hoops you have to jump through should scale with severity. Then if you find yourself writing scripts that screen-scrape interactive output to get around some safeguards, you're doing something very wrong.

Instead of rate limiting you'd be better off making it a two-key operation once you hit X threshold. Jr admin can delete 10, but the boss needs to confirm a deletion of 100.

You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.

Anything that requires human approval for routine operations quickly devolves into bureaucracy that adds a lot of manual steps without any real safety. Now, you may argue definition of "routine operation", but the thing in the article didn't sound like they were doing anything crazy.

There are people who can still do fleet wide root access commands and I don't think that type of thing will ever be removed (just very restricted) for this exact type of situation.

But just imagine the "ooooh shit" moment of this person.

Something similar happened to us, when an Engineer deleted part of our production database with a single command. Fortunately, we could reconstruct it from backups and replication logs.

I really feel for this person. I would feel extremely shitty.

To me, the more important part would've been the solutions they'll come up with to avoid something like this happening again. Is it going to just be "add a line to the playbook asking the engineer to double check the command" or will they make big changes across the system to prevent things like this happening.

I'm interested in something like that, too! I've got a few destructive Ansible commands that I check a few dozen times before running. But There's always a chance that I'm tired/distracted/whatever and run something silly anyway. Typically I put in things like config test checks and prompts, but damn it's scary how much power I have with this Ansible setup. I definitely don't want to be in this AWS position.

Reply on news.ycombinator.com