Hacker News Reader

Comment by pinko | Hacker News Reader

I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

View on HN

By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just trying to edit a character it's impossible to accidentally hit that delete key. An analogous system for Amazon that would have prevented this outage: delete 10 nodes, ok. Delete 100 nodes, box pops up saying 'To delete this many nodes you must type the following in to a message box: "I want to take down a dangerously large amount of nodes."'

It occurs to me that having to type the English version of the numbers would probably work in this scenario.

   s3-shutdown -c "one hundred fifty"

But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.

Maybe more like --force-with-lease

I was in fact thinking of the --force problem which is why I went another way.

Facebook uses a `--clowntown` flag to let people do things without normal checks. [0]

[0] https://www.quora.com/When-people-who-work-at-Facebook-say-c...

I also rather like their intentionally verbose and scary parameter for generating arbitrary HTML in React: https://facebook.github.io/react/docs/dom-elements.html#dang...

I'm remembering tools I've worked with where the cheeky dev required things like type the sentence "I know what I am doing and wish to proceed." in order to perform unsafe operations.

I've always wondered why ops hasn't adopted some of the best practices that have been around for years to avoid fat finger errrors. Like why don't we have systems where to do something dangerous requires two separate people run the command, or there's an approval step, or whatever.

The most common explanation: "cute" interactions make it harder to script the command-line tools because you have to account for the extra layer of indirection or write a bit of screen-scrape logic to get the command prompt input.

I've always found that explanation a little threadbare.

I think the hoops you have to jump through should scale with severity. Then if you find yourself writing scripts that screen-scrape interactive output to get around some safeguards, you're doing something very wrong.

Which is why there needs to be an override switch, but it needs to be very very explicit that you are going past the safeguards. And only a limited number of people who can use that override.

then you need to "sudo take them all offline now!"

"We can't because we put a safeguard in for that last week!"

"Just pull the power!"

"We've got redundant power - you can't!"

"So take the emergency chainsaw, and go cut the Internet cables."

We have physically redundant cables and microwave backup.

There are people who can still do fleet wide root access commands and I don't think that type of thing will ever be removed (just very restricted) for this exact type of situation.

Instead of rate limiting you'd be better off making it a two-key operation once you hit X threshold. Jr admin can delete 10, but the boss needs to confirm a deletion of 100.

You can always build the safeguard to require approval from a peer (or superior) to an action that is normally considered dangerous, at least for overriding the throttle.

Anything that requires human approval for routine operations quickly devolves into bureaucracy that adds a lot of manual steps without any real safety. Now, you may argue definition of "routine operation", but the thing in the article didn't sound like they were doing anything crazy.

Reply on news.ycombinator.com