It remains amazing to me that even with all the layers of automation, the root cause of most serious deployment problems remain some variant of a fat fingered user.
It remains amazing to me that even with all the layers of automation, the root cause of most serious deployment problems remain some variant of a fat fingered user.
The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".
Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit).
This usually then caused the customer to recant on how he/she never meant for anybody to get fired...
It was a funny solution and we got away with it for years, for one because it was pretty rare to get customers that mad to begin with and for another because Frans never wrote any blog posts about it ;)
But I was always waiting for that call from the labor board to ask why we fired someone for who there was no record of employment.
It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.
Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.
It's still mind blowing and very amusing that this is a thing in our world!
Cause that sounds pretty great.
Look at the recent GitLab incident - one guy messed up and nuked a server. Okay, that happens sometimes, go to backups. Uh oh, all the backups are broken. Minor momentary problem just turned into a major multi-day one.
That's a problem and one which could be preventable with training (or, arguably, firing and hiring). Maintaining your backups properly should be someone's duty, designing and testing systems to minimize impact of user error should be too.
If someone doesn't test their backups, you train them to test backups. If someone lies about testing the backups, maybe you fire them. But if someone trips and shatters the only backup disk, you don't yell at them - you create backups that an instant of clumsiness can't ruin.
I did overstate, training is perfectly reasonable, but I often see it cited exactly when it shouldn't be, as a solution to errors like typos or forgetfulness.
Instead, you make a machine verify the backups simply by using the backups all the time. For example, at work I feed part of our data pipeline with backups: Those processes have no access to the live data. If the backups break, those processes would provide bad information to the users, and people would come complaining in a matter of minutes.
Just like when you have a set of backup servers, you don't leave them collecting dust, or tell someone to go look at them every once in a while: you just route 1% of the traffic through them. They are still extra capacity, you can still do all kinds of things to them without too much trouble, but you know they are always working.
Never, ever, force people to do things they don't gain anything from. Their discipline would fade, just like it fades when you force them to a project management tool they get no value from.
One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. The rest of the year it's only worth keeping an automated process reporting whether or not the things are being made, and people keeping an eye on change management to be sure no changes are made to the known-to-be working process that can break it without the new process incurring an explicit vetting cycle. Gitlab wasn't apparently testing or engaging in monitoring what was supposed to be an automated process. That's where they got burned.
Process monitoring may be boring as hell, but it's seldom wasted effort, and will prevent massive, compounded headaches from bringing operations to a chaotic halt.
Nope. Nope. Nope.
You test every backup by automatically restoring from it in a sandbox and verifying its integrity and functionality in the restored state.
Backups are worthless unless verified for their intended use of recovering a functioning system.
And constant "this succeeded" messages don't scale well.
Someone rm -rf / ing the server will happen eventually with near 100% certainty in any company and can be mitigated by tested, regular, multiply redundant backups.
Cosmic rays flipping bits will happen with near 100% probability at the scale someone like Amazon works at can by mitigated by redundant copies and filesystems with checksum style checks. Similar with hard drive failure.
Earthquakes will happen in some areas with near certainty over the time periods companies like Amazon presumably hope to be in business and could be mitigated by having multiple datacenters and well constructed buildings. Similar for 'normal' scale volcanoes.
Fires will happen but they can be mitigated (with appropriate buildings and redundancy).
Small meteorite stikes are unlikely but can be mitigated by redundancy.
Solar activity causing an electomagnetic storm - yeah one can shield one's datacenter in a Faraday cage but in this situation the whole world is probably in chaos and one's datacenter will be the least of one's concerns (unless shielding become standard in which case you'd better be doing it). Similar applies for nuclear war, super volcanoes, massive meteorite strikes or other global events at the interesting end of the scale.
But yeah there are going to be things that get missed. They key is having an organization that (1) learns from its mistakes and (2) learns from others' mistakes and continually keeps their risk modeling and mitigation measures up to date. And note that many of the hazards that are worth mitigating have the same mitigation i.e. redundancy (at different scales).
That's a great line. How should I attribute it?
Based on Amazon's decision to improve the tooling such that this category of error would be (hopefully) impossible to reproduce, I would lean more towards that being the case.
He won't make the same mistake because no one makes the same big mistake twice? I wouldn't bank on that alone.
The problem of user error can be mitigated by an appropriate level of OCD.
But OCD can't be trained, you either have it or you don't.
Tests and configuration scripts don't prevent all breakage. But when you have them, you can say, "We missed that, let's add it," or "That failed, but it's a false positive. Let's add this edge case to this test."
If you have no automation, tests or auditing systems around running deployments, you can't do any of this.
By the way - this is not just Amazon's problem now. We know the internet has a single point of failure. So does a lot of IoT.
When will we experience the first Suicide DevOps?
https://www.wired.com/2010/11/1110mars-climate-observer-repo...
(Specifically https://www.youtube.com/watch?v=6OalIW1yL-k#t=3m but it's worth watching the whole clip (or even the whole movie) if you haven't seen it before. It's from Terry Gilliam's "Brazil".)
It has? I have yet to see the day where I can neither reach my email provider nor Google nor Hackernews. My local provider might screw up occasionally, or some number of of websites go unreachable for whatever reason. But I fail to come up with anything short of cutting multiple see cables that causes more than 50% of servers to be unreachable to more than 50% of users.
[1]: https://en.wikipedia.org/wiki/Space_Shuttle_Challenger_disas...
Jeff Bezos once said: "Good intentions never work, you need good mechanisms to make anything happen"
Amazon is taking the right approach here. The fact that a system as complex and important as S3 can be taken down is a failure of the system, not the person who took it down accidentally.
1. https://en.wikipedia.org/wiki/Capability_Maturity_Model#Leve...
The certification is more for the organization/unit and the people working do not realize what they are for. Another thing that usually becomes a problem is the rigidity of the certification. Saying you need X, Y and Z documented is easy, but it doesn't work for projects that maybe don't have Y. So people make up documentation and process just to be compliant, this soon becomes a hinderance to the work. At this point people either abandon the process or follow it and the work suffers.
(I lied about the "insta" part)
I've had the privilege of either working for myself, the company that acquired mine and let me run the dev, or at Google. From that perspective, and what I understand about ops, the rarity is not having the attitude mentioned in the parent.
Plus, managing humans in a 'rat out' system would be incredibly inefficient. Now you need lots of employees just to listen to the ratting!
I seem to recall an EC2 or S3 outage a few years ago that boiled down to an engineer pushing out a patch that broke an entire region when it was supposed to be a phased deployment.
I could be mis-remembering that but it's important that these lessons be applied across the whole company (at least AWS) so it would be a bigger mark against AWS if this is a result of similar tooling to what caused a previous outage.
(Source: am a self-identified post-mortems connoisseur. :)
Is mere extra training the right solution here?
Maybe they need something like the procedure that's used in missile silos:
Not allowing the shutdown system to function at all without the explicit authorization of least two people.
That's a lot more than just extra training, and a lot better than a two-key system.
Probably a bad example. The system was a pain in the ass, so they went and circumvented some of its restrictions.
http://gizmodo.com/for-20-years-the-nuclear-launch-code-at-u...
> Those in the U.S. that had been fitted with the devices, such as ones in the Minuteman Silos, were installed under the close scrutiny of Robert McNamara, JFK's Secretary of Defence. However, The Strategic Air Command greatly resented McNamara's presence and almost as soon as he left, the code to launch the missile's, all 50 of them, was set to 00000000.
> Oh, and in case you actually did forget the code, it was handily written down on a checklist handed out to the soldiers.
I may have to upgrade that to take the mighty power of Cloud (TM) into account, though. Billions and trillions of fuck ups per second are now well within reach!
This quote (paraphrased) actually dates all the way back to 1969:
> To err is human; to really foul things up requires a computer.
If you made that up, I tip my hat off to you as payment for all my future uses of the phrase.
This is one of the things that happens with windows, getting up a server is so easy, that people believe that they don't have to understand what's under the hood, and then, we get a lot of miss-configuration and operational issues.
In other words, there's a lack of humility about "unknown unknowns".
For example on GCS (Google's S3)...A storage class specifies how many locations the data is made available. All storage classes share the same durability (chance of google loosing your data) of 99.999999999%, but have different availability (chance of being able to retrieve data).
git commit -m 'typo'
It says that their ideal-case failure rate is 11 nines; that's how much you should lose to known, lasting issues like machines failing and cutting over.
Amazon's actual SLA offers 2 nines and 3 nines as the credit thresholds. So they're stating the reliability of their known system, and the rest is for events like this.
The alternative is that I use the insurance to pay my legal fees when you sue me for not meeting my uptime guarantees.
My worst DELETE fail however was:
DELETE * FROM table WHERE [long condition that resolves to true for all records]
Now i write
SELECT or SELECT COUNT(*) over and over again until i see the data i expect and then change it to a DELETE/UPDATE.It's not my personal habit but some folks I know turn off auto commit and BEGIN a transaction every time they enter an interactive SQL sessions. They then default to ROLLBACK at least once before COMMITing them.
That and having a user with read-only permissions or a read replica
root@baz # shutdown now
W: molly-guard: SSH session detected!
Please type in hostname of the machine to shutdown: foo
Good thing I asked; I won't shutdown baz ...
Surprising to see such a simple protection neglected.For instance: https://thenextweb.com/shareables/2014/05/16/emory-universit...
Note: Automation is great, you just can't be sloppy with it. EVER.
edit:fix minor typo
A playbook actually represents a lack of automation for a particular task.
The playbook itself should be automated, with automated tests that validate its correctness.
Too often people will put up with the, "well, we only do this once a month so it's not worth automating". Literally, I script everything now, just in simple bash... if I type a command, I stick it into a script, and then run the script. Over time you go back and modify said script to be better, eventually this turns into more substantive application. At a certain point, around the time that you have more than one loop, are trying to do things based on different error scenarios, it's probably time to turn to rewriting it in another language.
The simplest thing this does for me, is guarantee that all the parameters needed are valid and present before continuing.
Though, an alternative to switching to another language is using xargs well. Writing bash with some immutably has been pretty invaluable for my workflows lately. For example
seq 1 10 | xargs -P10 -I{} ssh $host-{} hostnameI used to follow runbooks/playbooks written on the internal wiki when I worked at Amazon.
At $work, certain types of frequently-occurring alerts have playbooks that document how the alert in question can be diagnosed and how known causes can be remedied. Something like "Look at Grafana dashboard X. If metric Y is doing this and that thing, the cause is Z. Log on to box 16 and systemctl restart the foo.service."
So, fat-fingering something is imminently possible.
You have some process that starts out being "deploy this app with this java code". You deploy once and while, so it's not a big deal. But then those changes get a bit more frequent and so you pull out the common bits and the process becomes "make this YAML change in git and redeploy the app".
That works until you find yourself deploying 5 times a day, so you turn it into a MySQL table, and the process becomes "write a ROLL plan that executes this UPDATE x=y WHERE t=u; command"
After a while you get super annoyed at some quirk of the commands and figure, "Ok, fine, I'll just add an endpoint and some logic that just does this for the command case."
Then you wanna go on vacation and the new guy messed up the API request last week, so you figure, "I'll just add a little JS interface with a little red warning if the request is messed up in this way or that before I go".
You get back from vacation and some original interested party (whoever has wanted all these changes deployed) watched the intern make the change and thinks they could just do it themselves if they had access to the interface. You're wary, but you make the changes together a few times and maybe even add a little "wait-for-approval" node in the state machine.
Life is good. You've basically de-looped yourself, aside from a quick sanity check and button press, instead of what was a ~2 hour code + build + PR + PR approved + deploy process.
Then that interested party goes to work for Uber and the rest of your team adds a few functionalities on top of the interface you built and it all goes pretty well, until you realize that now that this thing that used to be 20 YAML objects is now 50k database records, and a bunch of them don't even apply anymore. So you build a button to disable some group of them, but after getting it deployed you realize it's actually possible to issue a "disable all" request accidentally if you click a button in your janky JS front-end before the 50k records download and get parsed and displayed. Oops! This mistake that you and the original interested party would have never made (because you spent the last 2 years thinking about all this crap) is probably a single impatient anxious mouse-click away from happening. So you make a patch and deploy that.
Congrats! You found that particular failure mode and added some protections for it, and maybe added some other protections like rate-limiting the deletions or updates or whatever. That's cool, but is that every failure mode? I bet it isn't. What happens when someone else thinks you have too many endpoints and just drops to SQL for the update?
Basically, yeah, of course you think of this stuff while iterating on it. But you figure "only power users are on the ACL" or "my teammates will understand the data model before making changes, or ask me first" or "that's what ROLL plans are for" or "I'll show a warning in the UI" or whatever. Fundamentally, you're thinking about a way to do a thing, if you're even thinking about it at all.
So yeah, that's what I've spent the last year or two doing. :-)
If the computer knows exactly what actions would be a mistake, why can't it just do the correct actions (those that aren't a mistake) automatically?
That's good UI design in tools with powerful destructive capabilities. You make the UI to do lots of things v.s. the few things you do routinely different enough that there's no mistaking them.
Which is why I'm pointing out that to design UIs like these you should fall back on slightly different UIs depending on the severity of the operation.
"1382345166 agents will be affected. Proceed? (y/n)"
Was that ~100k or ~1M agents? I can't tell unless I count the number of digits, which itself is slow and error-prone. It's worse if I'm in the middle of some high-pressure operation, because this verification detour will break my concentration and maybe I'll forget some important detail.Now if the number is formatted for a human to consume, I don't have to break flow and am much less likely to make an "order-of-magnitude error":
"1,382,345,166 (1.4M) agents will be affected. Proceed? (y/n)"
I always attempt to build tooling & automation and use it during a project, rather than running lots of one-off commands. I find this usually saves me & my team a lot of time over the course of a project, and helps reduce the number of magical incantations I need to keep stored in my limited mental rolodex. I seem to have better outcomes than when I build automation as an afterthought.Imagine I just entered a command to remove too many servers that will cause an outage:
"Finished removing servers"
(better than no message, I suppose)
vs "Finished removing 8 servers"
(better, it's still too late to prevent my mistake
but at least I can figure out the scale of my mistake)
vs "8 servers will be removed. Press `y` to continue"
(better, no indication of impact but if I'm paying
attention I might catch the mistake)
vs "40% capacity (8 servers) will be removed.
Load will increase by 66% on the remaining 12 servers.
This is above the safety threshold of a 20% increase.
You can override by entering `live dangerously`."
(preemptive safety check--imagine the text is also red so it stands out)Simple example: I have a git hook which complains at me if I push to master. If I decide "screw you, I want to push to master", it can't assess my decision, but it easily fixes "oops, I thought I was on my branch".
> The dosage you ordered is an order of magnitude greater than the dosage most commonly ordered for this medicine. Continue? y/n
That said, the only way to completely prevent mistakes is to make the tool unable to do anything at all.
(Or to encode every possible meaning of the word "mistake" in your software. If you could do that, you would probably get a Nobel prize for it.)
For securing very dangerous commands, I'd recommend asking the user to retype a phrase composed of random words, or maybe a random 8-character hexadecimal number - something that's different every time, so can't be memorized.
I've almost deleted my heroku production server even though you need to type (or copy paste....ahem...) the full server name (e.g. thawing-temple-23345).
I think the reason was that because in my mind I was 100% sure this was the right server, when the confirmation came up I didn't stopped to look if indeed this was the correct one so I mechanically started to type the name of the server and just a second before I clicked ok, I had this genius idea to double check.... Oh boy... My heart dropped to the floor when I realized what was I about to do.
You could say that indeed Heroku's system of avoiding errors worked correctly....
However the confirmation dialog wasn't what made me stop... Instead it was my past-self's experience screaming at me and remembering me that ONE time where I did fucked up a production server years ago (it cost the company a full day of customers' bids... Imagine the shame of calling all the winning bidders and asking them what price did they end up bidding to win....)
My point is, maybe no number of confirmation dialogs however complex they are, will stop mistakes if the operator is fixed on doing X. If you are working in a semi-autopilot mode because you obviously are very smart and careful (ahem..) you will just do whatever the dialog asks you to do without actually thinking what you are doing.
What, then, will make you stop and verify? My only guess is that experience is the only way. I.e. only when you seriously fuck up you learn that no matter how many safety systems or complex confirmation dialogs there are you still need to double and triple check each character you typed, lest you want to go through that bad experience again....
But I still believe auto-pilot mode is a real thing (and a danger!) .
My point is that I'm not sure if it's even possible to design one that actually cuts errors to 0.
And if that's indeed the case, even if it's close to 0, it's still non-zero, thus at the scale Amazon operates at, it's very probable that it will happen at least one time.
Maybe sometime in the future AI systems will help here?
I've also built complex systems that have been run in production for years with relatively few typo-related problems. The way I do it is with the design patterns like the one I just mentioned, which is also what TeMPOraL was talking about (and I guess you missed it.)
If you have the same kind of confirmation whenever you delete a thing, whether it's an important thing or not, you're designing a system which encourages bad auto-pilot habits.
You'll also note that Amazon's description of the way that they plan on changing their system is intended to fire extra confirmation only when it looks like the operator is about to make a massive mistake. That follows the design pattern I'm suggesting.
Personally, I don't believe it is without making the tool impotent. But you can try and push down the error probability down to arbitrarily low value.
You could go further and try to prevent cat-on-the-keyboard mistakes, which is maybe what you're describing (solve this math equation to prove you are a human who is sufficiently not inebriated). Or even further and prevent malicious, trench-coat wearing, pointy-nosed trouble-makers.
The point is, yes, it is possible. That's what good design does.
One thing I have been doing for my own command line tools is making a preview feature for what a command will do and make the preview state be default. It's simple, but if the S3 engineer first saw a readout of the huge list of servers that were going to be taken offline instead of the small expected list we probably would not be talking about this. There's obviously a ton more you can do here (have the tool throw up "are you sure" messages for unusual inputs, etc).
So, this means a) strong superhuman AI (good luck), b) deciding from an ambiguous input to one of possibly mistaken actions (good luck mapping all possible correct states), or c) drool-proof interface ("It looks you're trying to shut down S3, would you like some help with that?").
TL;DR: yes, but it's a cure worse than the disease.
I don't know. I was suggesting it wasn't realistic to do that, and therefore it wasn't realistic to implement a UI that prevents you making mistakes.
You could feedback a clarification, but if that happens too often nobody will double check it after they have seen it over and over.
A confirm step in something as sensitive as this operation is important. It won't stop all user error, but it gives a user about to accidentally turn off the lights on US-EAST-1 an opportunity to realize that's what their command will do.
no, because developers still produce code with bugs.
The tool as a whole should incorporate a model of S3. Any action you take through the UI should first be applied to this model, and then the resulting impact analyzed. If the impact is "service goes down", then don't apply the action without raising red flags.
Where I work we use PCS for high availability, and it bugs the heck out of me that a fat-fingered command can bring down a service. PCS knows what the effect of any given command will be, but there's no way (that I know of) to do a "dry run" to see whether your services would remain up afterward.
In practice, it would likely be very hard to make a model of your infrastructure to test against, but I can imagine a tool that would run each query against a set of heuristics, and if any flags pop up, it would make you jump through some hoops to confirm. Such a tool should NEVER have an option to silently confirm, and the only way to adjust a heuristic if it becomes invalid should be formally getting someone from an appropriate department to change it and sign off on it.
By the way, this is how companies acquire red tape. It's like scar tissue.
I suspect locking everyone down in the way you suggest would cost more in lost productivity (and costs for the infrastructure that would be required for greater auditing, etc.) than is lost in outages like this.
Those lines are not reflective of what Amazon is but what picture Amazon wants to paint now. They have clarified it is their error and not some hacking attempt. Secondly they have not vilified the engineer in question because already Amazon's culture is a bit of a ??? in public mind.
But they have got it right. Shit happens and this is not the first time it has happened or the last time it will happen. Also it will happen with Microsoft, Google and everyone else.
May be we will build even better technologies that will rely on two different cloud providers instead of 1.
I think Borland do some RAD systems, and Microsoft have an IDE of sorts on the way too.
EDIT: Please note that this is humour.
Actually the accounts I've read seem to indicate that most missile operators simply decided they would never launch no matter what. God bless them, for that.
This one did not carry a warhead. Others do...
Turns out the answer to your question is simply: luck.
EDIT: derp, my bad, I read as "unauthorized" which was "authorized".
Anyway, to me this firstly sounds like a "tool" or command that was too powerful with not enough safeguards. Who knows, the command might even be ambiguous.
typos happen, if the system didn't stop them that's a design issue or accepted risk.
" At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended."
"Fire you? I just spent $10 million training you!"