When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!
There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.
0. https://landing.google.com/sre/interview/ben-treynor.html
When is appropriate to fire someone for making a mistake?
Some ideas:
- gross negligence / malice
- ethics
- publicity damage / internet pitchforks
Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."
There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task.
This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...
This isn’t a newbie getting set up on his dev db who was given too much access to production. They’re pointing the finger so that mechanism that facilitates single point of failure should be independently audited
when you're dealing with processes as critical as this one...
there should be no single point of failure.
Even our President has a backup, a failsafe, and a safeguard for the purposes we're discussing here. And that backup has a backup, a failsafe, and a safeguard. Etc etc etc.
It was an open secret that nuclear retaliation could be instigated without White House or even Pentagon approval if the Soviets tried to decapitate Nato.
A rational player would only put effort into a retaliatory strike capability to the extent that actually having a retaliatory strike capability strengthened the enemy's perception that they would be obliterated if they struck first.
I generally agree with the sentiments of this thread - you shouldn't fire people for the kinds of mistakes that all humans frequently make. However, I also think that this was a severe and costly incident - a large number of people thought for a while they were in significant danger, and investigating who is responsible, and making corrections with regards to their employment, may be reasonable depending on the exact circumstances.
I'll give an example. I work in an office building with many floors that have exact same layout. One day I was on a floor that wasn't mine, and went to where my desk should be, and tried to sit down. I was shocked, for a moment, to discover my coworkers had all been replaced by strangers. In this case, I made an absentminded mistake of not paying enough attention to my current location - but the consequences were trivial, so the mistake itself is no big deal. Now, on the other hand, suppose while driving down the road, I absentmindedly ignore a red light, plow through an intersection, and kill a family of five. This is also an absent minded mistake, but the predictable consequences of being absent minded while driving are a lot more extreme than being absent minded while walking about an office building.
To apply the logic of my example to the case at hand - imagine that in one scenario an employee is, during a boring moment, cleaning his keyboard with his chat application open. This results in that employee sending a nonsense message to a coworker. Silly mistake, and no big deal. Now, suppose that same employee has the "Missile Alert" app open and decides to clean their keyboard. This is a similar mistake, but with potentially bigger consequences. I think it's fair to treat these two actions differently.
Of course, the circumstances affect what the reasonable standard of care is. Oopsies by ship captains, surgeons, and even drivers are held to a different standard than me making a typo on some random thing I'm writing or a simple program.
In your example, running a red light is commonly taken as one of those things you just don't screw up on (and that can't be eliminated through automation at this time).
No there aren't. There are certain activities that must be always correct, you implement those by making a reliable system out of unreliable jobs, not by making the people 100% reliable.
We see it all the time in IT - people totally misjudge the 'risk'. 1% risk of breakdown quite often seems acceptable to decision makers until it actually happens. Then all hell breaks loose.
I believe it is much better to fix it (require a second authentication before sending those massages) and search for similar problems within the whole department to make sure something like this never happens again. Make sure everybody, including new hires, are aware of this incident, what the consequences were and that everybody is encouraged to report similar bugs and nobody has to be afraid to be fired when they make a mistake. Otherwise you just create an environment where everybody blames anybody but themselves and the “smallest” guy gets fired for mistakes other’s. This way, things get never fixed probably because everyone ist just afraid of the consequences.
if you direct the blame 1 level further, then it was whoever was responsive for budget/hiring/spec that didn’t ensure that there was a UX designer on the project, but at the same time you can kinda understand why not: the value of UX is frequently not understood, especially when people think “it’s just internal; why make it easy to use if only we see it?”. the answer of course is that you pay for internal systems every day in employee time.
anyway, the point i guess is you can probably trace the “blame” back far enough that it’s too far removed to say it’s anyone’s fault per se, but it proves the point that you need design across the board, and that’s frequently not well understood
From what I understand, we were told the record would never get removed, and we relied upon that. Could our error handling been better? Yes. Could the customer's procedures been better to prevent the record from being deleted? Yes. It was a mistake on both parties.
Who should have gotten fired?
[1] Very scary SLAs with the Oligarchist Phone Company.
[2] It's actually a DNS query. NAPTR records to be precise. Search on that if you are interested.
[3] It was also important because that was the only record we could query and not get charged for it. Everything in the telephony world cross-company (and even cross-department) gets charged. Somebody makes money; somebody pays money.
In this particular case no real harm was done and firing would probably be excessive.
I wish to believe, but probably there was real harm, probably someone will come up with some million figures.
Statistically, probably someone died during those minutes of terror and someone will try to attribute to this incident.
What is interesting that employee probably meant no harm, but compare this to that SWAT murder a month earlier. The prankster meant no harm either, but he's going to spend years in jail. Both for sending a wrong alert.
Swatting is intentional, triggering the alert wasn't (at least, so it seems). It's not like the operator pressed the button knowing that it would trigger an alert and expected it not to be taken seriously; he didn't mean to trigger the alert at all.
Ongoing car slams into you and kills your whole family while you were driving drunk; you go to jail for manslaughter and get sued by other driver for all you've got.
Are you kidding? The people who were involved in this didn’t do it to introduce the swattee to new friends. They did it to terrorize someone who made them mad.
What I meant is that “oh I didn’t mean it” going to be interpretted differently.
Edit: if you downvote me thinking that I somehow support prank swatters - you are wrong. Otherwise you fail to see the paradox of intended vs unintended outcomes: https://en.wikipedia.org/wiki/Mens_rea (as I understand it's interpreted very different in the US).
The prankster intentionally sent a false alert and intended, at a minimum, to cause emergency response resources reserved for use in circumstances that pose a grave danger to be misallocated and to cause terror in the intended target; that involves both a diffuse but significant public harm and a narrowly focussed private harm.
It is not at all a case where no harm was intended.
If someone really, as you suggest, recklessly disregarded some procedures because they just couldn't be bothered. Say there's a checklist and they just winged it instead. But maybe there isn't a checklist and they made a mistake.
In general though, firing people to send a message or to throw a sacrifice to the crowd isn't particularly admirable practice.
Why not fire the manager of the person who made the blunder? That would work better on every level.
Her story was regarding an employee adding water to acid instead of the reverse, thereby creating an acidic cloud initiating an evacuation of the building [we're talking high volumes of acid and water here; think pumping in the water using a garden hose].
"You'll never have me doing this again," the person managed through tears.
"No, I'll have you do this from now on as my go-to person," was the response, "because I know you'll never do this again."
This was our "Add Acid" lesson.
The other perspective is that most people would not need to learn that lesson in the first place. This employee is unlikely to make this type of mistake again, but broadly speaking, is he less likely to make mistakes than someone who wouldn't have made this mistake in the first place? Doubtful.
The procedure needs to be reviewed of course. Why is such an important event the responsibility of one person? At least two persons should have to press two buttons and it should be made such that one person cannot press them both.
People saying nothing happened - I don't agree. This could cause serious problems for individuals, heart attacks, accidents. This is not without risk.
A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, "I suppose after that set of mistakes you will want to fire me." Watson was said to have replied, "Not at all, young man, we have just spent a couple of million dollars educating you."
After the sale feel through the VP of sales scheduled an appointment with the sales guy. The sales guy began cleaning up his desk and boxing his things anticipating what the outcome of the meeting would be.
Upon meeting the VP the sales person profusely apologized and accepted responsibility and said he understood why he was being let go. The VP interrupted him and paused for a moment before telling him "I did not invite you here to fire you. I wanted to see how you were doing and ensure that you learned a valuable lesson. The last thing I thought about doing was letting you go after spending millions of dollars on your education and training."
Therein is a key responsibility of the manager: if they've learned, you're 100% right. If they haven't, don't care, don't get it or whatever, then you also need to take responsibility & replace that person.
This doesn't work in all contexts, and I suspect it fails for a large majority of contexts. If the system always prevents you from doing the wrong thing, it will also often prevent you from doing the right thing.
For example, I submit that the philosophy "ability to make a mistake means that the system is designed wrong" completely falls down in the case of going to a new restaurant for lunch and finding out you hate the food there.
I wish this claim, which is very common in discussions like these, came with some analysis of why this odd philosophy which is a terrible idea in so many contexts is instead a good idea in the context at issue.
No way we should accept errors like this as some sort of blameless par for course in software engineering. Especially as SV grows more and more into life-critical systems like autonomous vehicles. If there was deep incompetence in architecting a system like this then yes, potentially the employees responsible for that architecture could be disciplined. If the responsible employees were too junior to design appropriate safeguards correctly, then the finger points at the managers who set up the team that way.
It sounds like the parties to blame here lie somewhere in the management chain between the poor unfortunate button-pusher and the director of all emergency management for Hawaii.
[0] https://en.wikipedia.org/wiki/Japan_Airlines_Flight_2#The_"A...
This is different than Capt. Asoh's defense because Asoh was directly responsible for
> attempt[ing] an automatic-coupled Instrument Landing System (ILS) approach, something neither [he nor Capt. Hazen] had done before on a recorded DC-8 flight.
In any case, your reference to the defense is fantastic. In the profane spirit of what Asoh told the National Transportation Safety Board investigators--"As you Americans say, I fucked up."--I think it's not too off the mark to observe "What an Asoh". [0]
[0] I tried resisting this but it's just sitting there, y'know. Also, I'm sure my American eyes are mispronouncing Captain Asoh's name. For what it's worth, we should all be so forthright to own up to the mistakes we've made.
I was shocked at the 2.5mi deviation until I realized that this is about 51 seconds of flight at his airspeed of 177mi/hr.
Still, that's a pretty astonishing distance from the runway.
As I commented below, if this happened when the stock market was open, this could have had a huge impact financially. Over 30 minutes with no updates is an eternity for news to hit Wall Street and algo trades start kicking in.
There is a lot of fearmongering just lately around the idea that there's going to be a nuclear war with North Korea because the president likes to talk shit on Twitter. I've had so much practice, with my explanation of all the reasons that won't happen, that by now I could do it as an elevator pitch if I needed to. That's why so many people are having this "weird panicked reaction" - panicked, yes, but not weird, when so much effort has been spent to insist that an arrantly implausible counterfactual is not just possible but likely.
(On that note, I expect the next wave of articles and op-eds to revolve around Hawaii's recent "close call", and why it means everyone should be much more scared than everyone is already. Who cares about externalities like the health of the republic when there's attention to be monetized?)
Could they do it so say, two people have to input a code or something from different machines, or something more physical like turning two keys, hitting a button etc
Didn’t stop people from doing it.
That very afternoon a plastic guard was installed over the button so you had to lift it first.
Must be reading some DevOps blogs about blameless post-mortems.
I think a small group of people feel "mission accomplished".