Hacker News Reader

Comment by packetslave | Hacker News Reader

BINGO. If someone makes a mistake like this, unless it was deliberate or negligent (e.g. he/she didn't follow established procedures), you don't fire them! You fix the technical or procedural failures that allowed them to make the mistake in the first place. Heck, you should probably THANK them for making the system better by uncovering a bug!

When I was still mostly a Noogler, I took a large portion of Google completely down worldwide for 6 minutes. Guess what? We did a postmortem, fixed the procedural and technical problems that allowed it to happen, and I'm still here!

View on HN

This was (probably still is) the best approach at AWS in the early days (source: I worked there from 2008 to 2014). Never blame the person, find out why he/she did the mistake and make sure it doesn't happen again.

An interesting variation to this thought is:

When is appropriate to fire someone for making a mistake?

Some ideas:

- gross negligence / malice

- ethics

- publicity damage / internet pitchforks

Additionally, it doesn't seem like the forgive mistakes approach meshes well with the startup advice of "it's never too early to fire."

Incompetence should probably be number one.

There is a big difference between someone making a mistake due to unclear or incomplete process and someone doing that mistake because they are not competent to perform that task.

This then raises the issue that hiring and leaving incompetent people in place is itself proof of an organisational incompetence...

Love to hear the story how you took down Google for six minutes? Was this a long time ago?

This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.

Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"

There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky).

There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.

Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read.

0. https://landing.google.com/sre/interview/ben-treynor.html

GFE? SRE?

SRE is Site Reliability Engineer; GFE is the "Google Front End"

Was a noogler about a year ago. Story still told.

It appears we have a living legend here!

They made him a slave of the packets though.

I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."

db31.lpp01 lol

When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(

Makes me think of the stories of the early days of school computing, and how the admin would give the kids that hacked said computers part time jobs as they had demonstrated themselves both interested and talented.

They say worker hit real button rather than test button. Some folks don’t believe this b/c test button is hit on 1st of month. It is not the 1st of month so why was any button hit, especially one that is so major, how could be accident?

Oof! I'm not envious of the way you must have felt when you realized what had happened. We've all (probably) had the same sort of feeling on a much, much smaller scale and I've never been one to enjoy that heart-sinking-into-my-shoes feeling!

Reply on news.ycombinator.com