Hacker News Reader

Comment by nodesocket | Hacker News Reader

Love to hear the story how you took down Google for six minutes? Was this a long time ago?

This was in 2012. Let's just say that a router config only getting partially applied (because of a software bug) can cause a LOT of traffic to suddenly go places it's not supposed to go.

Oh wow, I was a noogler there 4ish years ago, and they still tell that story as an example of "its okay to break things, we care about fixing them after so they dont break in the future"

There's a couple of good stories about massive outages and good incident response. Mine is just one of them (and at some level, I was very lucky).

There's also the one where all the frontend servers worldwide went into a crash loop from a bad configuration push. The SRE doing the push noticed some "weirdness" and rolled back even before the full scope of the issue was known. That one's in the SRE book.

Site Reliability Engineering.[0] Google's SRE book is a pretty interesting read.

0. https://landing.google.com/sre/interview/ben-treynor.html

GFE? SRE?

SRE is Site Reliability Engineer; GFE is the "Google Front End"

Was a noogler about a year ago. Story still told.

It appears we have a living legend here!

They made him a slave of the packets though.

I think my favorite quote about my outage was "boy, it's a good thing you didn't decide to go take a crap right about then..."

db31.lpp01 lol

When they decommed it, I asked if I could have db31 to go Office Space on. They said no :-(

Reply on news.ycombinator.com