Google Cloud Downtime Postmortem

> On an unrelated note: kudos to Google for publishing this postmortem and hope that this becomes an industrywide practice.

I think most already do this. I have seen AWS also publish detailed postmortems for outages like this. Ex: https://aws.amazon.com/message/41926/

Sometimes bugs are triggered by very specific data or queries (aka "queries of death"). GFEs have many tests, so I'd go for the subtle bug.

With all due respect, it's not a postmortem, it's an advert. It doesn't really say anything other than "We had a problem, we fixed it.". There are virtually no technical details in there other than "something would restart spontaneously, which shifted the load somewhere else". Maybe i'm a bit jaded by cloudflare and aws writeups, but this really isn't anything special or worthwhile reading.

It reads to me like the bug required a particular feature to be enabled, and the feature wasn't enable when the software was rolling out, then the feature was globally knife-switched to the "on" position without an orderly staged rollout.

I actually think it looks pretty bad that one of the action items in this report is to make a feature dashboard for GFEs. They've been saying they will do that for years, and the team that operates GFEs is considered the most elite of all SRE teams. Most famous outages of Google products have been caused by bogus configurations pushed to GFEs or network devices in front of them.

As it says in the note, the bug was in a feature that was latent but not yet being used. Then a configuration change started hitting the feature.

With highly redundant systems such as this, you generally need multiple layers of things going wrong all at once to notice an issue. This was the case here as well.

This is super high level summary not a postmortem, in fact it doesn't say anything about the bug and why there was no test for it, what repair items are done, etc..

Amazon and Microsoft's post mortems are much more to the point, one can actually learn from them to not make the same or similar mistake.

> ... this bug had not been detected in either of testing and initial rollout.

It makes me so happy[0] that the big smart corps have the same problems that us plebs have.

I tell people that testing can never precisely duplicate the production environment but do they believe me?

Also, this is an argument for feature switches vs staging environments.

[0] But not in a schadenfreude sense.

  Either tests must be poorly written or
  the bug must be very subtle.

The easiest way to make a web service work in testing but fail in production is a problem with the settings that are necessarily different between environments.

For example, you must have different database credentials between test and production, and you must limit who can read the production credentials. If the production credentials are malformed, a service that worked in test will fail in production.

And the same applies to your SSL certificates, your settings for enabled/disabled features, your flashy markers that stop people mistaking production for test....

Another possibility is that the change was not tested at all or very minimally tested -- and they would not want to admit to that publicly.

+1'ed you on the props to GCP, but on Amazon prime, that was only on their retail site, right? Do vendors there have SLA or is there otherwise an obligation to publish a post-mortem on the incident? I think it's different when it's a platform/service provider but the prime day outage only hurt Amazon. My recollection is that AWS service outages had prompt and thorough published incident reports.

My guess is that amazon's post mortem is basically: we misjudged how many people would be hitting our site. not that interesting or noteworthy.

> Meanwhile.. waiting for the Amazon prime day post mortem.

Why are you comparing a cloud provider with a retail site?

There was an issue in AWS Frankfurt yesterday, I'm waiting the post-mortem on that.

Maybe they were on reddit. Seriously though, 30 seconds vs 3 minutes is a long time in the reliability world but not atrocious.

Reading the post mortem, I'm guessing it's more like it took them 30 minutes to figure out what was happening and then they rolled back the deploy and it was fixed.

Edit: Reading further down, they actually admit that it was just a roll back

> At 12:44 PDT, the team discovered the root cause, the configuration change was promptly reverted, and the affected GFEs ceased their restarts

You're getting downvotes because this was not a transparent and open report. It was vague and was more advertise-y than postmortem-y.

Not saying Amazon is perfect by any means either, but there's a lot of room for improvement. Good postmortems give everyone ideas on how to solidify their own processes and prevent other issues. This was just fluff.

Well the outage itself died. Hopefully. Otherwise it would just be an incident status update.

It's called a 'post mortem' in the medical world because the patient died. It involves a high level of inspection into what went wrong and what can be learned to prevent it. I assume the term was adopted from there into project management.

It might make a little more sense in the world of shipping software in retail boxes, where products/projects had a 'done' date. The project is dead, what contributed to it's demise? Or you might generalize death into failure, and that's why we use the term instead of post-incident.

The earliest I know of: https://books.google.com/books?id=mskkmVkpIUcC&lpg=PP1&pg=PP...

I've seen the act of analyzing project failures by this name in software engineering/management ever since.

Maybe it was a bit more relevant in the days of large waterfall software project management, where failure often meant the end of a project with no product launched. Sometimes after a "death march": https://en.wikipedia.org/wiki/Death_march_(project_managemen...

But it does seem natural to me that it has been carried over to current days, and applied to analyzing failures in the context most relevant to modern software development.

Never stopped to think about the weirdness of the term's application. It feels so natural to me, I'd suggest we all start calling the fixed service as "resurrected". As in: "Google Cloud Global Load Balancers were resurrected at 13:19." (-:

The term 'post-mortem' doesn't appear anywhere on that page. It was added by the OP in the title. In reality it should be called an incident report.

If we want to be technical, a post-mortem in the tech world is commonly used to outline failures that occurred during a normal event (i.e. a software release) not a random production issue.

Bobby Tables?

We've been on Appengine for years, and one of the un-intuitive benefits is that when something goes wrong, all you can do is sit back and tell your customers "a team of the best sysadmins in the world are working on it right now".

I've met many people who hate that abdication of responsibility, and would prefer to be heroically hacking solutions at 3am when their database replication fails. Maybe they feel they have to be punished for failing their customers.

My advice is keep testing GC, because in my experience it is very reliable. And once you realise that, the peace of mind is awesome.

My understanding is yes; I only use regional, which encountered no outages. This hugely affected appengine users. I've not encountered a lot of GKE users that use Global LBs yet (I've run in GKE for roughly 4 years).

It's a global load balancer.

genuine question, whats Valhalla? Google returned something Norse.

All major cloud platforms have occasional unplanned downtime. AWS had an outage last year that took many sites offline for hours. A single instance of Google having such unplanned downtime is meaningless without more datapoints.

As for whether it's useful for serious business. Well. Proof by example? This has sixteen pages of case studies: https://cloud.google.com/customers/. That is by no means all of GCP's serious customers.