Hacker News Reader

Comment by snewman | Hacker News Reader

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

View on HN

But it's not a post mortem? Since the article states:

> Next Steps

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem to ensure we understand all the contributing factors

Replying to my own comment:

As many have pointed out, this was not actually the postmortem, just an "update". I still find it pretty weak. There has been plenty of time to assemble basic details, such as a rough timeline. Yes they are busy with the actual response and cleanup, but this is a big professional team. The update feels less like "we're on it, here's what we have so far, more to come", and more like a bland PR minimization exercise.

I'll note that they've found time to determine that "For most Google users there was little or no visible change to their services", "YouTube measured a 10% drop in global views during the incident", and "approximately 1% of active Gmail users had problems", and yet they don't mention the time at which the incident was fully resolved or even when it started! Did the impact last for a few minutes, an hour, or multiple hours? Reading this, I have no idea. But I do learn that "the Google teams were keenly aware that every minute which passed represented another minute of user impact", they "brought on additional help to parallelize restoration efforts", "networking systems correctly triaged the traffic overload", and "low-bandwidth services like Google Search recorded only a short-lived increase in latency". In other words, this update on what seems to have been a very major incident consists mostly of vague, positive statements. It does say "we take it very seriously", but it doesn't make me believe it.

(I would add that I worked at Google from 2006 to 2010, and based on that experience I'm sure they are taking this very seriously and that there will be an excellent internal postmortem. But man, reading this sure makes it hard to remember that.)

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ...

This is not the post-mortem, that is still to come.

The postmortem is currently only visible to Googlers.

What happened?

TBH the linked update is a pretty accurate summary of the postmortem in my opinion.

> linked update

Where?

https://cloud.google.com/blog/topics/inside-google-cloud/an-...

(I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra.

A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

I think that's likely to be answered in the post-mortem, which is still to come.

Obviously to prevent things like this, Google needs more binary search tree whiteboard trivia problems in the interview process.

Prioritize configuration traffic so they can fix the problem quicker.

Test changes on smaller parts of the network before pushing them to critical parts would be a first guess.

Config changes tend to be nasty in that their implications are often hard to oversee until they have been made, and if the effects preclude you from making another config change then you've just cut off the branch that you were sitting on.

Google is best-in-class when it comes to this stuff, the thing you should take away from this is that if they can mess up everybody does. And that pretty much correlates with my experience to date. This stuff is hard, maybe needlessly so but that does not change the fact that it is hard and that accidents can and will happen. So you plan for things to go wrong when you design your systems. Failure is not only an option, it is the default.

Which seems to have been what they were trying to do; according to this update, the config change which caused the problem was intended to apply to a smaller portion of the network than it actually hit. But automated enforcement of procedures like this can also be tricky; how's a machine supposed to know that "this change was already tried on a smaller part of the network"?

This reminds me of an AWS outage from a few years ago, I seem to recall they started baking thresholds to protect against known points of failure after that so misunderstandings/finger slips couldn't bring down the system.

The article leaves it to the reader to decide whether or not we will be updated with the results of that post-mortem.

While this article leaves it ambiguous, the incident status report[1] makes it clear that customers can expect an actual postmortem:

"We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."

[1] https://status.cloud.google.com/incident/cloud-networking/19...

Its the pre-post-mortem.

The mortem ?

Perimortem?

In pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")

Yeah, the issue really is that post-mortem's been adopted as a noun.

This is a de facto post-mortem <thing> (allowing the adoption of 'mortem' to mean problem, but not the phrase as noun) just perhaps not the full analysis it's been taken to imply.

Pre-post-erous mortem?

The purgatorium.

It's definitely not a postmortem. Seems like just a quick update.

The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.

Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened.

If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?

The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.

We began noticing erratic behavior with some of our GCE instances at around 11:50 Pacific on the day of the outage, and Google posted a notice on their on their status page that GCE was having an outage about 30 minutes later. They also updated the status page every hour (or when they had new information), which is what they said they would do in their status updates.

While it sucks that multiple regions malfunctioned simultaneously for several hours, I can't really fault them for their communication about the issue.

I wasn't able to load a google doc from drive, or my calendar (both g suite), around 11:45 PDT.

There were an hours' worth of tweets before the status page changed. For Ops quite a shitty hour of uncertainty.

Gmail was down for me for at least an hour and all the lights were green on their status page

AWS's post outrage summaries are pretty much the gold standard.

e.g.

https://aws.amazon.com/message/2329B7/

https://aws.amazon.com/message/41926/

To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007. And balancing the scales, the AWS status page is notorious for showing green when things are ... not so verdant.

I have seen full <public cloud> internal outage tickets and the volume of detail is unsurprisingly vast, and boiling it down into summaries - both internal and external - without whitewashing, without emotion, to capture an honest and coherent narration of all the relevant events and all the useful forward learnings is an epic task for even a skilled technical writer and/or principal engineer. You don't get to rest just because services are up, some folks at Google will have a sleep deficit this week.

Google also posts detailed postmortems for their more significant outages.

Some examples:

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/cloud-pubsub/19001

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/compute/18012

Given that this was a multi-region outage that lasted several hours and impacted a substantial number of services, I'd expect a detailed postmortem to follow.

I hope one of the things Google learns in the post mortem is that the next day summary should clearly include that a full post mortem is coming in the next few days or however long.

Half the people in this thread are overlooking that fact and going into outrage mode.

“post outrage summaries” hehe

I love reading the AWS post-mortems since they're always very detailed in describing the roles of the impacted systems, the intention of the action that caused the outage, the actual action triggered, all the nuances involved in the bug or irregularities from expected behavior, impact to systems, complications, and resolution. It paints a very complex and through picture of how their massive outages are a collection of generally simple failures or oversights that had to all line up just right for catastrophic failure.

Every time I read a Google post-mortem, they seem to hand wave everything away as "a configuration error", "bug", or "bad deploy" and their resolution always has the generic "implement changes to things" that says absolutely nothing. Honestly, when the the causes of these massive disruptions are so simply dismissed, it portrays their system as frail amateur work.

as the other comments here state - wait until they post the full post mortem to draw conclusions. This reads like a statement to Gmail/Youtube customers, not a post-mortem for GCP customers.

This isnt the postmortem though. It says they're still working on that.

I don't experience this as anything like a lack of transparency.

The incident was less than 2 days ago, is resolved, and we have a preliminary report from the "VP, 24x7", which is easily digestible by the average GCP customer with more details undoubtedly to come.

they typically post a full post-mortem in the status page of the original incident: https://status.cloud.google.com/incident/compute/19003

What do you expect from a blackbox setup?

Reply on news.ycombinator.com