I read their SRE books, all of this stuff is in there: https://sre.google/sre-book/table-of-contents/ https://google.github.io/building-secure-and-reliable-system...
have standards slipped? or was the book just marketing
All the standard tools for binary rollouts and config pushes will typically do some kind of gradual rollout.
In some ways Google Cloud had actually greatly improved the situation since a bunch of global systems were forced to become regional and/or become much more reliable. Google also used to have short global outages that weren't publicly remarked on (at the time, if you couldn't connect to Google, you assumed your own ISP was broken), so this event wasn't as rare as you might think. Overall I don't think there is a worsening trend unless someone has a spreadsheet of incidents proving otherwise.
[I was an SRE at Google several years ago]
> a policy change was inserted into the regional Spanner tables that Service Control uses for policies. Given the global nature of quota management, this metadata was replicated globally within seconds
If there’s a root cause here, it’s that “given the global nature of quota management” wasn’t seen as a red flag that “quota policy changes must use the standard gradual rollout tooling.”
The baseline can’t be “the trend isn’t worsening;” the baseline should be that if global config rollouts are commonly the cause of problems, there should be increasingly elevated standards for when config systems can bypass best practices. Clearly that didn’t happen here.
I hope the pendulum swings the other way around now in the discussion.
[disclaimer that I worked as a GCP SRE for a long time, but not left recently]
HN likes to pretend that FAANG is the pinnacle of existence. The best engineers, the best standards, the most “that wouldn’t have happened here,” the yardstick by which all companies should be measured for engineering prowess.
Incidents like this repeatedly happening reveal that’s mostly a myth. They aren’t much smarter, their standards are somewhat wishful thinking, their accomplishments are mostly rooted in the problems they needed to solve just like any other company.
So I would say there is a difference between AWS architects and engineers (although I know first hand that certain things are subobtimal, but...) and those of several other companies who have less customers but experienced successful attacks (or data loss). Even if you take Microsoft, there is huge difference in security posture between AWS and Azure (and I say this as a big fan of the so-called "private cloud" (previously know as just your own infra)).
https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...
https://www.lastweekinaws.com/blog/azures_vulnerabilities_ar...
(2022)
I think you're only seeing what you want to see, because somehow bringing FANG engineers down a peg makes you feel better?
A broken deployment due to a once-in-a-lifetime configuration change in a project that wasn't allocated engineering effort to allow more robust and resilient deployment modes doesn't turn any engineer into an incompetent fool. Sometimes you need to flip a switch, and you can't spare a team working one year to refactor the whole thing.
This seems to imply that the person in charge at G was right to cause this outage... and that Google is very short-staffed and too poor to afford to do proper engineering work?
Somehow that doesn't inspire confidence in their engineering prowess. Sure seems to me that bad engineering leadership decisions is equivalent to bad engineering.
This has been the case for many teams since January 2023
If anyone needs to flip a global switch, and can't convince their leadership to allocate the resources to do it safely, engineering culture is dead, at least locally to that system.
But I'm not convinced lack of headcount was the problem here, the incident report makes it sound like there's an established pattern for feature flagging even for global changes like this.
Putting aside the fact that this team seems unperturbed by global deployments and all the other scary things, high impact changes should use every mechanism available to shrink fault containers. It would be inexcusable to roll this change out without tha feature flag mechanism if this were a regional roll out.
Skipping the feature flag when this is global is simply incomprehensible. It goes beyond headcount, it should never have been considered in the first place.
You can write as many unit tests as you want, and integration tests that your system works as you expect on sample data, static analysis to scream if you're doing something visibly unsafe, staged rollout from nightly builds to production, and so on and so on, but eventually, at large enough scale, you're going to find a gap in those layered safety measures, and if you're unlucky, it's going to be a gap in all of them at once.
It's the same reasoning from said book as why getting another nine is always going to involve much more work than the previous ones - eventually you're doing things like setting up complete copies of your stack running stable builds from months ago and replaying all the traffic to them in order to be able to fail over to them on a moment's notice, meaning that you also can't roll out new features until the backup copies support it too, and that's a level of cost/benefit that nobody can pay if the service is large enough.
When working on OpenZFS, a number of bugs have come from things like "this code in isolation works as expected, but an edge case we didn't know about in data written 10 years ago came up", or "this range of Red Hat kernels from 3 Red Hat releases ago has buggy behavior, and since we test on the latest kernel of that release, we didn't catch it".
Eventually, if there's enough complexity in the system, you cannot feasibly test even all the variation you know about, so you make tradeoffs based on what gets enough benefit for the cost.
(I'm an SRE at Google, not on any team related to this incident, all opinions unofficial/my own, etc.)
Opinions are my own.
But if you change a schema, be it DB, protobuf, whatever, this is the major thing your tests should be covering.
This is why people are so amazed by it.
The document also doesn't say there wasn't testing in staging or prod.
So do we. If you change the schema it doesn't matter if it's unit testing, you either have a validation stage for all data coming in beforehand to check these assumptions (the way many python people use pydantic on incoming data, for example), and that must be updated for every change, or every unit must be defensive against this sort of thing.
- And they used machines without ECC and their index got corrupted because of it? And instead of hiding the head in shame and getting lessons from IBM old timers they published a paper about it?
- What really accelerated the demise of Google+ was that an API issue allowed the harvesting of private profile fields for millions of users, and they hid that for months fearing the backlash...
Dont worry, you will have plenty more outages from the land of we only hire the best....
On GKE, we see different services (like Postgres and NATS) running on the same VM in different containers receive/send stream contents (e.g. HTTP responses) where the packets of the stream have been mangled with the contents of other packets. We've been seeing it since 2024, and all the investigation we've done points to something outside our apps and deeper in the system. We've only seen it in one Kubernetes cluster, and it lasts 2-3 hours and then magically resolves itself; draining the node also fixes it.
If there are physical nodes with faulty RAM, I bet something like this could happen. Or there's a bug in their SDN or their patched version of the Linux kernel.
Hm, checksum validation isn't really possible since these are protocols out of our control, like Postgres, NATS, and various other HTTP-based services.
But we could write some kind of simple canary process constantly doing dummy network access and detecting corruption.
Google’s standards, and from what I can tell, most FAANG standards are like beauty filters on Instagram. Judging yourself, or any company, against them is delusional.