Post Mortem of Google Outage on 14 December 2020
status.cloud.google.com
status.cloud.google.com
My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular application stored only a few MB at the time, because it was brand new and hadn't accumulated anything yet. I had a very angry customer calling up asking me to please explain why they were seeing an average of 0.1 IOPS...
From: https://docs.aws.amazon.com/efs/latest/ug/performance.html
"The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage).
AWS now provides a performance floor of 1 MiB/s, but at the time there was no floor. If I remember correctly, this application had something like 2 MIB of data, which was constantly being updated by various processes, so there was no quota being accumulated. The system performance went from something like 1 Gbps to 100 bytes per second instantly. It took 10 seconds for a 1 KiB I/O to complete. Fun times, fun times...
Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking it should work? Fixed cost per block transferred?
2) Wrong relationship; you're paying for IOPS to the block store, so you'd be trampling on other accesses to the same block store.
3) This penalizes them less - in that workloads that do lots of small IO on small files will actually be able to request the IOPS they need, instead of IOPS being (wrongly) dynamically allocated.
See the section "Specifying Throughput with Provisioned Mode" here: https://docs.aws.amazon.com/efs/latest/ug/performance.html
This way, if you need 10MB but also 25,000 IOPS, then you'll pay for ~10% of a drive. You'll get your minimum required 25,000 IOPS....and also 200GB or whatever share of the drive is required to get you those IOPS.
At the end of the day, I'm not entirely sure it matters whether the cloud provider breaks it out like this as long as it at least has a little gray UI element under the specified storage space slider that reads out the IOPS to you.
It would be exactly the same as the current situation where customers do this manually. So probably not particularly necessary - why make customers fill in additional fields they might not need to?
I do think cloud providers should make it clear during requisition and read back how many IOPS you're getting, just for clarity.
It does seem like having temporarily high quotas and then throttling back seems to break developer experience. Good deeds and punishment - but in this case there's a solid underlying reason why it has detrimental effects.
> General Purpose SSD (gp3) - IOPS 3,000 IOPS free and $0.006/provisioned IOPS-month over 3,000
> General Purpose SSD (gp3) - Throughput 125 MB/s free and $0.0476/provisioned MB/s-month over 125
I didn't get an alert.
There was nothing in the logs.
There wasn't anything in the portal to indicate that something had changed.
Everything was up and responding, just really, really slowly.
The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time.
After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on.
Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.
The golden rule of both backups and monitoring is: There are no time machines.
It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once?
Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured?
That's not a rhetorical question!
Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't.
Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end.
Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream!
Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes?
PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.
Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS.
And yes, that's with the hardware investment and wage costs rolled in.
I feel like AWS and others created some sort of one ring, and just reeled admins in, pulling them from the wild, so that with a shortage of greybeards, people can't find those with the 'sysadmin temperament', and are therefore stuck with cloud.
And it is indeed a specific temperament, to create stability, which means creating some constraint, yet to forge that constraint in the most effective way, whilst doing one's best to enable devs to work most effectively.
AWS has constraints, but because it's a wall of "this is the way external corp does it", it gets far less flack than the guy that smells like onions, when he says "we can't do this safely" or "this policy must be followed".
Yes, OK, this is a bit of a rant. Sorry.
The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best.
The cloud is eating their lunch for a reason.
Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app.
Everyone wants unified logging built-in, not Splunk on top of fifty different log formats, none of which can be easily correlated with each other.
Everyone wants a unified IAM system, nobody likes to deal with expired SAML or LDAP certificates in the middle of a change.
Etc, etc...
Compared to having to deal with 10 different teams just to spin up a single VM and 15+ teams for a moderately complex application with HA/DR and monitoring, being able to simply click through a cloud portal is an absolute joy.
It's the same thing that made VMware so popular. Nobody liked to have to get finance involved and order new kit six weeks ahead just to be able to spin up a tiny web server.
Primarily, only buy what you can work with easily. Mainly, esure raid monitoring can be scripted, and you really need not depend upon the manufacturer's horrid software (which I agree is just that). A bit of IPMI for failed power supplies, or what not. If a box dies, it dies, that's what failover is for.
I don't your logging complain, you need it no matter what you do. Just use an rsyslog server, logcheck or other app, monitor for important events, done. I've never used anything off the shelf, and yet all I hear are complaints from those that do.
I get how people might want to eschew the above for simple stuff, but once you start needing heavy monitoring, it's all the same ball of wax.
I kill the job, start again, and the problem is there from the very beginning. Kill again, review my code for an hour or two (python, grequest) tweak some parameters, start the job again and it seems fine, problem solved? Nope, halfway through the same issue occurs.
More testing showed even that "halfway through" wasn't consistent.
It's the hidden quota, and it wasn't even consistent: Sometimes it would go 10%, sometimes 70%. I don't know if they even had static limits or some type of dynamic system. Then you get throttled, only requests fail without any indicative error message except the "failed" or "unavailable" (I forget which)
I determined the limit was tied to the API key. Given that error messages & the unknown limit made working within their hidden limits difficult, and support was neither sympathetic or forthcoming with details necessary to work within their limits, I simply created multiple API keys & rotated between them on requests. Probably not the best behavior for a tenant of a shared system, but I didn't see much choice when the alternative was to not use an essential advertised feature.
It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd like that.
Even if you called support, there was not a lot they could do, because the rate limiter is a low-level thing built into an internal load balancer somewhere, and it is difficult to override it for a single account.
The worst part was that you could hit the limit while not actually taking any action! If you just had certain Azure Portal screens open, the JavaScript would refresh things in the background, constantly consuming the API rate limit quota. Some screens do so many calls that you'll hit the rate limit in a matter of minutes if you leave your browser on those. If you fail to close all browser sessions of all admins right away, you can lock yourself out for days.
Gotta love it!
Queue overflows are not uncommon.
"For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog."
Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
They used to advertise 100 per second, but then they pushed a big update that doubled the read QPS limit, and conveniently left out the reduction in LIST qps from the docs.
It took several hours and some back and forth with support to realize that the burst IOPS quota of the provisioned underlying EBS disks on the EC2 instances forming the ECS cluster had been depleted so disk performance completely tanked to the point the docker agent couldn’t be reached for 4 minutes.
And here I thought the whole point was that these things would be abstracted away...
Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough.
And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message.
Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.
Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.
But a certified AWS Architect, especially at the higher levels will easily spot the big gotchas from just your architecture diagram. Consider getting certified yourself, if you want to be more confident around these in the future. But there really is no replacement for just putting in a lot of hours.
The project I'm working on right now is using a PaaS stack is a bit of a "moving target". I feel like those people running to catch a train that's still rolling along...
Both were solved by the same thing,
Provisioning a few terabytes of storage space with nothing on it.
Does the mean that baseline is only realized when at least a Tib of storage is actually being used then? In other words there was a distinction between how much you storage you were actually using vs how much storage you provisioned? Are there other services that use this same model?
Edit 1: Oh wow I didn't even realize there were multiple incidents. Thanks!
Edit 2 (after reading the Gmail post-mortem): AH! It was a messed up domain name! When this event happened, I said senders need to avoid taking "invalid address" at face value when they've recently succeeded delivering to the same addresses. But despite the RFC saying senders "should not" repeat requests (rather than "must not"), many people had a lot of resistance to this idea, and instead just blamed Google for messing up implementing the RFC. People seemed to think every other server was right to treat this as a permanent error. But this post-mortem makes it crystal clear how that completely missed the point, and in fact wasn't the case at all. The part of the service concluding "address not found" was working correctly! They didn't mess up implementing the spec—they messed up their domain name in the configuration. Which is an operational error you just can't assume will never happen—it's exactly like the earlier analogy to someone opening the door to the mailman and not recognizing the name of a resident. A robust system (in this case the sender) needs to be able to recognize sudden anomalies—in this case, failure to deliver to an address that was accepting mail very recently—and retry things a bit later. You just can't assume nobody will make mistake operational mistakes, even if you somehow assume the software is bug-free.
Also, that problem occurs due to lack of authorization, not due to authentication failure, right? Authentication failure is a different kind of error to the client—if the server fails to authenticate you, clearly you already know that it's not going to show you anything?
"Gmail - Service Details" ( https://www.google.com/appsstatus#hl=en&v=issue&sid=1&iid=a8...) shows 12/15/20 as the date.
That links to "Google Cloud Issue Summary" "Gmail - 2020-12-14 and 2020-12-15" (https://static.googleusercontent.com/media/www.google.com/en...) which mentions the 550 error code.
Obviously an error condition should not result in this. But complex systems. It happens.
Can't get quota? Certain pieces accidentally turn "TooManyRequests / 429s" or related semantics into 500s, 400s, 401s, etc. as you percolate upstream.
Auth is one of the most central components of any system, so there would be cascading failures everywhere.
If you don't want to scroll to the other thread, here's the postmortem: https://static.googleusercontent.com/media/www.google.com/en...
Both are noted as being related to “ongoing migrations,” though AFAICT not related ones. I would bet there’s a human factor connection- e.g., the day before there was a big meeting where a higher-level management gave multiple ops teams go-ahead on their respective plans, resulting in a multiple potentially breaking changes occurring at the same time.
(there is wiggle room, exception granting, and grandfathering on this policy but it's true for many things)
As of yesterday there are some google email customers like NOAA.gov that cannot receive emails from external mailservers (like my personal domain mailserver I run) because they are now proxying through some "security consultant service" ala mx.us.email.fireeyegov.com which causes the SPF validation to fail because it's no longer the external mailserver's IP that's sending it.
Received-SPF: fail (google.com: domain of superkuh@superkuh.com does not designate 209.85.219.72 as permitted sender) client-ip=209.85.219.72;
I've re-signed up for the email lists that gmail's behavior got canceled and subscribed to them with my personal domain/mailserver. It's incredible that a random $5/mo VPS has given me better uptime over the last decade than all of google's infrastructure.
https://tools.ietf.org/html/rfc5321
I haven't read the article, but SMTP response codes are very specific. If there were SMTP response codes in the 5xx range, that's a perm failure, end of story. Temp fail messages, 4xx, can be used for 'try later'.
That's how SMTP works. What you are suggesting would break email.
No it would not. Firstly, because you would just rejected twice; the sky wouldn't fall down. Secondly, because the RFC in fact permits this too; it's within spec even if you want to follow it blindly to the letter with zero consideration for the context. People already litigated this in the earlier discussion and there's no point rehashing it so I'll just leave it at this.
Meanwhile, the end user sent a message to bod@ by accident, instead of bob@, and your mail server keeps retrying to send mail, even when the remote mail server said "Hey! That account doesn't exist!".
It's a typo, but you've decided to 'make things better', so now, until that bounce happens, the end user won't find out they made that mistake.
Things work as they do for a reason, and the people saying 5xx means bounce! are right.
Obviously there are edge cases, and obviously you don't just retry every 1ms forever, but to assume that an error that comes back from a system you've called once is both (a) the product of a 100% working non-faulty system and/or (b) not in any way a transitory issue or failure is naive. This type of thinking is how we end up with brittle systems.
A cautious implementation might see an SMTP 500 and then may just choose to try it again once or twice a little later, to see exactly how 'permanent' that failure is.
What the "but the RFC says!" people seem to forget is that the spec is only meaningful when the system is working correctly. Good engineering never assumes that.
Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability.
A 5xx error means 'perm failure'. There are a variety of 5xx class responses, from 'user account deleted' to 'no such domain'. It is, in fact, a working mail system, which responds with 5xx, or 4xx (temp fail) or 2xx (received OK) messages.
A severely broken system is incapable of even responding.
For a mail system to respond with 5xx, when it is internally broken, is 100% a configuration issue. Every MTA on the planet, is designed (eg, postfix, sendmail, etc) to respond with a 4xx TEMP fail message if something is borked. A milter gone bad. A library missing. A full disk. A config issue. An issue forking. Memory limits. All of it.
By default, MTAs are designed to 4xx(tempfail) on those errors. Loads, and loads, and loads of work to ensure that. Code meticulously crafted. This is how good engineering works. This is how the RFC works.
You either get no ACK for your SYN, because it is so borked, or you get a 4xx if it can run, but has a failure condition.. OR someone did something very, very wrong.
I understand your angst, but the real problem isn't the RFC, or the sender not retrying. The problem is:
- the MTA was up and running
- it had an error condition on its back end
- whomever set everything up, didn't taking into account internal failure conditions, and to respond with a 4xx if that was the case
THAT is where the "good engineering" failed.
Mail accounts are deactivated all the time. Domains are deactivated all the time. 5xx tells us this. Someone replying with 5xx, when they did not mean to, has a configuration issue on their end.
That's all there is to it.
I'll put this another way.
What you want to do is make 5xx like 4xx, because you feel there should never be any way for a SMTP server to say "No, really, this email address doesn't exist.. don't bother trying again".
On a 5xx series response, you bounce. Where does the bounce go? Back to the original sender.
This is the 'closing of the loop'. Original sender sees the message was not sent successfully. They can now retry, resend, make a phone call, whatever.
"Hey Bob, what was your email address again? I got a bounce. Ohhh, damn, I typed bod@ instead of bob@. I'll fix and resend.
That's how it works.
That's how it works for:
- a typo in the email address
- when an account is deleted (and maybe your friend has a new email address)
- the remote mail admin did something horribly, horribly wrong, and it bounced as a result
(Horribly, horribly wrong being -- they told their own mail servers that accounts did not exist, who helpfully passed that info on.)
> Mail is decades old, built upon millions of hours of work crafting software, RFC standards, and works the way it does, including bounces, to ensure stability.
Same people argue that email is a hot mess exactly because it's decades old and comprised of a patchwork of standards. It's not exactly the poster child of how to do this sort of thing well.
> For a mail system to respond with 5xx, when it is internally broken, is 100% a configuration issue. Every MTA on the planet, is designed (eg, postfix, sendmail, etc) to respond with a 4xx TEMP fail message if something is borked. A milter gone bad. A library missing. A full disk. A config issue. An issue forking. Memory limits. All of it.
Configuration issues can be, and often are, temporary. Also, bugs are a thing.
> THAT is where the "good engineering" failed.
I agree with this.
> What you want to do is make 5xx like 4xx, because you feel there should never be any way for a SMTP server to say "No, really, this email address doesn't exist.. don't bother trying again".
Of course there's a semantic difference between "come back later" and "go away forever", I'm just arguing that a client that doesn't blindly trust what every connected system tells it is going to more often successfully achieve its goals than one which does.
That seems to be what you are advocating for.
You're lumping together "email content" with "SMTP". SMTP isn't a "patchwork of standards", it has a very specific RFC for it. SMTP works very, very, very well.
Configuration issues can be, and often are, temporary. Also, bugs are a thing.
This isn't just "a bug" or "a config issue", this is an edge case bug or config issue. I've handled literally hundreds of thousands of mail servers in high avail production, with some of those being extremely high volume.
What you're wanting to change the normal flow of operation, for an extremely rare edge case.
Of course there's a semantic difference between "come back later" and "go away forever", I'm just arguing that a client that doesn't blindly trust what every connected system tells it is going to more often successfully achieve its goals than one which does.
Not true here. In 99.999999% of cases when you get a 5xx response code, the correct thing to do is immediate bounce. This sort of mess, 5xx 'by accident', is insanely rare.
And it's not 'go away forever', it's "this specific mail cannot be delivered, please return it to sender, so they can examine the issue and deal with it".
It's "bring a human into the equation".
How is this an issue? For something which is very, very rare.
You apparently missed the whole point of newsletters automatically unsubscribing users on a 5xx error from the previous discussion that the topmost comment was referring to. Not that you are to blame for missing that because it wasn't restated here, but that's the context of the whole discussion. In this context I also heavily disagree with your statements and agree that among other examples automatically unsubscribing to a single "permanent" error is a brittle system.
If I have sent emails successfully to a certain email address before it is not wise to assume a permanent error on a single response that the RFC specifies as permanent. Email/SMTP is in it's core a stateless protocol, but as a mailing list service I can keep state and provide additional context to errors that can improve my service.
Finally, if you read the RFC 5321 4.2.1. Reply Code Severities and Theory it specifically uses "SHOULD NOT" instead of "MUST NOT" implying that there might be valid scenarios to act differently.
RFC 2119:
> SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.
There was no previous discussion / topmost thread, as I replied directly to the topmost comment, and this thread forks from that. No mention was made of newsletter unsubscribes thread upwards. Further, I made it quite clear I was discussing SMTP response codes, and not the article directly.
Ergo, your stated context, isn't the context of this whole thread/discussion. Further, from what I see, everyone in this thread is discussing MTAs, SMTP, SMTP return codes.
that the topmost comment was referring to. Not that you are to blame for missing that because it wasn't restated here, but that's the context of the whole discussion.
No, it isn't the context of this thread/discussion, as per above.
If I have sent emails successfully to a certain email address before it is not wise to assume a permanent error on a single response that the RFC specifies as permanent. Email/SMTP is in it's core a stateless protocol, but as a mailing list service I can keep state and provide additional context to errors that can improve my service.
In the context of the RFC, mailing list software should be viewed the same as a 'human being'. Of course it's fine to it to re-send, if the software wishes. Just like it is fine for you to send a mail with your mail client, get a 5xx, and re-try by clicking 'send' once again.
(5xx errors can happen during auth/etc stages too)
However, during this entire thread I've have been peppering the words 'MTA', "smtp server', 'bounce' and more.
Note that the mailing list software you're describing, is receiving a bounce. Bounces only happen from the MTA side. EG, mailing list software doesn't "bounce" anything, ever. Only an MTA does. Without that bounce, most mailing list software won't even know there is an issue.
Others upthread were advocating that MTAs don't bounce on an initial 5xx failure, regardless of my assertions that the client/end user should receive an immediate 5xx perm failure message.
Whether a human, or some automated software, as per my upstream statements, the proper thing to do is bounce back to the sender. 5xx, bounce, return. In 99.9999999% of cases, this is what is needed. This is the majority case.
Then, the mailing list software, the end user, can do as they wish. Including trying a resend. I don't see the conflict here, except it's apparent that a lot of people don't have much in the way of MTA experience. That's not even a knock on them, but it is a bit disheartening to see people suggesting massive MTA behaviour/RFC alteration for no reason.
To speak to mailing list behaviour, as a separate issue from above, repeated hits to 5xx targets will get you blacklisted faster than you can imagine. It's the equivalent of knocking on someone's door, them answering and saying "Sorry, Bob moved out", but you come back all day and night, banging on the door "HEY IS BOB THERE?!"
Yeah. That'll work out well. Just because you can do something, doesn't mean you even remotely should.
Just wanna point you to the context. The topmost comment was referring to his own comment and discussion about the original issue 3 days ago. You can find this here https://news.ycombinator.com/item?id=25438169
The topmost commenter also replied to you that there was that previous discussion here https://news.ycombinator.com/item?id=25473468
So while you didn't have the context, he and probably many others (like myself) did have that context. Again, this is not to blame you, because as I said it wasn't repeated in this thread and you couldn't have known.
> In the context of the RFC, mailing list software should be viewed the same as a 'human being'. Of course it's fine to it to re-send, if the software wishes. [...]
> Then, the mailing list software, the end user, can do as they wish. Including trying a resend. I don't see the conflict here,[...]
A lot of people were arguing 3 days ago that the end user (e.g. the mailing list software) should never try a resend and removing the email immediately (after the first 550 response) is correct behavior and mandaotry by the RFC.
The topmost comment here restated that this is in fact wrong. Based on what I quoted from you here, you're actually agreeing with the topmost comment you were initially disagreeing with.
Here is the relevant part from the topmost comment:
> When this event happened, I said senders need to avoid taking "invalid address" at face value when they've recently succeeded delivering to the same addresses. But despite the RFC saying senders "should not" repeat requests (rather than "must not"), many people had a lot of resistance to this idea, and instead just blamed Google for messing up implementing the RFC.
What I said, and with the context and nuance indicated in this thread, is different than what the top most comment's author said, and its replies.
This is, plain and simply, wrong. The RFC does not state that.
RFC 5321 4.2.1 Reply Code Severities and Theory
> 5yz Permanent Negative Completion reply
> The command was not accepted and the requested action did not occur. The SMTP client SHOULD NOT repeat the exact request (in the same sequence). Even some "permanent" error conditions can be corrected, so the human user may want to direct the SMTP client to reinitiate the command sequence by direct action at some point in the future (e.g., after the spelling has been changed, or the user has altered the account status)
SHOULD NOT is also clearly defined by RFC 2119:
> This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.
Understand context. The perm I state, is referencing that SMTP session, yet at the end of the session, your job is to bounce back to end user.
Else, how can the human fix things "after the spelling has been changed"? How can the human get involved without the bounce?
Having been impressed upon by that mindset, my design reflex is instead to aim for elimination of global dependencies entirely, rather than globally rate-limiting the impact of a global rate-limiter.
I’m not saying either is a right answer, but that there are consequences to being true to your philosophy. There are upsides, too, with Google’s integrated approach, notable particularly when you build end-to-end systems from public cloud service portfolios and benefit from consistency in product design, something AWS eschews in favour of sometimes radical diversity. I see these emergent properties of each as an inevitability, a kind of generalised Conway’s Law.
The infrastructure running the auth service, like any service, is going to have a quota system, whether it's global or not. The lesson learned might be different if it wasn't global ("prevent fast changes to the quota system for the auth service") but the conclusion would be substantially similar - there is usually no good reason, and plenty of danger, for routine adjustments to large infrastructure to take place in a brusque manner.
That doesn't mean that non-infrastructure service needs to abide by the same rules...
Don’t assume that is the case. That’s exactly the kind of cultural assumption I’m speaking of.
Case in point, I routinely run services without quotas or caps and what have you, scale out for load, and alarm on runaway usage, not service unavailable or quota exceeded. I’d rather take the hit than inconvenience my customers with an outage. In this frame of mind, quotas are a moral hazard, a safety barrier with a perverse disincentive.
That principle of “just run it deep” works even at global scale, right up until you run out of hardware to allocate, which is why growth logistics are a rarely-discussed but critical aspect of running a public cloud.
The core learnings become, how to factor out such situations at all. That might be through some kind of asynchronous processing (event-driven services, queues, tuplespaces), or co-operative backpressure (a la TCP/IP) and so on. Synchronous request/response state machines are absolute murder to scalability, so HTTP, especially when misappropriated as an RPC substrate, has a lot to answer for.
What I mean is, it's going to have limits of some sort, right? The world is finite...
I don't think there is any way to avoid it. It sounds bad when it's software that's telling your service it can't write to disk, rather than the disk not having any more free sectors on which to write, but it's exactly the same thing. Everyone has a quota, and left unchecked, your software will run into it.
(In the case of this postmortem, there was a bug in the software, which makes it all feel self-inflicted. But if it wasn't self-inflicted, the same problem would have manifested in some other way.)
There is a comment in this thread where the author says they take less risks when the safety systems are turned off. That is fine and nice, but is not really a good argument against safety systems. I have definitely had outages where something hit a quota I set, but I've had more confusing outages from something exhausting all physical resources, and an unrelated system failing because it happened to be nearby. I think you should wear a helmet AND ride safely.
There's a difference here; helmets are personal safety equipment, which is the proper approach: monitor and manage yourself, don't rely on external barriers. But did-you-know that a statistically significant proportion of drivers change their behaviour around riders wearing helmets? [1] (That's not a reason to not wear helmets, everyone should ATGATT; it's a reason to change driver behaviour through other incentives).
We cannot deny the existence of moral hazards. If you want to nullify a well-understand, thoroughly documented, and strongly correlated statistical behaviour, something has to replace it. Google would, apparently, prefer to cover a hard barrier with soft padding. That might help ... until the padding catches fire.
To your example, writing to disk until the OS reports "there are no more sectors to allocate" just means no-one was monitoring the disk consumption, which would be embarrassing, since that is systems administration 101. Or projecting demand rate for more storage, which is covered in 201, plus an elective of haggling with vendors, and teaching developers about log rotation, sharding, and tiered archive storage.
Actionable monitoring and active management of infrastructure beats automatic limits, every time, and I've always seen it as a sign of organisational maturity. It's the corporate equivalent of taking personal responsibility for your own safety.
[1] http://www.drianwalker.com/overtaking/overtakingprobrief.pdf
In general, AWS more often shifts the harder parts of global distributed systems onto their customers, rather than solving them for their customers, like GCP does. For example, GCP cloud storage (s3 equivalent) and datastore (nosql database) provide strongly consistent operations in multi-region configurations, while dynamodb and s3 have only eventually consistent replication across regions; and google's VPCs, message queues, console VM listings, and loadbalancers are global, while AWS's are regional.
Which of Google's nosql db provides strong consistency - bigtable? Just confirming
GCP datastore and firestore are strongly consistent nosql databases that are available in multi-region configurations [2].
[1] https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.... [2] https://cloud.google.com/datastore/docs/locations
In any case, Cloud Spanner provides strong consistency in multi-region deployments.
If anything. AWS is playing catch up.
Choice of language in representing this is rather telling, because AWS can (and does) pitch this as a strength, viz. that regionalisation helps customers (especially, significantly, bigco enterprise customers) reason about the possible failure modes, and thereby contain the blast radius of component failure.
They'd never comment on competitors in public, but the clear implication is that apparently global services merely gloss over the risks, they don't resolve them, and eventually it'll blow up in your face, or someone's face at least.
> there is no way around global dependencies
This sounds more like a challenge than an assertion. In my very long experience of tech, anyone who ever said, "you can't do that", eventually ate their hat.
A bank, for instance, may be required to prove it cannot replicate customer data across regions, and that no third party provider will replicate its data using their own BCM or DR systems.
Regardless of CSP, startups should think about rules on movement of data among data jurisdictions (such as GDPR) and architect accordingly.
If Facebook/Twitter/instagram is flaky you wait until it isn't and then post that update.
> a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0. An existing grace period on enforcing quota restrictions delayed the impact, which eventually expired, triggering automated quota systems to decrease the quota allowed for the User ID service and triggering this incident.
Grace period on enforcement of a major policy change is an excellent practice...but it also means months can go by between the introduction of a problem and when the problem actually surfaces. That can lead to increased time-to-resolution because many engineers won't have that months-old change at the front of their mind while debugging.
You need gradual rollouts. In particular, you need rollouts where the behavior of your system changes gradually as you apply your change to more of your instances/zones/whatever-rollout-unit. And the right speed is whatever speed gives you enough time to detect a problem and stop the rollout while the damage is still small enough to be "acceptable". With "acceptable" determined by the needs of your service (but if you say "no damage is ever acceptable" then I have some bad news for you).
Grace periods don't give you gradual rollouts like this; that's not their purpose. And I agree, grace periods can be a double edged sword for the reason you mention.
...
"Move fast! ...with stable infrastructure!" [1]
[1] https://www.cnet.com/news/zuckerberg-move-fast-and-break-thi...
https://static.googleusercontent.com/media/sre.google/en//st...
But yes definitely there are other things that slow down big companies.
Does anyone know if SREs in Europe fixed the problem or it relied on people in Mountain View? When an outage this big hits do devs get involved?
I've worked as an SRE and it sucks to be fixing developer's bugs in the middle of the night.
Of course in a global outage, nothing is fast enough.
The take away for me here is that maximizing resource utilization continues to be a hard problem and as you get better at it, the margin for errors is smaller and smaller.
[1] Sorry its an inside Google joke.
Did YouTube not update their authentication to a new version of the api, but they had a quota for old api calls that ran out?
"As part of an ongoing migration of the User ID Service to a new quota system"
"An ongoing migration was in effect to update this underlying configuration system"
it was not a new feature, not a massive hardware failure, it was migrating part of the working product due to some unclear reason of "best practices".
both of those migrations failed with symptoms suggesting that whoever was performing them did not have deep understanding of systems architecture or safety practices and there was no one to stop them from failing. Signs of slow degradation of engineering culture at Google. There will be more to come. Sad.
Can any single person at Google have a full understanding of all the dependencies for even a single system? I have no idea, as I've never worked there, but I would imagine that there is a lot of complexity.
- YouTube outage this November 2020
- August 2020 outage of Google Suite including Gmail
in both cases no postmortems were published
For the August outage, I believe there was a public pm. That said I can't find it now (I think there was some link rot somewhere, and I've escalated about that).
A migration is way more complex, it can touch multiple different components and at Google scale that mean different teams, it can be impossible to test this kind of migration without having a testing platforms as big as prod and if Google do it, maybe all gcp will not be enough.
This "unclear" reason can hide a bigger issue like a security bug fix, or just an important migration to go somewhere.
It look like covid19 hit everything, since it begins, attack increases a lot and security need to step up, really fast. Some manager have trouble to handle the full remote situation, some engineers have trouble too, all that combined can create small hole, so some outrage.
How many migration they do way more complex than this one without issue?
I don't think engineering culture is degrading, but shit happen and in extreme situations we see problems easier.
if this outrage was due by a feature or "massive hardware" , we can be very worrie about it
It's not malice, it's incentives and information flow. Integrating with a service that one is not responsible for, one can get tripped by unknown unknowns that the team that maintains the service has failed the document. And while a migration mistake is embarrassing, software engineering teams are generally rewarded for task completion, not for the time spent preparing for a failure that doesn't occur.
I want to push back on that. Of course, the reasons are unclear to an outsider.
Migrations are an unavoidable in any system that is still evolving (i.e. not dead). Old designs turn out to be too limited or too slow for an evolved use case, so you migrate them to a new service or a new data structure.
If you try to avoid migrations by building The Perfect Things[tm] upfront, you get lost in overengineering instead.
In my own work, I do migrations with some regularity, and they all have a clear goal, it's never what you call 'some unclear reason of "best practices"'.
https://static.googleusercontent.com/media/www.google.com/en...
It is linked from the Google Workspace status page here:
https://www.google.com/appsstatus#hl=en-GB&v=issue&sid=1&iid...
I can totally believe it. In my experience, the bigger the outage the stupider-seeming the cause.
For this kind of change I would probably have wanted some kind of shadow system that loaded the new config, received production inputs, produced responses that were monitored but discarded, and had no other observable side effects. That's such a pain in the ass that most teams aren't going to bother setting that up, even when the risks are obvious.
Seen it happen too many times.
(Presumably they wanted to make the status page depend on as few services as possible, to prevent a scenario where an outage also affects the status page itself, but whatever script they are using to publish updates to the page could also perform a check that the HTTPS version of the site is accessible, and if not, remove the redirect).
Could we get the URL of the submission updated please? (Also, it would be nice if the submission form added an "Are you sure?" step when people submit HTTP links).
Wow... how was this even possible? Did they do any testing whatsoever before migrating the live production system? They misformatting the domain name should have broken even basic functionality tests.
I wonder if they didn't actually test the literal "gmail.com" configuration, due to dev/testing environments using a different domain name? I had that problem when on my first Ruby on Rails project due to subtle differences between the development/test/production settings in config/environments/. Running "rake test" is not a substitute for an actual test of the real production system.
Now on a serious note, the increasing complexity of the systems and architectures makes it more challenging to manage and makes failures a lot harder to prevent.
The only thing post mortems show is that the company didn’t do their job or was careless or disorganized or confused. But we already know that, because they had an outage.
So what’s the point?
If problems were solved, nobody would write code. That new code or config is being deployed shows these problems (new feature rollouts, migrations, scaled resilience, etc.) are not yet formally solved. As such, things not known will become known — and usually be revealed in prod.
Similarly, in technology systems, there’s no such thing as human error, only uncaught error conditions.
Postmortems capture learning, so the conditions can be caught next time.
Publishing them shows the engineering organization understands and applies this learning loop.
For example Interactive Brokers was down completely for almost the whole trading session and they just said that they are sorry and that it was their database vendor who messed up. No more details, nothing. Just "we take the quality and resiliency of our systems very seriously".
Quotas are one way to isolate this impact to that service in particular.
Of course, when it's a critical service like authentication, it hardly isolates anything... but I can't think of a better alternative.
And that's a major PITA since I have 40-ish Gmail accounts. (Why so many? I tend to separate my different concerns for privacy and security reasons.)
On the scale of Google, you test in production, but with careful staged canary deployments. Even a 1% rollout is more than most of us have ever dealt with.
It's very unclear how they would have gone about Canary that period really, the canary should have been done on the quota side. The quota system should have enabled enforcement to 1% of the quota clients.
but it turns out it's actually hard to configure that sort of thing. The ways you can slice subsets of Google infrastructure are absolutely holographic, and it costs engineering time to change the slices.
"As part of an ongoing migration of the User ID Service to a new quota system, a change was made in October to register the User ID Service with the new quota system, but parts of the previous quota system were left in place which incorrectly reported the usage for the User ID Service as 0."
That's the sad part. That's how software becomes slower and slower with every year.