Hacker News Reader

Comment by wizwit999 | Hacker News Reader

So the bigger news here is GCP had a global outage. How'd that happen?

It mentions a pipeline deployment introduced a bug (https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ...) but why weren't they doing staged deployments to region(s) at a time?

View on HN

This is now 'par for the course' with GCP.

A recent example, Firefox's Jan 2022 outage was due to an unannounced change deployed by GCP to their load balancers.

https://hacks.mozilla.org/2022/02/retrospective-and-technica...

I've read several GCP related stories over the years so my opinion of them has fallen quite a bit. For some reason GCP does not get the flak they deserve, and their role in outages tend to be downplayed, while AWS and Azure are scrutinized and raked over the coals for smaller incidents.

They don’t get flak because most folks are in AWS or Azure. You notice when AWS or Azure hiccups because of this. GCP, not so much (besides Spotify and Snap, although I believe Snap’s 5yr $2B GCP commit deal recently ended).

https://info.flexera.com/CM-REPORT-State-of-the-Cloud

https://thenewstack.io/long-running-study-finds-uptake-of-pr...

Ah, good point about higher usage therefore flak, I hadn't considered that. Thanks.

Bjarne Stroustrup’s classic quote “There are only two kinds of languages: the ones people complain about and the ones nobody uses” certainly applies here.

Worth noting that Snap also has a commitment with AWS, they are not exclusively running on GCP.

Also worth noting that AWS is not exactly the most stable either.

>unannounced change deployed by GCP to their load balancers

Google doesn't need to announce every change they make. It was Firefox's fault for not supporting the protocol correctly.

Most outages have more than one cause. It’s true that Firefox had a nasty bug but it’s also likely that they’d have resolved the problem faster had a clear notification that GCP had upgraded their infrastructure right before the outage started, especially if that had an easy rollback option.

Changing a default behaviour, yes they do. If you feel they don't, then you might be happy with GCP's services. It does not inspire any confidence with me.

It was known in advance that HTTP/3 support would eventually be enabled for the load balancers. If you wanted to force support for it or force it to not support it there was an override setting. Mozilla did not override the behaviour meaning that they did not mind HTTP/3 to be advertised as being supported.

so this is my "worked in aws" informed take, but you absolutely do not force default behavior change on anyone unless it is absolutely necessary, and advertising http/3 isn't necessary.

AWS had a similar oopsie with their Load Balancer: introduced a new flag to strip some headers, enabled it by default for all existing public ALBs, and the effect was that all custom headers with an underscore in the name (“x-access_token”) started getting stripped. Took down a few APIs https://forums.aws.amazon.com/thread.jspa?messageID=923182

Traffic Directors config is global. I assume it was some kind of config system bug.

Reply on news.ycombinator.com