Ask HN: GCP Outage?

Every time pager duty hits, take a shot

Someone unplugged the Big Router

Every ui in gcp is ugly and painful and slow. Why?

I see they made great strides in the past 5 years - it used to take days =)

One of my multi-region clients is also affected by Brazil GCP.

And also that effort(multi cloud) >>> effort(multi region)

This isn’t a whole-cloud outage. It’s not even a whole-region outage.

Whole-cloud outages are pretty damn rare. The recent GCP issues are an exception to the general rule.

I’d posit that the complexity of a multi-cloud setup is generally going to reduce your service’s reliability more than relying on a single cloud does.

I've maintained a large multi-cloud architecture in the past. The problem is they really hit you hard on egress costs. Of course the motivation is obvious, they want to keep you locked in to their vendor. I did like that it gave a stronger leverage in contract renewals, but that was about it. The IAC was much more complicated and required more people/areas of knowledge. So it's definitely a tradeoff.

You are correct that it's "better" though if your goal is to have as many 9's of uptime as possible.

I worked at PagerDuty, so definitely not selling availability theater. We did multi-cloud / multi-region for many years, and the story is not so simple. Development is all about trade-offs, and deciding what risk you are OK with. Multi-cloud provided a relatively small amount of value (given how incredibly unlikely whole-cloud outages are, even full-region outages are quite rare) at the expense of 2x implementation overhead, 2x exposure to random cloud-specific operational events, and the need to develop for the common denominator of functionality, which leaves out a LOT of interesting cloud offerings. In the end, it ended up just not being worth it, and moving to the single-cloud multi-region config provided enough reliability even for the company where reliability is the primary differentiator.

In my current job as a technical due diligence advisor, I frequently recommend multi-AZ setup but specifically not multi-region, because the former is easy and worthwhile while the latter carries a lot more operational overhead (you become much more sensitive to various latencies and network jitters) and you now need to think about things like synchronous vs async replication, etc. Much better to focus dev effort on the product, rather than eke out an additional .001% of availability (unless availability is a super critical component).

I dunno. If just your employers site is down then you'll be expected to fix it, whereas if everyone is down there's less pressure.