Hacker News Reader

Comment by Hippocrates | Hacker News Reader

Every time a major cloud provider has an outage, Infra people and execs cry foul and say we need to move to <the other one>. But does anyone really have an objective measure of how clouds stack up reliability-wise? I doubt it, since outages and their effects are nuanced. The other move is that they want to go multi-cloud... But I’ve been involved in enough multi-cloud initiatives to know how much time and effort those soak up, not to mention the overhead costs of maintaining two sets of infra sub-optimally. I would say that for most businesses, these costs far exceed that occasional six-hour-long outage.

View on HN

I mean from the explanation[0], assuming that is correct (I don't have evidence to suggest it's false) - you don't need to be multi-cloud, and you don't even need to be multi-region. As long as you're spread out over multiple availability zones in a region you should be resilient to this failure.

Somewhat surprising to see how many things are failing though, which implies, either that a lot of services aren't able to fail-over to a different availability zone, or there is something else going wrong.

[0] https://news.ycombinator.com/item?id=29648992

AWS doesn't follow their own advice about hosting multi-regional so every time us-east-1 has significant issues pretty much every AZ and region is affected.

Specifically large parts of the management API, and IAM service are seemingly centrally hosted in us-east-1. So called Global endpoints are also dependent on us-east-1 and parts of AWS' internal event queues (eg. event bridge triggers)

If your infrastructure is static you'll largely avoid the fallout, but if you rely on API calls or dynamically created resources you can get caught in the blast regardless of region

Your last comment is really important, I think. I have always petitioned for "passive over active" design in distributed cloud systems. The recent outages, and also ones from the past, demonstrate why.

The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better. Related to that, I have been critical of lambda/FaaS/serverless infrastructure patterns for similar reasons. Always felt like a brittle house of cards to me (N.B. I do still use aws lambda, but keep it constrained to non-critical workloads).

> The fewer API calls you need to make in-band with whatever throughput is generated via your customer demand, the better.

Agreed; however, this is somewhat difficult to do correctly. There are all sorts of systems that might have hidden dependencies on managed services. e.g. AWS IAM roles will almost always be checked at some point if your services need to interact with AWS managed services.

I think cloud providers could meet developers half way here, by providing ways to reduce API usage; but I'm not sure if it aligns with their incentives.

AWS IAM is designed with a control plane / data plane dichotomy. Even if the control plane is completely dead and all API requests are failing, services in a steady-state (i.e. not responding to changes via API calls) can still rely on IAM roles using their cached information. For example, in the recent us-east-1 outage when you couldn't start new services because IAM checks would fail, existing EC2 instances that rely on IAM instance profiles to access services like S3 could still do so even though IAM was down.

I was gonna respond with the same commentary here. That has been my experience beyond just IAM controls and why I advocate for passive systems for critical workloads.

Sometimes this _can_ be costly. For example with something like autoscaling, thats an active system I've seen fail when seemingly unrelated systems are failing. The result is scaling out systems intentionally ahead of time to deal with oversubscription or burst traffic which can leave you with (costly) idle compute.

I don't mind this tradeoff personally, but can understand that budget constraints are going to be different org to org.

It's not an AWS incentive thing really, it's a developer/consumer incentive thing.

It's like the duality of modular code. If you want to manage one change in a lot of places, it's easiest to change it in the one module that everything else sources. But that means that one change to that module can take down everything. The alternative where you copy+paste the same change everywhere is the most resilient to failure, but also the most difficult and expensive.

AWS provides a lot of modular, dynamic things because that's what their customers want to use. But using each of those things increases the probability of failure. It's up to the customer to decide how they want to design their system using the components available.... and the customers always chose the easy path rather than the resilient path.

The great thing is that with AWS, at least you have the option to design a super freaking reliable system. But ultimately there's no way to make it easy, short of a sort of "Heroku for super reliable systems". (I know there are a few, but I don't know anything about them)

I like the way you framed this, its a tradeoff mainly. You can build something fault-tolerant and highly-available, even during AWS outage events, but you have to give up a ton of the product offerings in their suite.

I've managed to stick to EC2/ELB and S3 as passive systems for the vast majority of what we build at my org (~90% of our stack). And for the most part, AWS failures are hitless for us as a result.

Yeah, my thought is not specific to this scenario. Indeed multi-AZ is a low cost and probably good idea because you often have a shared service management, control plane, and cheap bandwidth between things. Of course, when things fail they often ripple as may be the case here. I don't think clouds have their blast radius perfectly contained and they certainly don't communicate those details well.

One incident I recall was involving our GCP regional storage buckets, which we were using to achieve mutli-region redundancy. One day, both regions went down simultaneously. Google told us that the data was safe but the control plane and API for the service is global. Now I always wonder when I read about MR what that actually means...

That’s true for this failure but the prior two for AWS were region wide and the one for GCP last month was global.

Perhaps is us, the customers (and our customers, and the customers of our customers, ...), the ones who should get used to the status of "things can go wrong"? Except for some specific scenarios (medical-related stuff, for instance), if my favourite online shopping place is down, well, it's down, I'll buy later.

I know the Oracle OCI cloud has a reputation for never going hard-down, but also realize HN seems to loathe Big Red (understandably, to a degree, though OCI is pretty nice IME and _very_ predictable).

I don't think it's unfair. They aren't the worst villain, but they are up there.

> I doubt it, since outages and their effects are nuanced.

Your point here deserves highlighting. A failure such as a zone failing is nowadays a relatively simple problem to have. But cloud services do have bugs, internal limits or partial failures that are much more complex. They often require support assistance, which is where the expertise of their staff comes into play. Having a single provider that you know well and trust is better than having multiple providers where you need to keep track of disparate issues.

I agree with you. I think that having multi-AZ is the first thing to figure out before wanting to do multi-cloud, which is just another buzzword taken out of management's bullshit bucket :)

Agree, and multi AZ is usually easy. IME with AWS and GCP the control plane is the same, the scaling works across AZ, bandwidth is free and latency is near zero. The level of effort to do that is simply ticking the right boxes at setup time IME.

Cross-AZ bandwidth is far from free and the biggest reason companies avoid it (IMO). Also latency is not near zero but I don't think that's the primary reason.

That's true. The moment your data leaves the region you start paying for egress and that can get expensive quickly. Still beats the crap out of multicloud, though :)

I’ve seen at least half a dozen full region AWS issues in the past 8 months.

You really need multi-region and also not be relying on any AWS service that’s located only in us-east-1 (including everything from creating new S3 buckets to IAM’s STS).

Who says this? I have literally never once seen this.

Reply on news.ycombinator.com