Hacker News Reader

Comment by LAC-Tech | Hacker News Reader

I thought the whole point of AWS that it was a distributed system with no one point of failure? In that case, how did it have an outage?

View on HN

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

Multi-AZ deployment and multi-region failover is considered best practice.

AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure.

At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)

A meteor did not hit us-east-1 yesterday.

You misunderstand- our planning was to prepare for a scenario of that extremity.

Software bugs recognize no boundary

AZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1

Everyone wants to be multi-AZ, multi-cell, but it's a multi-year project, especially for services that have been around for a while. My last team had been working on it for a couple of years when I left.

You can always do better because failures can always be bigger/wider. multi-instance < multi-AZ < multi-region < multiple vendors

AZs are in one physical location. If that physical location has a problem, all AZs within it will go down.

>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region... AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

https://aws.amazon.com/about-aws/global-infrastructure/regio...

I think it'd be correct to call the AZ a failure boundary, not the failure boundary. the is hardly the first time a failure has exceeded an AZ.

Our EC2 instances in us-east-1 didn't actually go down. But IT engineering was completely disrupted because our SSH login mechanism relies on the API to show you the list of instances for you to select which one to start an IAM handshake with. Our support phone line was also down. Even SQS kept chugging along just fine. I'm actually glad we're not on Lambda because of this.

I love GCP so much because of their ssh login infrastructure. It’s just beautiful.

The cli is gorgeous, the web gui is terrible.

do they not teach people what a failure domain is anymore?

https://en.wikipedia.org/wiki/Failure_domain

Clouds aren't magic. They require a certain amount of operational confidence in order to understand that, yes, an entire region can fall out from under you at any time and it's your responsibility to detect and deploy into an unaffected region if possible.

edit: Generally, one entire region will not fail. However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.

In my experience, once an item is done done done, it’s out of your hands.

Honestly, no. Not frequently enough.

There are also way too many successful, public cloud-native businesses running without any semblance of a Business Continuity or DR Plan among any of their teams. I wish I could name and shame some of the more egregious cases I have seen.

> do they not teach people what a failure domain is anymore?

Empirically? No.

But seriously, instead of making every dev team consuming an AWS product hire the extra engineers required to build a system that spans multiple failure domains — which lets be honest, companies won't — why hasn't Amazon just hired the engineers required to do it for me?

> Generally, one entire region will not fail.

Laughs in global failures.

People get slagged on for not having inter-region redundancy. But unless your business model can accommodate that, you’re introducing another failure domain and alot of money for duplicative infrastructure, network fees, etc.

For many use cases, it’s acceptable to shrug and blame AWS for a failure. It’s harder when your high availability solution fails independently, which they almost always do more than US-East-1

If your site goes down while everyone else's goes down, it's weirdly forgivable.

And being the only one up doesn't win as many market cred points and you'd think.

Exactly. Often people want the perception of reliability.

For many services, it makes more sense to make it reliable than not. For other services, it makes more sense to think about the engineering of the solution in the field.

Example: McDonald’s product images on kiosks are apparently in S3 and not cached locally. Seems like a dumb idea to me, but I wouldn’t try to build a more reliable cloud storage backend to control that risk.

> However, core services like STS rely on us-east-1 so it's particularly susceptible to disruption.

FYI this example hasn’t been true for a while. STS regional endpoints are generally what you should be using these days. The “global” us-east-1 endpoint still works, and may be the default for some clients, but isn’t a requirement.

https://status.aws.amazon.com/ says that multiple network devices failed.

They very carefully did not actually say that. The weasel wording used could be a routing configuration error, among other things.

A lot of AWS is very old and it's very hard to go back and make everything nice and reliable without breaking things when you have a shitload of customers 24/7 using your services.

So yes, in theory much of AWS's services are probably very reliable and distributed across AZs and regions, but in practice there's likely a whole bunch of debt where one thing gets fucked up and it cascades.

A profitable "accident".

As an AWS employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.

As a corporate employee, you’re under too much artificial deadline pressure to address technical debt like this… let alone engineer a service properly in the first place.

I guess on a load balancer or DNS level? A request has to hit a single domain name / IP before it's load balanced to the distributed system right?

Multiple devices in very different parts of the world can respond to the same IP address. A great example are DNS servers like 1.1.1.1 and 8.8.8.8- you get routed to the nearest site that can handle your request. A common example of how this is implemented is via anycast: https://www.cloudflare.com/learning/cdn/glossary/anycast-net...

Reply on news.ycombinator.com