Hacker News Reader

Comment by andrewguenther | Hacker News Reader

Region outages happen, which is why the guidance is to build in multiple regions, but even Amazon sometimes doesn't take their own advice. Sometimes the reason is good, sometimes it isn't.

View on HN

Our EC2 instances in us-east-1 didn't actually go down. But IT engineering was completely disrupted because our SSH login mechanism relies on the API to show you the list of instances for you to select which one to start an IAM handshake with. Our support phone line was also down. Even SQS kept chugging along just fine. I'm actually glad we're not on Lambda because of this.

I love GCP so much because of their ssh login infrastructure. It’s just beautiful.

The cli is gorgeous, the web gui is terrible.

I thought the guidance was to be multi-AZ, as an AZ is the failure boundary?

AZs are physically located near each other, usually within a small enough radius that they could be all impacted by the same natural disaster. In some cloud providers and regions, AZs are simply different parts of the same building (IIRC one of the Japan regions of Azure was essentially this, but don't quote me). And evidently, the share some infrastructure.

At a previous job where we needed to always be up, our disaster recovery plan assumed that the us-east-1 site had been hit by a meteor (not literally, but that's how we explained it to each other to put ourselves in the mindset.)

A meteor did not hit us-east-1 yesterday.

You misunderstand- our planning was to prepare for a scenario of that extremity.

AZs are physical boundaries, but the networking and software is interconnected. Regions are (mostly) isolated, though global services like IAM and CloudFront often have their main control plane in us-east-1

Software bugs recognize no boundary

Everyone wants to be multi-AZ, multi-cell, but it's a multi-year project, especially for services that have been around for a while. My last team had been working on it for a couple of years when I left.

I think it'd be correct to call the AZ a failure boundary, not the failure boundary. the is hardly the first time a failure has exceeded an AZ.

You can always do better because failures can always be bigger/wider. multi-instance < multi-AZ < multi-region < multiple vendors

Multi-AZ deployment and multi-region failover is considered best practice.

AZs are in one physical location. If that physical location has a problem, all AZs within it will go down.

>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region... AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

https://aws.amazon.com/about-aws/global-infrastructure/regio...

Reply on news.ycombinator.com