Hacker News Reader

Comment by JCM9 | Hacker News Reader

AWS didn’t “go down”. They had an outage in one AZ, which is why there are multiple AZs in each region. If your app went down then you should be blaming your developers on this one, not AWS. Those having issues are discovering gaps in their HA designs.

Obviously it’s not good for an AZ to go down but it does happen and why any production workload should be architected to have seamless failover and recover to other AZs, typically by just dropping nodes in the down AZ.

People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

View on HN

AWS is under-reporting the severity of the issue though. The primary outage may be in a single AZ, but there are parts of the AWS stack that affected all AZs in us-east-1, and potentially other regions as well. For example, even now I'm unable to create a new ElastiCache cluster in different AZs of us-east-1.

> I'm unable to create a new ElastiCache cluster in different AZs of us-east-1

Isn't that because Elasticache will distribute the cluster across AZs automatically?

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...

In this case, this was specifically with a single-AZ setup, using an AZ that was supposed to be unaffected.

Issues are across all us-east 1, not one AZ.

Load balancers are not doing well at all. The only way in this case to avoid an outage is to be cross regions or cross cloud which is quite more complex to handle and require more resources to do well.

And I hope that nobody is listening your blaming and pointing fingers advice, that's the worst way to solve anything.

It's AWS job to ensure that things are reliable, that there is redundancy and that multi-AZ infra should be safe enough. The amount of issues in US-EAST-1 lately is really worrying.

Some load balancers may be having issues but I have multiple busy workloads showing no issues all morning. One big challenge can be that some people reporting multi-AZ issues are shifting traffic and competing with everyone else, while workloads which were already running in the other AZs were fine. It can be really hard to accurately tell how much the problems you’re seeing generalize to everyone else.

I do agree that the end of this year has been a very bad period for AWS. I wonder whether there’s a connection to the pandemic conditions and the current job market – it feels like a lot of teams are running without much slack capacity, which could lead to both mistakes and longer recovery times.

I hope AWS will provide some explanation about those issues and what actions they will take to prevent those in the future

On our side we saw some EC2 VM totally disconnected from the network in 3 AZs.

Yeah, definitely needs good visibility. They’re asking customers to trust them to a large degree.

But if AWS can't handle the shifting traffic in response to an AZ going down then at that point you're just gambling on whether or not you're in the lucky AZ.

Echoing this. We had to manually intervene and cut off the faulty AZ because our ASGs kept spinning up instances in it and our load balancers kept sending traffic to bad hosts.

In the past I've seen both of those systems seamlessly handle an AZ failure. Today was different.

> People commenting that servers shouldn’t go down ect don’t understand how true HA architectures work. You should expect and build for stuff to fail like this. Otherwise it’s like complaining that you lost data because a disk failed. Disks fail… build architecture where that won’t take you down.

Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen. These entire datacenter failures should be anticipated, but to expect them is a bit too easy I think. There are plenty of hosters who don't have this stuff even once for the last decade in their only datacenter. I do not find it strange to expect or even demand that level but to protect yourself if it happens in any case if that fits your specific project and budget.

Edit; OK meant that raid-5 remark in the same context as the hosting; it can and does happen but it shouldn't; you should plan for a contingency but expect it goes far. We never had it (1000s of hard-drive, decades of hosting, millions of sites) and so we plan for it with backups; if it happens it will take some downtime but it costs next to nothing over time to do that. If we expected it, we would need to take far different measures. And we had less downtime in a decade than aws AZ had in the past months. I have a problem with the word 'expect'.

> Is that comparison fair? If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen.

There are plenty of situations where this might happen if they’re in your room: a lightning strike can cause a surge that causes the disks to fry, a thief might break in and steal your system, your house might burn down, an earthquake could cause your disks to crash, a flood could destroy the machines, and a sinkhole could open up and swallow your house. You may laugh at some of these as being improbable, but I have seen _all_ of these take out systems between my times in Florida (lightning, thief, sinkhole, and flood) and California (earthquake and house fire).

The fix for this is the same fix as being proposed by the parent post - putting physical space between the two systems so if one place become unavailable you still have a backup.

I have had a job where my small, internal tool, for debugging purposes, had to be deployed to a minimum of 3 datacenters. I had 2 of them in the US and one in Europe, and was asked to move one of the US ones to a datacenter that was in another coast, cause who knows, maybe an earthquake will knock off all of the US west coast. That is the paranoia level necessary to achieve crazy high uptime.

> If you have 2 raid-5 mirrored raid 5 boxes in your room and all disks fail at the same time, you should complain. And that won't happen.

Here are some examples where that happened:

1. Drive manufacturer had a hardware issue affecting a certain production batch, causing failures pretty reliably after a certain number of power-on hours. A friend learned the hard way that his request to have mixed drives in his RAID array wasn’t followed.

2. AC issues showed a problem with airflow, causing one row to get enough warmer that faults were happening faster than the RAID rebuild time.

3. UPS took out a couple racks by cycling power off and on repeatedly until the hardware failed.

No, these aren’t common but they were very hard to recover from because even if some of the drives were usable you couldn’t trust them. One interesting dynamic of the public clouds are that you tend to have better bounds on the maximum outage duration, which is an interesting trade off compared to several incidents I’ve seen where the downtime stretched into weeks due to replacement delays or manual rebuild processes.

More generally, any correlation between two items gives potential for a correlated failure.

Same manufacturer, same disk space, same location, same operator, same maintenance schedule, same legal jurisdiction, same planet, you name it, and there's a common failure to match

"Won't happen". The 40,000 hours of runtime bug did happen. I would recommend people to take backups and store them offline or at least isolated from the main storage.

Sure, I plan for it but I do not expect it. And it never happened for me over decades. But I did plan for it, just not in the way the parent said and it did cost me far less.

Good to know ;) still think I am lucky; have had very little harddisks fail across 1000s. And only very few unrecoverable (had to restore from backup) failures; those weren't HD failures but software failures; hds were fine.

>And that won't happen

HA! I had received new 16-bay chasis and all of the drives needed plus cold spares for each chasis. Set them up and started the RAID-5 init on a Friday. Left them running in the rack over the weekend. Returned on Monday to find multiple drives in each chasis had failed. Even with dedicated one of the 16 drives as a hot swap, the volumes would all have failed in an unrecoverable manner.

All drives were purchased at the same time, and happened to all come from a single batch from the manufacture. The manufacture confirmed this via serial numbers, and admitted they had an issue during production. All drives were replaced and at a larger volume size.

TL;DR: Drives will fail, and manufacturing issues happend. Don't buy all of your drives in an array from the same batch! It will happen. To say it won't is just pure inexeperience.

Guess I was lucky, we ran a lot of these over the decades when things were far more unreliable than now and never experienced anything like that. Manufacturing issues, sure, but we always had everything we bought run on stress for 48 hours and see if that killed it, if it didn't, it didn't usually break anymore (I have many of the machines from mid to end 2000s still and they don't have diskfailures now while they ran for many years).

The older hdds imo are more reliable than the newer ones. They're pushing for density, and mitigating the inheritly higher sensitivity through aggressive error correction.

Would like to know the manufacturer and model.

Sorry, this was back in 2006-2007 time frame. I have no idea on model numbers as that's just not information I ever cared to commit to memory.

My time frame is about since end 90s to now. I saw more failures in general in the olden days. I have quite a lot of rented servers at traditional hosters or Colo that have not been down outside kernel security updates for 10+ years. No hardware issues. I am now swapping them out for new servers which cost less for more, but hardware wise they have been pounder hard for 10 years without issues. All hotswap raid5 drives, so when broken, they or us just fixed it without downtime.

>AWS didn’t “go down”

The context of the parent seems to be that they intermittently couldn't get to the console. That seems fair to me. If we're blaming developers and finding gaps in HA design, then AWS should also figure out how to make the console url resilient. If it's not, then AWS does appear to be down.

I imagine it's pretty hard to design around these failures, because it's not always clear what to do. You would think, for example, that load balancers would work properly during this outage. They aren't. Or that you could deploy an Elasticache cluster to the remaining AZs. You can't. And I imagine the problems vary based on the AWS outage type.

Similarly, with the earlier broad us-east-1 outage, you couldn't update Route53 records. I don't think that was known beforehand by everyone that uses AWS. You can imagine changing DNS records might be useful during an outage.

Except many AWS services still route through us-east-1 anyway, which is why they have had huge outages recently. AWS isn't as redundant as people think it is.

Our API is just appsync (graphql) + lambdas + dynamoDB so, theoretically, we shouldn't have been affected. But about 1 in 3 requests was just hanging and timing out.

As others have said, they are not being forthright about the severity of the issue, as is standard.

100% agree. I'm actually surprised AWS hasn't built in a Chaos Monkey into their APIs/console so people can test their resiliency regularly if an AZ goes down.

edit: of course, AWS does have this: AWS Fault Injection Simulator

AWS Fault Injection Simulator does this.

Is that what they call us-east-1 nowadays?

us-east-1 has always been the canary region hasn't it?

TIL. Thank you!

Because then people would complain about AWS being less reliable than Azure / GCP.

Here's a secret that's now saved me from three outages this month:

Be in multiple AZs, and even multiple regions but if you're going to be in only one AZ or one region, make it us-east-2.

Reply on news.ycombinator.com