Hacker News Reader

Comment by mml | Hacker News Reader

Did I read this correctly in paragraph 2: " For two periods during the first day of the issue, the degraded EBS cluster affected the EBS APIs and caused high error rates and latencies for EBS calls to these APIs across the entire US East Region."

Their "control plane" network for the EBS clusters span availability zones in a region? If so, this would be the fatal flaw.

View on HN

I may have read it incorrectly myself, but I interpreted this as meaning the control plane was balanced across availability zones in order to provide durability in the face of a failure of one of the zones. In other words, Amazon is ensured their control plane is operational at all times.

The API failures were ultimately tied to the network problems that occurred, not to a failure of the control plane.

EDIT: I should finish reading before I reply. :) It would appear that the network issue in the one availability zone was so severe that the control plane ran out of threads to service API requests to any of the availability zones.

So while it's true the underlying problem was a network issue, the fact that the the control plane is spread across availability zones was responsible for part of the outage that occurred across the whole region.

My totally unqualified assessment of this aspect of the outage is that, while it might make sense to have a control plane spread across availability zones, they presumably need to have isolated control planes for each zone, instead of a shared plane as they seemingly have now.

They seem to have settled on a halfway house, pushing more of the control plane functionality down into the EBS clusters and making the remaining shared control plane more robust to the sort of failures that arose this time.

And a breach of the "share nothing" tenet, which is quite important here.

I don't see how transparently replicated storage could be implemented without sharing something.

In which case they shouldn't replicate across availability zones.

'shared nothing' is the only way to islandize failures.

And that's why they have multiple fully-isolated regions. Availability zones are a purposeful tradeoff that provides easier to use service with higher inter-zone communication performance and lower cost.

No, that's not what Amazon says.

The following is from the AWS web site [1]:

> Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region. By launching instances in separate Availability Zones, you can protect your applications from failure of a single location.

No mention of tradeoffs.

[1] http://aws.amazon.com/ec2/

The tradeoff is in that AZs are "engineered to be insulated" as opposed to being actually, or naturally isolated. Prior to their downtime, I've had plenty of conversations with folks that I work with about AWS and we've always assumed that AZs are not 100% isolated. I can see how someone can read "engineered to be insulated" the other way, but I generally read these kinds of materials as guaranteeing nothing beyond the most limited possible reading, and probably not even that.

The quoted statement doesn't say that isolation is 100% or that multiple AZs can't ever ever fail at the same time. It says that if only one AZ goes down and you have servers in another, then those servers will still be up, which should be obvious. Insulated doesn't even mean the same thing as isolated.

Sure, I don't disagree with what you are saying. However I think that the way Amazon presents the concept of an AZ is that it IS isolated from other AZ's in the same region.

Even the name, 'Availability Zone' implies that it is isolated from other 'Availability Zones' in the same region. And that text I quoted does nothing but substantiate that inference.

I just think that Amazon are misleading here. Maybe they shouldn't call it an Availability Zone.

The transparent replication is inside the availability zone; as I understand it, Amazon doesn't provide any sort of user visible direct sharing between multiple AZs, e.g. to copy or move an EBS you have to snapshot it first ... which was of course a control API they blocked during much of this mess.

Isn't this "flaw" a balance of features vs reliability? In most cases it's ok that the API spans the entire region, and makes it easy to address one API endpoint per region, allows Amazon to offer everyone different availibility zones within the region, etc.

The issue being that availability zones don't get throttled when they start to overload the API. The control system should have automatically throttled the misbehaving AV.

Which is probably far more difficult to do properly than I can imagine.

To me, it sounds like a large single point of failure, and the post-mortem doesn't seem to acknowledge that or discuss remedying it.

They setup a separate instance of it to help with API calls in the affected region, but it still sounds like it functions across AZs and is still vulnerable overall.

Uhh, did you even read it? Under "Impact to Multiple Availability Zones", last paragraph:

"There are three things we will do to prevent a single Availability Zone from impacting the EBS control plane across multiple Availability Zones. The first is that we will immediately improve our timeout logic to prevent thread exhaustion when a single Availability Zone cluster is taking too long to process requests. … To address the cause of the second API impact, we will also add the ability for our EBS control plane to be more Availability Zone aware and shed load intelligently when it is over capacity. … Additionally, we also see an opportunity to push more of our EBS control plane into per-EBS cluster services. By moving more functionality out of the EBS control plane and creating per-EBS cluster deployments of these services (which run in the same Availability Zone as the EBS cluster they are supporting), we can provide even better Availability Zone isolation for the EBS control plane"

Reply on news.ycombinator.com