Hacker News Reader

Comment by jedberg | Hacker News Reader

The service was operating normally, you just weren't architected to handle a common failure case. Don't blame Amazon, they warned you in advance. They specifically say that they don't consider it a problem if the problem is isolated to one zone, because your app should be able to work across all zones in a region seamlessly.

I'll be the first to tell you that their dashboard underestimates impact, but in this case it was totally accurate.

The system was working normally -- ie. you could still use us-west-1 without issue if you were in another zone.

View on HN

I don't think this is a fair use of the word "normally".

The service may have been working within SLA. Based on the data in this article, it was not working the way it works most of the time. I think most people would interpret "normally" to refer to the latter.

Or, let's drop the word "normally" and take a step back. A good dashboard should be designed to inform. Amazon's dashboard was not providing the information that Ably or their customers needed. And this isn't an isolated complaint; if I had a nickel for every time someone had complained about the green mockery of the AWS health dashboard, I'd have... well, several dollars, anyway.

I might agree that "most people" out of the general population would go for the latter, but "most engineers" would not. As an engineer, 90% of my job description is handling the cases where it's not working the way it works most of the time. The case where everything works when all dependencies are doing what they normally do is a prototype or an MVP.

Now, as a founder, my job is to build that prototype & MVP, but I'd expect that as a CEO it would be to hire people to handle the other 90% of cases.

I would tend to agree. This quote from the epilogue drove home that the author doesn't think that's reasonable though:

> However, I still do not believe that if one out of three availability zones in a region is practically offline that is reasonable to report that region as healthy.

I can agree that it should be easier to see and understand that there are zonal issues, but I can't agree that the region was unhealthy. Now if 2 out of 3 zones were down, that's a real concern (you can't run a highly-available service anymore).

Said another way: what if there were 20 zones in the region and only one were down?

Disclosure: I work on Google Cloud.

I've had this question for a while: do we know there are only 2-3 zones per region, or might there be 10-20 and, for example, my zone 'a' is your zone 'c', and neither of us has access to the other's zone 'b' (yours too old/loaded to be assigned to me; mine too new for you)?

Sort of. The "zones" are just logical constructs. They can be one building or many tied together with fat fiber links. And you're right that my "A" and your "A" may be different. They are randomly assigned the first time you launch a resource.

Also, there are decommissioned zones that new accounts can't see but old ones can if they've ever launched something there before (if they haven't then that zone letter will be reassigned to a new zone with no one the wiser). Typically they won't let you launch new things in that zone, so it's in your bet interest to move out of it, lest your resources get out of balance.

Edit: Oh and old accounts can always see new zones as far as I know. They just show up as new letters. But it's rare that they add new zones -- usually they just make old zones bigger.

Right. In some set of slides, James Hamilton has certainly said at least how many buildings there are for us-east-1 and it's a lot more than the up to four zones people see (IIRC, >26). Most customers will never notice the randomization of building => zone, but every once in a while you rely on a vendor like MongoLabs and its important you both are speaking about the same us-east1-b. The AWS folks let you tie yourself together in case you have multiple projects that need identical mappings.

Us-east actually has five zone, but one of them was decommissioned many years ago, so unless your account is very old, you can only see four.

As far as I know, that's the only zone that's ever been fully decommissioned.

Reply on news.ycombinator.com