Hacker News Reader

Comment by snewman | Hacker News Reader

> Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.

Exactly. It seems likely that Amazon tests the restart operation, but it would be hard to test it at full us-east-1 scale. Running a full S3 test cluster at that scale would likely be a prohibitive expense. Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it? Also, hindsight is 20/20, but before this incident it might have been hard to identify "full-scale restart of the index subsystem" as rising to the top of the list of things to test.

One approach is to try to extrapolate from smaller-scale tests. It would be interesting to know what kinds of disaster testing Amazon does do, and at what scale, and whether a careful reading could have predicted this outcome.

View on HN

> Failure at every level has to be simulated pretty often to understand how to handle it

Keep in mind, S3 "fails" all the time. We regularly make millions of S3 requests at my work. Usually we get 1:240K failure rate (mostly GETs), returning 500 errors. However, if you're really hammering an S3 node in the hash ring (e.g. Spark job), we see failures in the 1/10K range, including SocketExceptions, where the routed IP is dead.

You need to always expect such services to die in your code, setting the proper timeouts, backoffs, retries, queues, and dead letter queues.

If you retry the get does it succeed usually?

Yes, retry has always worked (*us-west-2).

Sometimes it's a 404 for an object written 1 sec prior, other times it's an S3 node that died mid request. Retry gets you to a different node.

> Perhaps the "index subsystem" and "placement subsystem" are small enough for full-scale tests to be tractable, but certainly not cheap, and how often do you run it?

Rough guide:

CT = cost of 1 full scale test with necessary infrastructure and labor costs added up

CF = amount of money paid out in SLA claims + subjective estimate of business lost due to reputation damage etc

PF = estimate of probability of this event happening in a given year

if PF * CF > CT, then you run such a test at least once a year. Think of such an expense as an insurance premium.

What Netflix does with their simian army is amortize the cost of doing the test across millions of tests per year and the extra design complications arising from having to deal with failures that often.

This is precisely why cells (alluded to in the write-up) are beneficial. If the size of a cell is bounded and you scale by adding more cells, testing the breaking point of the largest cell becomes an easier problem. There is still usually a layer that spans across all cell boundaries, which is what then becomes hard to test at prod scale (so you make that as simple as possible)

Testing a full zone test is only possible when they have a new zone available, unused. I bet they do these test, and they now have a new scenario to test.

They also probably have one or more test regions where they could perform a test like this. But it's presumably not at nearly the same scale as us-east-1, the region affected by this incident. And to a considerable extent the problem was one of scale. The writeup makes the recovery sound fairly straightforward; but due to the sheer size of S3 in this region, it took hours for the system to come back up, which was apparently unexpected.

(Nit: this incident affected a region, not a zone. us-east-1 is a region, which is divided into zones us-east-1a, us-east-1b, etc. S3 operates on regions.)

Reply on news.ycombinator.com