Hacker News Reader

Comment by shanemhansen | Hacker News Reader

> On Monday 11 March 2019, Google SREs were alerted to a significant increase in storage resources for metadata used by the internal blob service. On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure.

Just some guesses (not affiliated):

I wonder if there was something related to Tombstones and deletion of extra data? That can often be a failure of replicated sharded big data systems using sstables and tombstones for deletion.

Another guess might be that some sort of internal shard count was increased resulting in some lookup process needing to check n more shards per call.

View on HN

The root cause of a cascading failure can be really minor - just something tiny that tips the system over the point of no return.

The real question is why don't us systems-designers have better tests for cascading failure? Even Google has very few tests which deliberately overload a system (to cause it to fail), and then verify that it can recover on it's own promptly under typical loads.

Eg. consider a database which serves a web application. If the application times out querying the database, it retries up to 5 times. That is a system with an unrecoverable cascading failure, since if one day we get a small spike in users, the database becomes slow, and a few timeouts occur, some requests will get retried, putting more load on the database, making it slower, causing more timeouts, and the problem gets rapidly worse until a few minutes later all requests are failing. Even when the spike in users is over, the database will remain effectively dead, because it is still failing under load from all those retries.

A load test that didn't push the system over the edge and then test for recovery wouldn't be able to detect that.

Too many people loadtest things to the point of failure and then stop. All it takes is to continue the test for a few minutes longer with a slightly lower load to verify recovery!

Because there are unlimited number of failure modes for cascading failures. Going through all postmortems, you'll find it hard to find same kind of failures. Every time it's something unique.

Also it's difficult to get a good coverage by just pushing things to its limit. Let's say you have a system which has this RPC chain: A -> B -> C -> D. And not all requests cost the same and not all requests make way to D. In reality it's interesting that different composition of traffic may overload any of them. For example, some queries may hit expensive ML models while some other queries have a large number of candidates. In this case the former hits predictor service hard while the latter may hit index serving systems. I've actually read an outage happened during superbowl because sports-related queries are particularly expensive for some part of the system.

Reply on news.ycombinator.com