* Would this have been less painful with Netflix-esque patterns? Circuit breakers, throttling and the like? Failing hard on purpose is often easier to diagnose than failing fuzzily without clear cause.
* Would it be acceptable to maintain spare capacity? For a prod cluster on the main money path, I think everyone thinks yes. Who holds the decision bit for dev, which is a second-order source of money?
* Can I avoid the second act by having a clean way to (1) reset the entire system and (2) bring it up in an orderly fashion, as quickly as possible? Not every app needs to be relaunched simultaneously, there's presumably _some_ order of preference. Once the cluster state began to oscillate it doesn't seem like there were any easy options other than resetting it.