I am always astonished by how many layers these bugs actually have. It's easy to start out blaming AWS, but if anyone can realistically say they could have anticipated this type of issue at a system level, they're deluding themselves.
While none of the specific AWS systemic failures may themselves be foreseeable, it is not true that issues of this nature cannot be anticipated: the architecture of their system (and in particular, their insistence on network storage for local data) allows for cascading failure modes in which single failures blossom to systemic ones. AWS is not the only entity to have made this mistake with respect to network storage in the cloud; I, too, was fooled.[1]
We have learned this lesson the hard way, many times over: local storage should be local, even in a distributed system. So while we cannot predict the specifics of the next EBS failure, we can say with absolute certainty that there will be a next failure -- and that it will be one in which the magnitude of the system failure is far greater than the initial failing component or subsystem. With respect to network storage in the cloud, the only way to win is not to play.
[1] http://joyent.com/blog/network-storage-in-the-cloud-deliciou...
So yeah, you're right - these systemic failures can be anticipated, and Amazon's advice (to spread your important infrastructure across multiple AZs) would have protected users from most of this. (I feel quite a lot of sympathy for the engineers involved in the cross-AZ failures this incident revealed - the multi-AZ RDS and ELB failures are things customers doing everything they were told got bitten by anyway, and are probably rightly annoyed...)
IMHO, their biggest design problem is that they build their systems on top of each other (e.g. ELB is built on EBS and EIP). So when one system goes down, it takes down half a dozen others -- this is especially true of EBS, and especially dangerous because the services it takes down, like ELB, are the services people are supposed to be using to route around EBS failures.
Some services - e.g. Heroku - have lots of impacted customers but that's due to their architecture, not the underlying AWS.
June 28th 2012 - region-wide failure due to power outage and EBS dependency issues hitting ELB and RDS: http://aws.amazon.com/message/67457/
March 15th 2012 - 22-minute, region-wide networking interruption (no status report available)
April 2011 - cloudpocalypse, also a cascading EBS failure: http://aws.amazon.com/message/65648/
The other ones say “one of our Availability Zones” - which was rather my point: if you follow long-time accepted redundancy practices, you had far less - if any - downtime than people who put everything in one AZ or rely heavily on EBS volumes not failing.
What feels wrong here, is that Amazon systems are too trusting to other Amazon internal systems.
Very entertaining IMO. But I would suggest the speaker cut the off-hand ostracism of large groups of people.