The lesson is: don't build a system that requires or advertizes high avilability but that runs in a single datacenter. Load balance. Elect leaders. Batch, queue, and distribute. Fail-over. Fail fast. So long as we have earthquakes, lightning, human malice, and human error, single AZs will fail.
Remember the old saying about "Lies, damn lies, and availability zones"?
We're paying 2x for the RDS multi-AZ setup that is supposed to failover in these cases. So far no luck. Think I'll ask for a refund.
Same here! Even trying to boot up some replicas is painfully slow.
I gave up on Amazon (and ate the reserved instance fees) 2+ years ago over this. I was paying through the nose for multi-AZ RDS and when EBS failed in one zone, nothing failed over.