That is a good point about EC2 being a black box. When something really slows down (SimpleDB queries, network access, etc.) it takes effort to track down the problem(s).
I think that AWS is a good resource for scaling up non-realtime calculations like map reduce, but may not be a great choice for 24x7 hosting.
How is this any different from a large corporation? Having been on both sides, I'd argue that Amazon's mean time to response/resolution is as good as or better than what is generally seen in silo'd enterprise shops (provided you pay for AWS support- strongly recommended for those with mission-critical cloud infrastructure).
That's a good point Mark. Systems that aren't as sensitive to latency won't be affected as much by these types of issues. When you begin to require a certain level of network performance the lowest-common-denominator network configuration that cloud hosting offers may not be sufficient.
How much traffic was each LB seeing? The use of round-robin DNS does not always result in an even distribution of traffic.
The LB that was performing well was getting more traffic. I initially wanted to break out response times by LB because I thought maybe that was the underperforming machine, but my intuition was wrong and the opposite was true.