Hacker News Reader

Comment by louisgoddard | Hacker News Reader

Did a quick write-up on this here: http://webdev360.com/users-report-major-amazon-ec2-outage-bu...

Does anyone know if ops teams generally have to clear these sort of announcements with PR/comms? I assume they would, given how they get reported on.

View on HN

At a competent company, no. There are policies in place before the outage, but having your PR people in the loop slows things down to the point where you're worthless to your customers. The exception is you loop in legal, PR, etc. if someone is actually injured/dies, or if crimes are involved.

A lot of providers try to NDA their "ops to customer" service outage notifications, but most customers flagrantly violate those NDAs. Automated service dashboards are supposed to be automatic; ops teams often put in short statements (especially time to fix and any interim way to mitigate the outage).

Definitive statements after the outage are run by PR (and generally announced senior to ops), but service notifications of outages (vs. causes, compensation, and long term corrective actions) are not.

Thanks -- that's what I assumed initially, but had second thoughts after seeing how delayed the announcements were.

I suspect there is some human admin level disconnect between their network ops/routing and the AWS team itself. A connectivity outage wouldn't necessarily get detected within AWS, and they probably don't have good monitoring within the AWS product to detect problems like that. The network team presumably doesn't have a good way to push status updates to the AWS dashboard automatically (and it's kind of a grey area what is a "network outage" -- if you lost routes to just Pakistan, that's not really a big deal for most AWS customers. If you lose routes to everyone, yes, that's a big deal.)

Somewhat ironically, I can't get to that site.

I think our servers were down for a few minutes earlier -- should work now.

Reply on news.ycombinator.com