Hacker News Reader

Comment by willio58 | Hacker News Reader

It's crazy to me how long it takes Amazon to notify users that there is even an issue. I think it took 15 minutes for them to acknowledge an issue, thats a lot of time for our services.

View on HN

That's because it's a political issue inside AWS. They have the technology to report it automatically, but there's a strong pressure not to post "green-i" or yellow or red, because those things impact SLA payments.

So if there's any way they can spin it as not an outage they will try not to post it.

I participated in multiple sla penalty payments requests on the customer side and never once we even discussed status page color

I participated in many status page color discussions and we always discussed sla. Shrug

You mean on the AWS/service provider side? I've also participated in internal incident response on the service provider side and we again made SLA refund decisions based on actual customer impact not our status page. But then again we diligently update our status pages so there's that.

Yeah, on the AWS side.

That's a conflict of interest and it's a shame the proper governing bodies have not penalized Amazon for such shitty behavior.

This is my understand as well. I have anecdotally heard swapping some of those dots from green to red can require up to CTO approval.

Can confirm this, it's political. It's always great seeing the "rationality" for not doing the right thing. After the 3 outages last December, I can remember a certain person in a certain global outage slack channel laughing at customers who weren't "resilient" enough to withstand the outage. Then AWS focused a resiliency campaign to document all those customers risks, as if AWS's own poorly designed system wasn't the cause for peoples inability to fail over. Glad I'm out of that place, it's super toxic these days.

It would be nice if all services had a great story. The guidance for how to make Cognito and SSO multi-region is laughable.

I know many services that do this. And even if you fight with customer care for RCA they go silent and pass you around to other team members to whom you need to tell the problem again and again.

They are brutally political and incompetent.

It was 41 minutes from the first sign of trouble in our monitoring (an SQS queue backing up because the lambda it triggers wasn't getting invoked) to AWS posting the "informational" status.

I continue to be irritated that services that consistently return errors are characterized as "increased error rates" or "increased latency". They seem to use those phrases for any kind of outage.

It's all networking issues as you're remotely access them remotely, so of course it's increased error rates or increased latency. When a piece of gear stops responding the network tries to self-heal as designed which causes those issues.

Well, yes, but, for example: Characterizing both a 5% error rate and a 100% error rate as "increased error rates" is not as helpful as it could be.

I'm reasonably sure they know when a service isn't functional at all.

It's all in the messaging. I once worked at a place that was having bandwidth problems because of an "issue with an upstream vendor." The "issue" was we hadn't paid our bill in a couple months, so they disconnected us.

I don’t think you should rely on Amazon to tell you anything.

Yup, we're trying to fix this here: https://www.awareops.com/

Reply on news.ycombinator.com