Hacker News Reader

Comment by felixgallo | Hacker News Reader

"It was impacting a relatively small number of customers, but we should have posted the green-i to the dashboard sooner than we did on Monday."

Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.

View on HN

What's with this all-or-nothing attitude? What is so wrong about different severity levels?

Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers

Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region.

Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.

different severity levels are fine. That's what the indicator is for. Decorating the indicator, normally completely incorrectly, with a tiny extra indicator, is incomprehensible. That's why you've never seen a tiny red light in the upper corner of your green traffic light. Your 'check engine' light is not green with a tiny extra indicator in the corner. The metaphors go on and on.

Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on?

Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually!

Manually!

The question of timing of the green-I is valid, but different from your original complaint.

In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.

It's really easy to miss the indicator "i". I've been doing this for 4 years and it still doesn't catch my eye when scanning the page.

I agree 100%. That column of uniform green makes it easy to miss. It is a poor UI choice but can be easily fixed.

However it is hard to miss the difference in the column with details and the more link even upon a cursory scan.

I don't care what that "i" indicator is; if you're showing green while you're impacted, your status board is useless and I'm going to Twitter and HN to see if you're broken (which is what I already have to do for AWS).

When we've seen problems, some of my clients go there and see green and say "Amazon shows they're fine - it must be your code or something you did or didn't do - fix it". Well... it's green but it's not fine. And there's little I can do to demonstrate to someone that it's not code but infrastructure.

Green traditionally means we're good, ready to go, etc. Non-intuitive to use any green indicator for a situation deserving a write-up like this. It's yellow at best.

During the actual outage this write up was about, didn't it go red?

I wasn't watching the indicator. I started with the parent comment's assertion. If it was wrong, my comment doesn't apply to this outage and becomes a rule of thumb.

Yep, especially yellow is labeled clearly as "Performance issues". It is basically lying to make things appear better than they really are.

> Some small percentage of customers are affected, but most of you have nothing to worry about.

That's absolutely not what it means practically for any of the recent outages. It has come to mean "the service is down but not hard enough for us to admit it".

> By 2:37am PDT, the error rate in customer requests to DynamoDB had risen far beyond any level experienced in the last 3 years, finally stabilizing at approximately 55%.

Yeah, a small green-i should cover it.

I'm starting to wonder how much more I should trust AWS. Is everything swept under the carpet?

e.g., not even attributed or thanks for root escalation vulnerabilities that I surfaced for them: https://news.ycombinator.com/item?id=10261507

Reply on news.ycombinator.com