Hacker News Reader

Comment by zegl | Hacker News Reader

I love that every time this happens, 100% of the services on https://status.aws.amazon.com are green.

Those five 9s don't come easy. Sometimes you have to prop them up :)

I wonder how often outages really happen. The official page is nonsense, of course, and we only collectively notice when the outage is big enough that lots of us are affected. On AWS, I see about a 3:1 ratio of "bump in the night" outages (quickly resolved, little corroboration) to mega too-big-to-hide outages. Does that mirror others' experiences?

If you count any time AWS is having a problem that impacts our production workloads then I think it's about 5:1. Dealing with "AWS is down" outages are easy because I can just sit back and grab some popcorn, it's the "dammit I know this is AWS's fault" outages that are a PITA because you count yourself lucky to even get a report in your personalized dashboard.

Yep.

Random aside: any chance you are related to the Calculus on Manifolds Spivak?

I had to log in to say, that one of my favorite quotes of all time I found in Calculus on Manifolds.

He says that any good theorem is worth generalizing, and I've generalized that to any life rule.

Nope, just a fan. It was the book that pioneered my love of math.

https://aws.amazon.com/compute/sla/

looks like only four 9's

  > looks like only four 9's

That's why the Germans are such good engineers.

  Did the drives fail? Nein.
  Did the CPU overheat? Nein.
  Did the power get cut? Nein.
  Did the network go down? Nein.

That's "four neins" right there.

Every time someone asks to update the status page, managers say "nein"

It’s hard to measure what five-9 is because you have to wait around until a 0.00001 occurs. Incentivizing post-mortems are absolutely critical in this case.

It's 0.001; the first 2 9's count.

  5N  = 99.999%
  3N  = 99.9%
  1N5 = 95%

5N is <43m12s downtime per month.

I considered writing it as a percent but then decided against using it and moving the decimal instead. But good info for clarification.

EC2 or S3 showing red in any region literally requires personal approval of the CEO of AWS.

Is this true or a joke? This sort of policy is how you destroy trust.

From what I've heard it's mostly true. Not only the CEO but a few SVPs can approve it, but yes a human must approve the update and it must be a high level exec.

Part of the reason is because their SLAs are based on that dashboard, and that dashboard going red has a financial cost to AWS, so like any financial cost, it needs approval.

Sure, but... that just raises more questions :)

Taken literally what you are saying is the service could be down and an executive could override that, preventing them for paying customers for a service outage, even if the service did have an outage and the customer could prove it (screenshots, metrics from other cloud providers, many different folks see it).

I'm sure there is some subtlety to this, but it does mean that large corps with influence should be talking to AWS to ensure that status information corresponds with actual service outages.

Large corps with influence get what they want regardless. Status page goes red and the small corps start thinking they can get what they want too.

> Status page goes red and the small corps start thinking they can get what they want too.

I think you mean "start thinking they can get what they pay for"

I have no inside knowledge or anything but it seems like there are a lot of scenarios with degraded performance where people could argue about whether it really constitutes an outage.

One time gcp argued that since they did return 404s on gcs for a few hours that wasn’t an uptime/latency sla violation so we were not entitled to refund (tho they refunded us anyway)

Man, between costs and shenanigans like this, why don't more companies self-host?

1. Leadership prefers to blame cloud when things break rather than take responsibility.

2. Cost is not an issue (until it is but you’re already locked in so oh well)

3. Faang has drained the talent pool of people who know how

If you think that’s bad you should see the outages when you self host without a big enough team to really manage it.

Opex > Capex. If companies thought about long term, yes they might consider it. But unless the cloud providers fuck up really badly, they're ok to take the heat occasionally and tolerate a bit of nonsense.

You can lease equipment you know…

Yep. I was an SRE who worked at Google and also launched a product on Google Cloud. We had these arguments all the time, and the contract language often provides a way for the provider to weasel out.

Like I said I never worked there and this is all hearsay but there is a lot of nuance here being missed like partial outages.

This is no longer a partial outage. The status page reports elevated API error rates, DynamoDB issues, EC2 API error rates, and my company's monitoring is significantly affected (IE, our IT folks can't tell us what isn't working) and my AWS training class isn't working either.

If this needed a CEO to eventually get around to pressing a button that said "show users the actual information about a problem" that reflects poorly on amazon.

My friend works at a telemetry company for monitoring and they are working on alerting customers of cloud service outages before the cloud providers since the providers like to sit on their hands for a while (presumably to try and fix it before anyone notices).

Being dishonest about SLAs seems to bear zero cost in this case?

It's not really dishonest though because there is nuance. Most everything in EC2 is still working it seems, just the console is down. So is it really down? It should probably be yellow but not red.

if you cannot access the control plane to create or destroy resources, it is down (partial availability). The jobs that are running are basically zombies.

I'm right in the middle of an AWS-run training and we literally can't run the exercises because of this.

let me repeat that: my AWS trainign that is run by AWS that I pay AWS for isn't working, because AWS is having control plane (or other) issues. This is several hours after the initial incident. We're doing training in us-west-2, but the identity service and other components run in us-east-1.

I’m running EKS in us-west-2. My pods use a role ARN and identity token file to get temporary credentials via STS. STS can’t return credentials right now. So my EKS cluster is “down” in the sense that I can’t bring up new pods. I only noticed because an auto-scaling event failed.

We ran through the whole 4.5 hour training and the training app didn't work the entire time.

Seems like the API is still working and so is auto scaling. So they aren’t really zombies.

Partial availability isn’t the same as no availability.

The API is NOT working -- it may not have been listed on the service health dashboard when you posted that, but it is now. We haven't been able to launch an instance at all, and we are continuously trying. We can't even start existing instances.

Depending the workload being run users may or may not notice. Should be Yellow at a minimum.

Heroku is currently having major problems. My stuff is still up, but I can't deploy any new versions. Heroku runs their stuff on AWS. I have heard reports of other companies who run on AWS also having degarded service and outages.

i'd say when other companies who run their infrastruture on AWS are going out, it's hard to argue it's not a real outage.

But AWS status _has_ changed to yellow at this point. Probably heroku could be completely down because of an AWS problem, and AWS status would still not show red. But at least yellow tells us there's a problem, the distinction between yellow and red probably only matters at this point to lawyers arguing about the AWS SLA, the rest of us know yellow means "problems", red will never be seen, and green means "maybe problems anyway".

I believe the entire us-east-1 could be entirely missing, and they'd still only put a yellow not a red on status page. After all, the other regions are all fine, right?

"Good at finding excuses" is not the same thing as "honest."

SNS seems to be at least partially down as well

My company relies on DynamoDB, so we're totally down.

edit: partly down; it's sporadically failing

Little honesty from cloud providers will ensure regulators don't step in.

Zero directly-attributable, calculable-at-time-of-decision cost. Of course there's a cost in terms of customers who leave because of the dishonest practice, but, who knows how many people that'll be? Out of the customers who left after the outage, who knows whether they left due to not communicating status promptly and honestly or whether it was for some other reason?

Versus, if a company has X SLA contracts signed, that point to Y reimbursement for being out for Z minutes, so it's easily calculable.

I wonder how well known this is. You'd think it would be hard to hire ethical engineers with such a scheme in place and yet they have tens of thousands.

maybe we gotta consider the publicly facing status pages as something other than a technical tool (e.g. marketing or PR or something like that, dunno)

If you trust them at this point, you have not being paying attention, and will probably continue to trust after this.

Well, no big deal, there's not really a lot of trust there to destroy...

Unfortunately, errors don't require his approval...

Uhhhhh... what if the monitoring said it was hard down? They'd still not show red?

Probably they cannot. They outsourced this dashboard and it runs on AWS now ;).

It's like trying to get the truth out of a kid that caused some trouble.

Mom: Alexa, did you break something?

Alexa: No.

M: Really? What's this? 500 Internal server error

A: ok maybe management console is down

M: Anything else?

A: ...

A: ... ok maybe cloudwatch logs

M: Ah hah. What else?

A: That's it, I swear!

M: 503 ClientError

A: ...well okay secretsmanager might be busted too...

Funny I literally just asked my Alexa.

Me: Alexa, is AWS down right now?

Alexa: I'd rather not answer that

Wise robot.

That's a bit like involving your kid in an argument between parents.

There was a great response in r/relationship advice the other day where someone said that OP's partner forced a fight because they're planning to cheat on them, reconcile, and then will 'trickle out the truth' over the next 6 months. I'm stealing that phrase.

The very expensive EC2 instance I started this morning still works. Of course now I can't shut it down.

I don't see why they couldn't provide an error rate graph like Reddit[0] or simply make services yellow saying "increased error rate detected, investigating..."

0: https://www.redditstatus.com/#system-metrics

A executive has a OKR around uptime and a automated system prevents him or her from having control over the messaging. Therefore any effort to create one is squashed, leaving the people requesting it confused as to why and left without any explanation. Oldest story in the book.

Because Amazon has $$$$$ in their SLOs, and it costs them through the nose every minute they're down in payments made to customers and fees refunded. I trust them and most companies not to be outright fraudulent (although I'm sure some are), but it's totally understandable they'd be reticent to push the "Downtime Alert/Cost Us a Ton of Money" button until they're sure something serious is happening.

It literally is fraudulent though.

I don't think a region being down is something that you can be unsure about.

Oh, you can get pretty weaselly about what “down” means. If there is “just” an S3 issue, are all the various services which are still “available” but throwing an elevated number of errors because of their own internal dependency on S3 actually down or just “degraded?” You have to spin up the hair-splitting apparatus early in the incident to try to keep clear of the post-mortem party. :D

This is an incentive to dishonesty, leading to fraudulent payments and false advertising of uptime to potential customers.

Hopefully it results in a class action lawsuit for enough money that Amazon decides that an automated system is better than trying to supply human judgement.

Can someone just have a site ping all the GET endpoints on the AWS API? That is very far from "automating [their entire] system" but it's better than what they're doing.

Something like this? https://stop.lying.cloud/

Um, that's just a reskin of AWS's lying status page. No real new data there.

It should be costing them trust not to push it when they should though. A trustworthy company will err on the side of pushing it. AWS is a near-monopoly, so their unprofessional business practices have still yet to cost them.

> It should be costing them trust not to push it when they should though.

This is what Amazon, the startup, understood.

Step 1: Always make it right and make the customer happy, even if it hurts in $.

Step 2: If you find you're losing too much money over a particular issue, fix the issue.

Amazon, one of the world's largest companies, seems to have forgotten that the risk of not reporting accurately isn't money, but breaking the feedback chain. Once you start gaming metrics, no leaders know what's really important to work on internally, because no leaders know what the actual issues are. It's late Soviet Union in a nutshell. If everyone is gaming the system at all levels, then eventually the ability to objectively execute decreases, because effort is misallocated due to misunderstanding.

> It's late Soviet Union in a nutshell

How come an action of a private company in a capitalist country is like the Soviet Union?

Private companies are small centrally-planned economies within larger capitalist systems.

They talk about gaming the metrics. Something the Soviet Union was known for.

I can Google and see how many apps, games, or other services are down. So them not "pushing some buttons" to confirm it isn't fooling anyone.

The more transparency you give; the harder it is to control the narrative. They have a general reputation for reliability; and exposing just how many actual errors/failures there are (that generally don't effect a large swath of users/usecases) would do hurt that reputation for minimal gain.

Wow. Kudos to the reddit engineering team. That's one of the nicest status pages I have seen.

because nobody cares when reddit is down. or at least, nobody is paying them to be up 99.999% of the time.

> Goodhart's Law is expressed simply as: “When a measure becomes a target, it ceases to be a good measure.”

It’s very frustrating. Why even have them?

Because "uptime" and "nines" became a marketing term. Simple as that. But the problem is that any public-facing measure of availability becomes a defacto marketing term.

The older I get the more I hate marketers. The whole field stands on the back of war-time propaganda research and it sure feels like it's the cause of so much rot in society.

Also 4-5 nines is virtually impossible for complex systems, so the sort of responsible people who could make 3 nines true begin to check out, and now you've getting most of your info from the delusional, and you're lucky if you manage 2 objective nines.

On top of that, the "Personalized Health Dashboard" doesn't work because I can't seem to log in to the console.

I'm logged in; you're missing an error message.

We have federated login with MFA required (which was failing). It just started working again.

Scratch that... console is not loading at all now :)

No wonder IMDB <https://www.imdb.com/> is down (returning 503). Sad that Amazon engineers don't implement what they teach their customers -- designing fault-tolerant and highly available systems.

"some customers may experience a slight elevation in error rates" --> everything is on fire, absolutely nothing works

https://downdetector.com

Amazing and scary to see all the unrelated services down right now.

I think it's pretty unlikely that both Google and Facebook are affected by this minor AWS outage, whatever DownDetector says. I even did a spot check on some of the smaller websites they report as "down", like canva.com, and didn't see any issues.

You might be right about Google and Facebook, but this isn't minor at all. Impact is widespread.

When I worked there it required the signoff of both your VP-level executive and the comms team to update the status page. I do not believe I ever received said signoff before the issues were resolved.

I remember the time when S3 went down and took the status page down with it

Status pages are hard

When they have too much pride in an all-green dash, sure. Allowing any engineer to declare a problem when first detected? Not so hard, but it doesn't make you look good if you have an ultra-twitchy finger. They have the balance badly wrong at the moment though.

A trigger-happy status page gives realtime feedback for anyone doing a DoS attack. Even if you published that information publicly you would probably want it on a significant delay.

They sent their CEO into space, I am sure they have the resources to figure it out.

No they're not.

Step 1: deploy status checks to an external cloud.

I agree, but does come with increased challenges with false positives.

That being said, AWS status pages are up.

More like admitting failure is hard.

“Falsehoods Programmers Believe About Status Pages”

Not if you’re AWS. At this point I’m fairly sure their status page is just a static html that always show all green.

Well, it is.

It's widespread industry knowledge now that AWS is publicly dishonest about downtime.

When the biggest cloud provider in the world is famous for gaslighting, it sets expectations for our whole industry.

It's fucking disgraceful that they tolerate such a lack of integrity in their organization.

I wonder if the other parts of Amazon do this. Like their inventory system thinks something is in stock, but people can't find it in the warehouse, do they just simply not send it to you and hope you don't notice? AWS's culture sounds super broken.

My favorite status page, though, is Slack's. You can read an article in the New York Times about how Slack was down for most of a day, and the status page is just like "some percentage of users experienced minor connectivity issues". "Some percentage" is code for "100%" and "minor" is code for "total". Good try.

Makes you wonder if they have to manually update the page when outages occur. That'd be a pretty bad way to go, so I'd hope not. Maybe the code to automatically update the page is in us-east-1? :)

Something like that has impacted the status page in the past. There was a severe Kinesis outage last year (https://aws.amazon.com/message/11201/), and they couldn't update the service dashboard for quite a while because their tool to do manage the service dashboard lives in us-east-1 and depends on Kinesis.

Word on the street is the status page is just a JPG

I assume each service has its own health check that checks the service is accessible from an internal location, thus most are green. However, when Service A requires Service B to do work, but Service B is down, a simple access check on Service A clearly doesn't give a good representation of uptime.

So what's a good health check actually report these days? Is it just about its own status, or should it include a breakdown of the status of external dependencies as part of its folded up status?

It seems they updated it ~30 minutes after your comment.

Are they lying, or just prioritizing their own services?

https://music.amazon.co.uk is giving me an error since about 16:30 GMT

"We are experiencing an error. Our apologies – We will be back up soon."

amazon.com seems to be having problems too. I get something went wrong to a new design/layout which I assume is either new or a fail safe.

Looks okay right now to this UK user of amazon.co.uk

It depends on which Amazon region you are being served from.

It is very unlikely that Amazon would deliberately make your messages cross the Atlantic just to find an American region that is unable to serve you.

Willing to bet the status page gets updated by logic on us-east-1

Status service is probably hosted in us-east-1

Well yea, it's eventually consistent ;)

It starts to show issues now. I agree that it was a bit long before we can get real visibility on the incident.

Wonder why almost all Amazon frontend looks like it was written in c++

That page is not loading for me... on which region is it hosted?

The problem being that often times you can't actually update the status page. Most internal systems are down.

We can't even update our product to say it's down, because accessing the product requires a process that is currently dead.

That's why your status page should be completely independent from the services it is monitoring (minus maybe something that automatically updates it). We use a third party to host our status page specifically so that we can update it even if all our systems are down.

I'm not saying you're wrong, or that the status page is architected properly. I'm just speaking to the current situation.

Not right now. I think they monitor if it appears on HN too.

Even better, when I try to go to console, I get:

> AWS Management Console Home page is currently unavailable.

> You can monitor status on the AWS Service Health Dashboard.

"AWS Service Health Dashboard" is a link to status.aws.amazon.com... which is ALL GREEN. So... thanks for the suggestion?

At this point the AWS service health dashboard is kind of famous for always been green isn't it? It's a joke to it's users. Do the folks who work on the relevant AWS internal team(s) know this, and just not have the resources to do anything about it, or what? If it's a harder problem than you'd think for interesting technical reasons, that'd be interesting to hear about.

Reply on news.ycombinator.com