Hacker News Reader

Comment by jph00 | Hacker News Reader

> From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3.

Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with confidence about how the rest of the system is designed, frankly...

View on HN

Anyone who works on large complex systems will read this is and go "this was all preventable, but also completely understandable".

I agree with that in general but having your monitoring system be dependent on the thing it monitors is a pretty big goof. It possible that the dependency was very non-obvious and many layers deep, which is more understandable, but still...its pretty fundamental.

The monitoring system was not dependent on the thing it was monitoring.

The website that shows the public results of the monitoring, which is updates only by humans, depended on it.

An important distinction but doesn't negate the ideas that their users should be in the dark because of something this stupid.

They have a twitter account for such incidents and used it appropriately. They did not slack in relaying the outage to customers, and between that and the fact that no S3 services were operating I think the message was pretty clear: "We fucked up, give us a couple hours"

I'm, I don't understand.

My us-east-1 RSS feed said S3 had no incidents.

And apparently they had never tried rebooting some of the most important parts of that system. Just when you start to think that someone's really gotten it right you come to learn they're just fumbling around in the dark like everyone else.

My interpretation of this is that the indexing system was resilient to lost of a certain amount of capacity (probably around ⅓ + 1 host). As a guess, the indexing system probably used some form of consensus (e.g. paxos) which has had an active leader for years. Deployments stay within that capacity constraint, so while hosts have been restarted and replaced (data center migrations, hardware lease expiration, failures, upgrades, etc.), they may have not recently run into a situation where quorum wasn't available for a partition, especially at the scale of restarting the entire fleet.

Since restarting the entire fleet would incur downtime of all relevant S3 operations, it's unlikely that it was something ever intentionally done in production (and they may or may not have run that scenario in other environments).

Source: I used to run several large scale services at Amazon.

To put what @jonhohle said another way, Amazon had probably never brought up the entirety of S3 from Zero to Production-Ready on in a production environment before. I wouldn't necessarily classify this as "fumbling around in the dark." Perhaps they should have tested this in a simulated environment, but (to be fair) on a distributed fault-tolerant system, it probably wasn't a top-priority situation to test.

> Perhaps they should have tested this in a simulated environment

What makes you think they didn't?

Moreover, beyond a certain scale, it becomes really hard to simulate.

I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

Or have it pull from two sources, one local (S3) and one remote (GCE or whatever), and make a hard positive from either source signal "down." Otherwise the page would be down if just the remote source were down.

Precisely.

"we have changed the SHD administration console to run across multiple AWS regions."

You have to eat your own dogfood, if you're trying to sell a service product.

That being said, a better solution would be to stand up a separate infrastructure just for status pages.

"Except for status pages" is a perfectly reasonable exception to "eat your own dogfood".

If they did host their status page on a competitor's service, then they'd be reliant on that service, which might backfire if the competitor's service goes down while Amazon's own systems stay up.

What they really need is failover capability, which can fire up the status page on a competitor's service (or maybe on a completely separate disaster recovery site site owned by Amazon) in case Amazon's own services go down.

I'm sure Amazon's architects and engineers are more than capable of designing and implementing such a robust system and recognizing its importance. So it puzzles me as to why it wasn't done.

You can use several hosts, have two subdomains so if one is not red responding, engineers and managers know there are two status pages. Heck, have two different domains for them as well if there are dns issues. Amzstatus1.com and 2. Not dependent on Amazon domain anymore either.

Or to host it on a static .html page that gets rewritten every 60 seconds or so by an external process, running on physical servers. Minimal stack, so minimal attack surface.

That is effectively what happens, they just stupidly backed that file from S3.

http://status.aws.amazon.com/data.json

Considering the size of AWS, the number of services one service relies on would be too many, even if a single service many layers deep depended on s3, and when s3 goes down, the service as a whole will be affected. Honestly S3, it like a blackhole, everybody stores everything in S3 these days. Its a horizontal component but used in a vertical manner. Weird but true.

How many companies do you think running on AWS host their status page on S3?

Reply on news.ycombinator.com