Summary of the Amazon DynamoDB Service Disruption
aws.amazon.com
aws.amazon.com
Amazon. No. Please listen. You should never post a green-i. Green-i means nothing to anyone. It's a minimization of a problem. You should change the indicator to show that there is a problem. If there is a problem, that is what the indicator is for. Customers don't care about whatever weird internal political pressure causes you to want to show as little information as possible on that dashboard. Customers want to know that there is a problem, and we will totally be able to figure out if it is small or large on our own.
Green-i: Some small percentage of customers are affected, but most of you have nothing to worry about. So if you are a customer having problems and see this notice, you are re-assured that you are not crazy. And if you are NOT seeing problems, you probably won't. Amazon has not always been the best at posting this information quickly enough, hence a commitment to be better at messaging for those customers
Yellow: A significant amount of customers is affected. Potentially serious problems, and if you haven't seen them yet, don't be surprised if you will. Start monitoring your service health and preparing for a failover to another region.
Red: FUBAR. Happens rarely, but when it does you'll probably know even before your application alarms - you'll notice when half the internet shuts down.
Customers are totally not interested in the fact that some other random customer may not be experiencing problems and that amazon has a wide variety of services, some of which may not be affected in certain geographies. Customers go to that panel with one express goal: to discover if the problems they are seeing with their servers could be related to them. It is a triage check. The question is not, is Amazon super great and are the availability engineers the bestest on average over time? The question is, what the fuck is going on?
Amazon has suffered catastrophic unavailability, and the green-i has appeared belatedly, an hour or more into the problem. Because as revealed in this post, it is manually put up by engineers. Manually!
Manually!
In your hypothetical scenario, the customer that is going to the panel to "find out what the fuck is going on" is not going to miss the indicator i, and will read the description of the message.
That's absolutely not what it means practically for any of the recent outages. It has come to mean "the service is down but not hard enough for us to admit it".
Yeah, a small green-i should cover it.
e.g., not even attributed or thanks for root escalation vulnerabilities that I surfaced for them: https://news.ycombinator.com/item?id=10261507
As much as I admire and rely on AWS' scale to build architectures and fault tolerant applications, it can't be ignored that the marketing towards going "full cloud" doesn't take into account how hard it is to build resilient architectures in the cloud.
I see those disruptions events as stop signs: when the cloud itself fails to scale, I rethink a few decisions we all make when surfing those trends.
http://yourdatafitsinram.com/ also comes to mind.
Metrics depend.
So managing your stuff is hard, but you are in control and can do things in a way you believe is completely safe for you. Or you at least may incur in the same events sometimes, but perhaps paying a lot less for the same services. Or you can create your deployment with characteristics which are often impossible (a lot of RAM for each server is an example) to be cost effective in the cloud.
It's not stupid to use AWS services but is not stupid to manage your operations, either in your own hardware or at least using just bare metal and/or the virtual machines service certain providers give you, but still being in part accountable, responsabile, and in control, of your system software deployment and operations.
I mean, you can argue that the cloud is better. But how often is Heroku and AWS down? About the same as physical providers (I concede S3 is pretty solid though).
You call up Bull, Fujitsu, or Unisys for the same thing.
You call up HP. You ask for a NonStop solution. You get same thing for at least 20 years.
You call up VMS Software. You ask for an OpenVMS cluster. You get same thing for at least 17 years.
Well-designed OS's, software, and hardware did cloud-style stuff for a long time before cloud existed without the downtime. Cloud certainly brought price down and flexibility up. Yet, these clouds haven't matched 70-80's technology in uptime yet despite all the brains and money thrown at them. That's a fact.
So, shouldn't be used for anything mission critical where downtime costs lots of money.
Edit to add: mainframes also run user and server type workloads. Some of those are predictable, some aren't. Bull's virtualize whole desktops. The mainframe as a whole, esp important services, are usually still available despite issues with those. For instance, my company splits stuff between critical on mainframe or AS/400's plus non-critical on whatever is useful ("best-of-breed" they say...). The critical stuff is either on the IBM stuff or leverages it in client-server setup. Those apps either always work or (rarely) they fail-safe in an obvious way that does no damage. Nobody I work with can remember those systems going down over 10 years they worked there. The other stuff regularly has issues across the board. The key difference is effective architecture and how it's implemented.
One day, they failed. One went offline - for reasons never revealed, at least to me - and the secondary didn't come up. Radio silence, kaput. But an airline that housed mainframes in the same DCs had their booking system fail at exactly the same times (with national headlines to match).
The myth of mainframe uptime is exactly that. La-la-land for hardware & services salesmen.
Meanwhile, the companies I've worked at all had mainframes without trouble from them that people said. Problems were virtually always the app developers or the pain of doing 21st century stuff with 60's-80's architecture or legacy code.
Basic strategy was putting something in front of them that can redirect to the new system upon a trigger. Let's assume its functionality + tons of data. The new system first gets the data moved to it in batch form for efficiency reasons. Once it catches up enough, it starts syncing in a more online fashion until it gets to point that it's syncing in real-time. All kinds of tests are performed on that system throughout this process. Eventually, a change-over happens that should be barely perceptible. The inability to do this is usually due to fragile architecture or tightly-coupled implementations which are unfortunately all too common in enterprises.
Note: It can also help if your app was written in something like Common LISP or Erlang that supports live updates. That with the delta approach (version A->A/B->B) equals upgrades with no downtime. ;) Combining it with clustering approach is quite powerful but clustering approach is more applicable to tools majority uses.
that said, even with those disruptions and whatnot happening on Amazon as a warning, I am not skilled enough nor have time enough to build a non cloud resilient infrastructure.
I was looking to go with redundant vps at first, because amazon does have high cost for us, however, just learning all the things that can go wrong in the first very part, the load balancer, and all the gritty details one have to consider for just this little component to support interruption free failover, made me rethink the cost benefit of going managed.
it is true that going cloud doesn't really remove outages risks completely and it will not be as resilient as an infrastructure built with skill and love by the best out there, but how many shops can actually roll with their own solution and get an equivalent level of availability?
scaling web nodes is within my capabilities, building a ha database is already quite above my skill but I may manage, testing database failover, making sure it works, making sure that it can actually recover from one node dying and that the application stay live meanwhile? that's way above what I can reasonably do and what my company can afford to pay maintenance for.
6:33 AM PDT We are investigating increased error rates for API requests in the US-EAST-1 Region.
Source: http://status.aws.amazon.com/
"For us, availability is the most important feature of DynamoDB"
I would think that durability should be the most important feature of DynamoDB; better to have intermittent outage, or reduced capacity, but not have data loss - but perhaps there is something about DynamoDB which suggests availability is rated more highly than durability?
I personally didnt know they used meta data tables in Dynamo, and that sounds like they lost one of the most important traits which gave Dynamo its super-high availability abilities, which is having all the cluster state in the clients rather than in the servers.
http://www.allthingsdistributed.com/2012/01/amazon-dynamodb....
There are many things to like about DynamoDB, but one of the things I really dislike is the 'abscence of an error == success' pattern that they implement for many operations. With less frequent metadata requests, I imagine there is a greater chance of silent failures (ie availability looks fine, durability is gone).
Could be wrong though, I'm not going to pretend to know more about distributed systems than the DynamoDB developers.
If you suspect the design of DynamoDB is sloppy, I encourage you to read this: http://cacm.acm.org/magazines/2015/4/184701-how-amazon-web-s...
Use full orange or even half orange circle or something else if you want to convey 'subset of customers may have issues'. If possible move 'having issue' items to the top - this way people do not have to scroll down to see whats broken.
(I'm sure it had happened before, just the first time we had seen them.)
The whole point of status page is to to help determine if the issue you're experiencing is on your side or the SaaS, and not to show as much green as possible.
The way currently AWS status page works it simply fails to provide any functionality and might as well be shut down.
The colors on it start to change when you already see tons of articles about AWS being down.
It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.
More importantly, did they use randomised exponential backoff?
Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.
I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!
My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.
> ... subsequent impact to other AWS services that depend on DynamoDB ...
And the metadata service is part of DynamoDB:
> The membership of a set of table/partitions within a server is managed by DynamoDB’s internal metadata service.
"EC2 Auto Scaling stores information about its groups and launch configurations in an internal table in DynamoDB"
"CloudWatch uses an internal table stored in DynamoDB"
"Customers attempting to log into the Console during this period saw much higher latency in the login process. This was due to a very long timeout being set on an API call that relied on DynamoDB"
Seems like poor architectural design to have all of these storing state in the same instances of DynamoDB that are used by customers. If a new feature like GSI is added it should under no circumstances ever impact other services.
Would it have been wiser for Amazon to implement a completely separate instance of DynamoDB for service offerings that depended on it? Or is this just simply cost ineffective? Help me understand, thanks. =)
source: was on an AWS service team for several years
On one hand, one can design a service with minimum dependencies to survive other services' outages, but that means duplication of effort and increased cost, as well as slower delivery of features. Or one can focus on adding value.
It seems though, from the update, that CloudWatch is going to have some sort of caching for most recent data.
But good to know! Thanks!
It is often the case that you aren't monitoring the metric that would have told you the world was about to end. However,
> their processing time was near the time limit for retrieval
This is a specific response time with a known, fatal, limit. Not seeing that trend seems unfortunate.
Critical lesson for running multitenant SaaS services at scale: you need to monitor not just overall error rates, but individual tenant/customer error rates.
Most of the historical data isn't shown publicly but I could probably do something with that.