Hacker News Reader

Comment by nasalgoat | Hacker News Reader

This seems like another reason to not rely on Amazon-specific services, other than the obvious vendor lock-in.

At least in the event of an instance outage you could conceivably migrate off Amazon to another VPS provider. No one using DynamoDB has an alternative.

View on HN

How would this have helped? In the event of an AWS outage, you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure. This outage lasted a few hours, wouldn't it take you that long to switch providers anyway? If you don't abdicate responsibility for maintaining the infrastructure, you lose a lot of the operational benefits of using a cloud service...in general, I think it makes a lot more sense to embrace whatever cloud provider you're using (provided you made a good choice) so that you can focus your efforts on your product and not on maintaining infrastructure.

> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure

No, they are fixing their infrastructure. The point here is that all single provider systems are destined for periodic failure. Not relying on one single provider is, in theory, a service's means of providing higher reliability. This is the general argument for regions and availability zones on Amazon, but that relies on trusting there is no single point of failure with the system (i.e. inside Amazon).

I've worked for several companies who run the majority of their services on AWS, yet maintain functional systems on other providers in the case of an AWS outage.

> wouldn't it take you that long to switch providers anyway?

It might, or might not, depends how you built it.

> you have all-hands of one of the biggest tech companies in the world working to fix your infrastructure.

And it still was out for hours... You might see it as a great success, but you can also interpret it as "what good is it, even if with so many people behind the scenes they still failed for hours...".

The problem is that because that services ends up being embedded in so many services and products, half the internet ends being down. Even Amazon dogfoods their own stuff so services rely on each other. Dynamo is down, maybe SQS will be down or analytics as well.

This really is the first major failure of DynamoDB since when I started to use it, when it was in beta over 4 years ago. Not sure many companies would be able to run a multi-million QPS scale database with such a great availability record. In any case we just moved traffic away from us-east-1 and into us-west-2 and all is fine still so you should design at least your critical systems for regional failures.

Very good point. I wrote a blog post on this topic to try and characterize different types of lock-in. http://www.eightypercent.net/post/types-of-lock-in.html

This was as also part of the strategy when we decided to open source kubernetes. Having an open alternative made the commercial offering (Google Container Engine -- GKE) much stronger because of the reduced dev lock-in.

I don't disagree that you would want to have a rough idea of what migrating off of DynamoDB would require but wouldn't the easier step be using redundancy across regions first? Most of the sites which have suffered downtime due to AWS outages have been operating in only a single region (or even AZ!) and adding the extra level isolation is usually going to be a lot easier than dealing with multiple vendors or having to maintain more of your infrastructure directly.

In this case AWS sign in is down (perhaps it depends on Dynamo), so redundancy across regions can only help you if fail over is fully automatic which has it's own problems (split brain syndrome...)

AWS sign-in appears to be working but in general it's true that you need some sort of automatic failover. However, isn't that true for every option other than accepting downtime? It doesn't seem like an AWS-specific challenge other except at the implementation level.

One of the first things Amazon tell you about AWS is the API is king - the console is hosted in us-east-1 but the API is hosted in each region independently.

You don't need automatic failover, but you do need your own scripts to talk to the AWS API. The whole point of AWS is automation.

There are many high profile companies down (airbnb, IMDb, tinder, ...) so apparently this is not so straightforward.

I don't see airbnb or IMDB being down. If there's some advice on what small time apps (like mine) can do to get thing up again sooner, please let me know.

Explains the weird 500's i was getting when trying to do a review this morning.

Running multi-region services has associated costs. It's also non-trivial engineering effort to convert most services to a multi-region setup. It's definitely worth the investment though. Another option is to run multi-region on the same provider (like AWS) without using proprietary services like DynamoDb but maybe something similar like Cassandra.

I don't much use it anymore but this is something I always liked about AppEngine: baked into the pricing is automatic replication over three regions.

Serious AWS web apps are also distributed across multiple regions/availability zones, but many companies don't do that. It would be good for hosting services like Heroku, which I use, would replicate across zones also.

To be fare to Amazon, they tell you you architect and built across availability zones.

dynamo db is affected in the entire region, however, not just a single AZ.

That's why parent post mentioned multiple regions. No other AWS region was affected...

DynamoDB now support cross-region replication [0] so you can build more resilient applications with it

[0] http://docs.aws.amazon.com/amazondynamodb/latest/developergu...

I don't think cross-region replication would've helped in this case:

"The replica tables are intended to serve as read-only copies of the data; however, it is possible to write data to a replica table. If you write data to a replica, those changes will not be propagated to the master, or to any other replicas."

You could at least fall back to a read-only mode. That could be very helpful, compared to going down completely.

You can relatively trivially build multi-master cross-region replication in DynamoDB by using kinesis and writing to kinesis instead of DynamoDB directly. On the consuming end of Kinesis you then fan out to all the DynamoDB (or whatever other database you want to use) regions[1]. Admittedly this only works with some relatively relaxed constraints on the latency you can see, intra region latency can go up to 1 second although rarely, while cross-region is around 3 seconds. An important role is also played by the structure of your objects and how accepting they are of concurrent updates coming from different regions (which is the main reason why the default replication in DynamoDB is not multi-master).

[1]: http://tech.adroll.com/blog/data/2015/06/26/kinesis.html

DynamoDb was only down in only one region, for the first time in years. It's hardly a reason to migrate off, especially with DynamoDb still being available in every single other region throughout. It's simpler to fail over to another AWS region than to an entirely different cloud provider with a different API.

Source for service with better uptime than Amazon?

I believe he meant things like using RabbitMQ instead of SQS (SQS isn't very good anyway IMHO) and things like that. More services running on plain old instances rather than AWS specific APIs, Azure specific APIs, etc.

This prevents you from getting too locked in and losing control over your application.

I honestly can't believe that Netflix can't even load their home page without DynamoDB and all this other stuff.

Edit: Looks like Netflix is back online (10:53AM EST)

Yes, exactly. It's tempting to drink the Kool-Aid on the various services but the more you rely on them, the less flexible you become and more open to overall service failures like this one. You can take advantage of the benefits of EC2 without relying on the other services.

I mean, if even Netflix can't stay up during this, what hope does a startup have?

Netflix -IS- up, and working fine. Pretty amazing, imho.

http://downrightnow.com/netflix

https://imgur.com/NNYLUZK

There is only Zuul?

Can you hint at what this means, for the uninitiated?

I think he was making a reference to ghostbusters: http://knowyourmeme.com/memes/there-is-no-dana-only-zuul

Ok, lmgtfy. https://github.com/Netflix/zuul

Pocket just came back up also. Issue might be resolved?

I was watching Netflix this morning and a number of titles had troubles loading repeatedly. They did eventually work though.

This is silly. Do you think you and your staff will maintain your homebrew RabbitMQ installation with better uptime than Amazon's team? And how much time/expense/manpower are you willing to devote to setting it up and monitoring it? And is that going to make your startup more successful than if you spent those resources developing the features your customers are asking for?

Yes, competent people with good planning can easyly do a lot better than that.

We do serious hosting for government and company and if we had a outage like amazon had yesterday, we would loose at least half of our clients and pay heavy cash penality.

I don't understand why people give so much a break to amazon. They have one of the worst uptime.

Kinda agree, same goes to AWS tools, Opsworks screwed us up back then due to AWS outage, and decided to decouple as much as possible from Opsworks.

You still depend on some provider. Or, are you talking about multi cloud installations ?

Alternatively, run a fallback data store.

I would highly recommend http://www.datomic.com as a database for multiple reasons, but one of them is that fact that it's totally portable between DynamoDB, PostgreSQL, MySQL, etc.

You can create a Datomic backup from any of these databases and restore them into a different one with the exact same semantics.

Isn't datomic closed source?

That's what it looks like. And after 10 seconds of reading I couldn't tell what it actually does.

It is closed source, just like DynamoDB.

However - unlike DynamoDB - it has a quite usable free version and an even better starter version, which are good enough for production use. (The DDB local version is really only good for development) By production use, I mean for example a non-replicated MongoDB/PgSQL/MySQL, which is what I see from many smaller companies... They would be already really better of with Datomic.

If you really need more features, like high availability and transparent query caching with memcached, then the cheapest paid version has a one-time cost of $3000: http://www.datomic.com/pricing.html

You can easily save that amount of money by not spending 3-6 months developing tons of unnecessary queries, try to maintain that code and try to optimize its performance for scale...

On the "open-sourceness" note, I feel obliged to mention https://github.com/tonsky/datascript which IS open-source, and has a Datomic-like interface, but it's "just" an in-memory implementation.

But we are getting off-topic here I guess...

Reply on news.ycombinator.com