Hacker News Reader

Comment by jlafon | Hacker News Reader

It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?

View on HN

This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).

That makes sense. I had only seen the Memcache & Redis datastores being used (and both being comparable to DynamoDB).

Why would you use multiple datastores? Isn't it much more difficult to atomically store data this way? Assume a transaction succeeds in store 1, and fails in store 2, you'd need to roll back the transaction in store 1 after it has committed (!)

Because there is no such thing as a one size fits all datasotre. Some data may make way more sense stored in a relational database, while some data makes more sense as key:value pairs. Some data may be infrequently accessed and stored to disk while others are stored in memory for quick access. Trying to make a single datastore work for many use cases will cause more pain than it is worth.

You wouldn't likely use multiple data stores for the same data. That is, you probably wouldn't have a single system which stores data in both Redis and MySQL.

However, you can imagine different use cases for each of these stores and you can potentially increase operational efficiency by having a common layer which handles the distribution of data for each store.

Keep in mind that they define

DC := AWS region

Rack := AWS AZ

That should tell you that they need multi-region availability, something that DynamoDB will likely never provide.

This is the sort of project you do to get cross-region replication on your existing data stores, many of which look more like PostgreSQL than KV stores.

Reply on news.ycombinator.com