Dynomite – Making Non-Distributed Databases Distributed | Hacker News Reader

Dynomite – Making Non-Distributed Databases Distributed

techblog.netflix.com

185 pointsstefanwild60 comments

It's a little unfortunate that consistency is only mentioned as a footnote. If this is being used internally at Netflix, how are they dealing with consistency issues in the face of failed or conflicting writes?

This currently looks like the Dynamo-alike sharding without the read-repair part making values eventually convergent in all the replicas. This latter feature is much needed, but they are already planning it apparently.

Except no mention of vector clocks, so it's unclear how one would detect or repair a conflict. Maybe that is an implementation detail that is intended to be ironed out later...

With a replication factor of 3 or more you don't need a vector clock, you can just choose majority wins.

That being said, I believe it is using a vector clock, but I'm not sure.

You still need ordering amongst writes. Without that you will get into a situation where there is no majority, all three nodes have different data.

You need quorum writes and reads. Like Cassandra you can use timestamps even though vector clocks would be much preferred.

Converging to a single value (which is a liveness property) and what the values will contain (which are usually safety properties) are different stories... when the partition heals, if the nodes detect they have a different version for a given piece of data, they can just remove it, and the system is convergent, which means it is eventually consistent (a liveness property). What happens in the merge function instead provides additional safety properties. For example in the case of a CRDT that models a Set type, with the merge function that is "union of all the values in the nodes", you can state the safety property that if a Set element X was added into a node, at no time will happen that the nodes will be able to talk again, and the system will converge into a Set which excludes such value X.

Vector clocks are not mandatory, depends on what you want to build. There are different options like CRDTs or using a merge function which guarantees a weaker form of consistency.

If it's based on Dynamo, then it's using vector clocks.

Netflix is a Java-heavy shop (the client talked up in the blog post is the Java client). I was pleasantly surprised to see that dynomite is implemented in C with an eye towards both BSD + Linux. I'm curious if this was because a GC'ed language like Java or Go was tried and proven not up to the task, or what.

My guess would be that the ecosystem in their (seed) target set of databases and protocols is written in C. In one case, they're using Twitters twemproxy to provide connection counts for redis/memcached.

Not just connection counts; when I first looked at the github I was surprised at how close the code organization was to twemproxy, and when I looked further a huge amount of the code is built on top of existing twemproxy code.

Netflix probably saw an opportunity to built the Dynamo layer on top of what Twitter had already written, and decided to just keep building on in C instead of rewriting a fairly large codebase.

My guess would be that they went with C to make it easier to hook into existing data systems like MySQL or Redis.

My guess is that this has a performance reason

Netflix always makes for an interesting case, but what always strikes me is how incredibly "trivial" their application really is: It's a simple application at a very large scale. This doesn't dismiss their accomplishments, of course, but often it does seem like Netflix spends an enormous engineering effort on fairly low value aspects of the service.

You're discounting the "very large scale" portion of your statement.

What may seem fairly low value from the outside or consumer perspective is likely of huge value internally to ensure you can always watch Netflix without even thinking about everything that has to happen to deliver as much video as they do to their customer's browsers.

Well, they make it seem trivial, while dealing with every watch and every pause, every rating updating complex recommendation engine algorithms. They deal with licensing and region awareness and proxying by ISP and running their own CDNs and...

And that's the magic.

To boot, they do it at enormous scale on a multitude of platforms all while their own destructive code (Chaos Monkey) runs around destroying servers.

It's true, some things like "stream me this" are maybe not rocket science, but at Netflix scale, everything becomes a bit more difficult.

Netflix is good at providing the service on any device, and having 100% uptime.

Their requirements are 100% availability, and an API's that never breaks, but consistency rarely matters.

The engineering effort is cool, but it is solving a problem that I don't have. The only thing that matters to netflix is they list the catalog somewhat correctly, and always deliver a stream.

I have a different problem, thus I am uninterested in their magic. If I had a similar problem or they signed my paycheck, I'd care.

Not to deride what you are saying, but when I see comments that dismiss Netflix as a site that just simply streams movies consistently; it tells me that the commenters haven't seen and appreciated the difficulty of the problems they are actually solving under the hood as evidenced by their github page.

http://netflix.github.io/#repo

Neither that or the number of open engineering positions tells you the difficulty of the problems they are solving. It might just tell you the difficulty of the problems they are making: If Netflix has a technology side with a history and the executive ability to overcomplicate, they might be making little problems into big problems. My perspective as an outsider is to look at the complexity of their app platform (again, separating the video streams which happen from systems that never seem to be the foundation of these tech blog entries), and it is unarguably not a complex system.

Early on I remember the heroics that Netflix went through running on EC2, building numbers of complex tools for varied performance, reliability issues, etc. At the time many much larger sites just quietly worked 24/7, minus the heroics, and minus the effort, often by using purpose-suited dedicated servers. Later, once Amazon rolled out SSDs, Netflix triumphantly announced how much of an improvement it was to their product, again demonstrating that they were creating a problem (huge numbers of horrible I/O machines) that they then solved with gusto, albeit unnecessarily.

I'm not trying to be overly down on Netflix, but it is a company that seemed to make tech blog entries a product of the company years ago, and the result is that people have bought into this notion that they're doing some hugely complex task. They aren't.

Sorry but you are just flat out wrong. Operating at the scale that they are is in fact complex. They aren't creating all of this software just because they are having fun in their free time.

They aren't creating all of this software just because they are having fun in their free time.

Sorry, but this is an invention of yours. No one said they're doing anything flippantly.

But they may be doing it ignorantly: Success in one field (in this case turning a mail subscription business into a stream service) doesn't imply technical innovation or leadership, and often is despite it.

This industry is absolutely rife with people creating solutions to problems they themselves invented and caused. In the case of Netflix, an enormous amount of their solutions have been founded around the notion of deploying on huge numbers of miserable Amazon EC2 instances, and then dealing with the problems related to that. Others simply deployed distributed data centers hosting their own purpose suited, reliable hardware with big fat storage arrays, and the problem is solved. It's like trying to build a car out of toothpicks and then detailing the innovations you created in toothpick redundancy and robustness.

Sorry but this just reeks of naiveté. When the few areas of tech that I have actually been involved with in the real world come up on HN there are always comments stating that things are just so simple and why so and so is doing it wrong and why it so much easier if they just do something simplistic...

That just isn't the case in the real world when working with tens of millions of customers. This will become apparent when you actually get out there in the field and realize systems involving tens of thousands of machines hosting dozens of different services to millions of people is actually not simple.

I've yet to see a developer who says "well, that's easy" about a non-trivial problem actually follow through and demonstrate it to be true. Normally because they spend 5 minutes looking at the problem and see the complexities multiply like a virus before their very eyes...

You state this as if I've claimed that I'm going to create a Netflix clone just to prove something, which of course is nonsense. Netflix is a large company that has been around for years, yet their solution remains relatively simple for the scale of their operation.

Starting with a patronizing "sorry", looking down from heights, doesn't somehow give your comment more credibility.

There are a large number of services of the user scale of Netflix, with dramatically more complexity, that don't seem to have the heroic issues that Netflix does. Facebook and Reddit both are of a complexity scale multiple-magnitudes greater than Netflix. Someone else absurdly tried to draw Apple and Google in, but again, what they do absolutely dwarfs the entirety of Netflix's operation, to the point that it becomes almost laughable.

Netflix is a very simple application. Scaling it can be difficult, of course, but this is hardly some new grounds. Further, the particularly make-up of Netflix makes it one of the most profoundly scalable platforms going (silos, little to no need for transactional integrity. It is profoundly simple). You don't have to believe this, but your comically patronizing responses just sound...laughably unskilled.

> Netflix is a very simple application. Scaling it can be difficult, of course, but this is hardly some new grounds. Further, the particularly make-up of Netflix makes it one of the most profoundly scalable platforms going (silos, little to no need for transactional integrity. It is profoundly simple). You don't have to believe this, but your comically patronizing responses just sound...laughably unskilled.

Have you actually work with any of their backend engineers, seen any of their infrastructure and why they made the design decisions they did, or actually talked to anyone at Netflix, or is this all just speculation?

I'm sure they'd love to hire you to to greatly simplify their overly complicated architecture otherwise...

this seems like a pretty common fallacy for people to think about an organization from the outside

How is it a 'fallacy'? To be clear, this has nothing to do with the actual streaming (which does not happen from EC2) but instead are the basics like viewing history and categories. It absolutely is the simple part of the application, albeit, as mentioned, at a very large scale. Given that someone else mentioned it, does Netflix even bother with the whole recommendation algorithm thing anymore? My personal experience has been that it gives the most superficial of recommendations.

And if we're being honest, the Netflix app layer is kind of terrible. I could enumerate the problems, discoverability being king, but it is not the compelling part of the service.

The recommendations even contradict their own ratings. Netflix often recommends complete crap to me, and even says I'm likely to rate it one star. It even offers me things I've rated as not interesting or one star.

Meanwhile, app level features like restricting certain types of content isn't available. I want to be able to delete crap from the Netflix catalog when my kids log in.. I don't care about nudity, language, etc. but more about just how dumb some of the cartoons are.

An idea's utility is often orthogonal to its complexity.

Comments like this really annoy me.

What exactly is trivial about building a complicated multisource streaming video service with sophisticated recommendations (plus all the other "little" stuff like payments)?

Google is "trivial." It's just a box for doing text search on a graph database. Apple is "trivial." They just make pretty skins for commodity hardware & software. Etc.

When you consider 99% of everything in your field trivial, maybe you should consider refining your definition of trivial.

Comments that contrive absurd strawmen annoy me, even if they provide the author that righteous sense of striking down an imagined position.

Netflix currently has 94 open engineering open reqs, plus another 22 data/analytics open reqs. Perusing through the list should give you a sense of how complicated their products are: http://jobs.netflix.com/jobs.php

When I saw this internally a few months ago, I said that I haven't been more excited about a new project in a long time. I personally think this will solve a lot of scalability problems people have while letting them keep their favorite database.

It looks very promising. I like the fact that it doesn't only work for AWS environments.

Funnily enough Dynomite was also the name of an Erlang implementation (https://github.com/moonpolysoft/dynomite) of Dynamo that seemed to get off to a very promising start before it was abandoned a few years ago.

Cool, it's something that Microsoft SQL Server (Enterprise edition) offers out of the box (Peer-2-Peer replication). The only problem with P2P replication is that, at large scale, requires massive amounts of bandwidth to keep everything synced, and it gets worse the more nodes you add.

Depends how things are replicated. In Dynamo style replication everything isn't replicated everywhere. Based on consistent hashing, and how vnodes etc are configured, a key lands itself in more than one nodes so that things are generally available. So scale-out in a dynamo-like system happens more gracefully.

Any ideas if this supports Postgres?

Also, I noticed comments about an architecture document here: https://github.com/Netflix/dynomite

Anyone know where the doc they are referencing is located?

It doesn't support Postgres yet but that is on the (longer term) roadmap.

However, since it is open source, contributions are welcome! :)

I don't know if it was intentional but the name reminds me of the character JJ in the TV series Good Times. Awesome show that was. It sounds solid overall but one thing caught my eye, "A client can connect to any node to read the data. Similar to writes, a node serves the read request if it owns the data, otherwise it forwards the read request to the data owning node in the same rack.". I wonder if that could lead to an owning node arbitration storm when more than one node in a rack owns the data?

For projects with similar goals, see also Twitter's Gizzard (abandoned) and LinkedIn's Voldemort.

Also Riak. However, one interesting aspect of Dynomite seems to be offer default support for clustering across multiple data centers with pluggable stores.

Riak (enterprise version) does that, and has pluggable stores (albeit limited to in-memory, bitcask + eleveldb)

Add Mashape's Astronode to the list: https://github.com/Mashape/Astronode-Broadcaster

Perhaps I missed it, but is there any discussion on how their sharding mechanism works? How do they define a shard and where is ownership of various shards managed so a "non-owning node can redirect to an owning node"?

I questioned tbat as well, could be a property of the underlying store (memcache hash). How would mysql sharding work without being application managed?

It's interesting that Netflix decided to write this rather than using Amazon's own DynamoDB. I wonder specifically if DynamoDB was too expensive (as I have found), or was there some other reason?

This is completely different than DynamoDB. Essentially this is a framework for sharding or horizontally scaling multiple datastores (MySQL, Memcache, Redis, etc).

That makes sense. I had only seen the Memcache & Redis datastores being used (and both being comparable to DynamoDB).

Why would you use multiple datastores? Isn't it much more difficult to atomically store data this way? Assume a transaction succeeds in store 1, and fails in store 2, you'd need to roll back the transaction in store 1 after it has committed (!)

Because there is no such thing as a one size fits all datasotre. Some data may make way more sense stored in a relational database, while some data makes more sense as key:value pairs. Some data may be infrequently accessed and stored to disk while others are stored in memory for quick access. Trying to make a single datastore work for many use cases will cause more pain than it is worth.

You wouldn't likely use multiple data stores for the same data. That is, you probably wouldn't have a single system which stores data in both Redis and MySQL.

However, you can imagine different use cases for each of these stores and you can potentially increase operational efficiency by having a common layer which handles the distribution of data for each store.

This is the sort of project you do to get cross-region replication on your existing data stores, many of which look more like PostgreSQL than KV stores.

Keep in mind that they define

DC := AWS region

Rack := AWS AZ

That should tell you that they need multi-region availability, something that DynamoDB will likely never provide.

nitpick, the code check for NULLs at a low level but misses checking NULLs at higher levels like here:

https://github.com/Netflix/dynomite/blob/master/src/dyn_dnod...

Are there any deployment notes/examples?

not really, the best you are going to find so far: https://github.com/Netflix/dynomite

Setup and Installation Wiki - https://github.com/Netflix/dynomite/wiki/Getting+Started#set...

Reply on news.ycombinator.com