Redis cluster, no longer vaporware
antirez.com
antirez.com
The article itself states that Redis Cluster will lose acknowledged writes during partitions (no P), will be eventually consistent (no C), and won't be available during partitions (no A).
On the other hand, it sounds like Redis Cluster offers few hard guarantees; instead, it promises that failures should be rare 'in practice'. Which is a fine thing for a tool to do, of course, but it makes things less amenable to the kind of stress-testing Jepsen does -- since running inside Jepsen's little universe is about as far from normal operation as you can get. If you already know that a system can fail in a certain way, getting Jepsen to reproduce that failure tells you very little.
If you'd like to make this kind of testing possible, it would be useful to state as many 'positive' rules as possible, which Redis Cluster should always respect -- things like "if a majority of nodes are fully connected, they should always accept writes" and "an unpartitioned cluster should always agree on the same value" -- alongside the documentation on ways it might fail. This way, clients can be assured of the 'bare minimum' that the system supports, and tools like Jepsen can give you more useful information.
It's great to see that Redis has a official story for clustering / failover out; like you said in the post, the worst distributed systems are the ones you have to rewrite every single time. It's going to be interesting watching this evolve.
It's not too hard to get memcached running for cluster of machines, nor is it really all that difficult to get started with cassandra for basic applications, although I haven't done it. I've heard ReThinkDB wants to be a serious contender and that they've tried to solve these problems, but I haven't read about how great it is nor have I heard anyone use it in production so I don't know where it stands.
Async replicated systems can be (but are not necessarily) AP: each system can process requests independently of the other, and they'll get consistent eventually (how they arrive at this consistency is different between systems, and is not always "correct" for the general use case).
Sync replicated systems can be (but are not necessarily) CA: as long as replication is not broken they are available and consistent. These systems have the upside in that they allow you to serve read-only data during partitions, but have slow writes because of the sync replication.
Both types can have features removed, for whatever reason. For example, you could have a sync replicated system that doesn't serve any requests when the replication is broken. Or you could have an async system that doesn't provide a mechanism to correct inconsistencies (a la MySQL async replication). What you describe sounds like an imperfect version of the async system: not highly available, not consistent, and not partition tolerant. Given that better AP systems exist, ones that actually provide things like availability during partitions and eventual consistency mechanisms/guarantees, will Redis Cluster eventually support these features?
Edit: please don't take any of this as criticism. Redis is an awesome product that I use and rely on often. Thanks for all the great work.
[reverting to non-standard, dummy definitions below just to explicitly define some things]
Redis Cluster is available as long as (50% + 1) of the master nodes are reachable by each other. So, 15 masters will be more "available" than 3 masters if they are deployed across a sane network topology. Your masters will have replicas as well, so if a single master instance fails, a replica for that master will be elected to be the new master.
Redis Cluster is consistent in that each replica is an exact copy of its attached master (by default, each master owns N/16384 of the keyspace, where N is the total number of masters). You can lose writes if the master doesn't have time to replicate before the master fails, but that's the nature of async replication. (You can request your Redis client "wait" for at least N replicas to receive your commands before the command completes; this gives you some assurances.) It's possible for you to get an "inconsistent read" if you read from an async replica after writing to the master, but you have perfect read-after-write consistency if only talk to master instances.
Redis Cluster nodes on the minority side of a partition will, by default, deny reads and writes until they re-join the cluster. Redis Cluster prefers data integrity over availability by default since Redis Cluster has no merge operations. You can optionally allow the minority side to keep accepting commands, but that definitely will not guarantee any consistency of your data when the whole cluster reappears.
Given that better AP systems exist, ones that actually provide things like availability during partitions and eventual consistency mechanisms/guarantees, will Redis Cluster eventually support these features?
The one limitation for each of those features is: Redis is an in-memory database. Riak uses multiple KB of metadata for each object in the DB. With Redis, it's common to have tens of millions (or hundreds of millions) of keys on one server.
Availability during partitions with eventual consistency (if you want more than last-write-wins) requires CRDT-like things, which requires metadata, which requires more memory usage per-key.
Full and usable availability during partitions is nice to think about, and it would be great to one day have the option to maybe have in-memory Redis "better data consistency for smaller, manageable datasets" with CRDT merge operations so you get a Reis-dynamo type thing.
But, since Redis Cluster is already 4 years under development, it's not worth burning brain cycles on until real users start using Redis Cluster and we can see where the needs-vs-features plot falls. (plus, the issue backlog for other Redis development improvements is about 100 tasks long, so speculative improvements have to fall in line with the balance of Cluster vs. Standalone vs. Master-Replica vs. New Commands vs. Improving Existing Commands vs. Bug Fixes vs. User Contributions vs. Doc Updates vs. Evangelism vs. ...).
basically we can rule out synchronous systems: product mismatch from the POV of Redis. We are left with AP systems. AP systems trying to get as good as they can from the POV of Availability, and the consistency model provided (even if not "C"), do these two additional things that Redis Cluster is not able to do:
1) They are available in the minority partition. Redis Cluster has some degree of availability, but not to that extend. True "A" means, serve queries even if you are the only node left.
2) They provide more write safety, unless you use it with last-write-win strategies or alike.
"1" is actually related to "2". If you merge values later, you can accept writes even if the node has no clue about what is the state of the rest of the system.
About "2", if you check around, you'll immediately figure out that there are no systems available that are AP and can easily model Redis data structures with the same performances, and having the same number of elements per value. This is because to merge huge stuff is time consuming, requires a lot of meta data, and so forth.
Now there is another constraint, that I don't want a system with application-assisted merging. For example recent Riak versions provide data structures with a default merge strategy, so we have examples of similar stuff. I can assure you that there is either meta-data needed to merge, that would make Redis a lot less memory efficient, and sometimes, there are simply no good strategies. For example take the "Hash" type. There is a partition and in one node I set as "name" field of user 1234 "George" and in another node "Melissa". There is no better merge strategy than last-write-wins, basically.
Now, that said, there are certain things that don't add major overhead, AND, improve write safety. For example if the connection is declared as "SAFE" we could cache all the SADD commands (and others having similar properties) until they are acknowledged by all the nodes serving this key. Unacknowledged writes are retained as a "log" of commands, and are re-played when the node is turned into a slave of a new master.
So there are plans to improve the consistency model, but it is unlikely that we go towards the Riak model, but the good thing is, there are stores like Riak that are exploring this other way of doing data structures without application assisted merge functions, with per element meta data and so forth.
IMHO what there is to ponder is that at the end of the day, is that a distributed database is the sum of what it offers when there are not partitions, and what it offers when there are partitions. Often to improve the latter requires to give up something in the former: simplicity, space efficiency, data model freedom, ...
Do you have any idea how many people run master/slave mysql? That's async and will definitely give you data drift over time. There's an entire suite of Percona tools to detect and repair broken mysql datasets because mysql breaks data so often due to replication problems.
The world is much more async and much less consistent than anybody realizes (cash machines are even async), but everything still works pretty much okay. The last line of defense for solving irreconcilable consistency problems is customer service.
Redis/Mysql has it's problems, as does all databases. However, as a practitioner it's important I know the shortcomings of each, so that I can know exactly what a certain database is offering in terms of safety and whether it's a good fit for my use-case. Antirez being open about them is a huge +1 in my book.
Data drift or loss of acknowledged writes isn't correct for a wide range of use-cases. If you know the limits, you can choose the incorrect system, and deal with the fallout when that "rare" event does happen.
Very correct! Most of the mysql problems derive from it being statement replication and not log shipping. A proper postgres replication setup will be less prone to having data inconstancies.
It describes a huge proportion of the data I've ever worked with, because there are massive amount of use cases where what you are working with does not have to be (and as the systems scale: rarely is) the single source of truth for that data.
Redis beats memcached the moment you need more than a bunch of blobs.
E.g. our in-house capacity monitoring uses Redis for a ~1 hour time series view of our systems. We don't care about data loss - the odds of losing the data within an hour of an outage we need to care about is small enough to be justified and if that ever becomes a concern we can run two and split our updates between them. For long term storage, we migrate rolled up data to couchdb at the moment (doesn't matter what - we could use anything really; it's rarely queried and basically to let us get an occasional longer historical view to budget for resources etc).
We don't particularly care about integrity as long as it's "right most of the time" because the data gets constantly corrected, and precision in the averaged longer term numbers is not particularly important either (I want to be able to project when we run out of disk, for example - I don't care if disk usage is at 96% or 98%, as both are way too close for comfort).
Yet memcached is less attractive because it means far more book-keeping in the app. With Redis we encode part of the information in the keys (keys indicate which level of roll-up the data is at, the name of the data item, and the starting timestamp of the period the data is for) in a way that lets us easily retrieve a list of keys to do the roll-up. With Redis' Lua support we could probably do even more of the roll-up process in Redis itself, but I haven't looked at that yet.
There are a lot of apps like this, where your data volume makes hitting disk annoying (this replaced a Graphite based system - Graphite was thoroughly trashing an expensive disk array and still regularly getting too slow to us; the Redis based system uses ~10% of a much slower machine) and where the cost of losing some time window worth of data is low enough not to matter because it just means a blip in a data stream that is rapidly obsoleting itself.
> This article is part of Jepsen, a series on network partitions.
Covers MongoDB, redis, RabbitMQ, among others. All articles start with "call me maybe", thus the Jepsen reference.
I had alwas just assumed Redis released this a while ago and that is why AWS started to support Redis along with Memcache in their clustered ElastiCache product
http://aws.amazon.com/about-aws/whats-new/2013/09/04/amazon-...
Redis Cluster on the other hand orchestrates multiple redis instances to function as one, thereby preventing the need for redis servers that are very large and preventing the need for presharding[1].
[1] http://blog.zawodny.com/2011/02/26/redis-sharding-at-craigsl...
They definitely don't use redis cluster, because redis cluster requires explicit client-side support. That support is only available in a limited number of clients today.
Plus, despite the name and where it's hosted, Twitter never used it in production for Redis. Redis support was just added "because we can."
It probably works for very direct use cases, but it's one of those things that could probably bite you very severely under edge cases, and you'd be left on your own to figure out what happened.
Then there's the massive problem of "everyone" using MongoDB and we know how much pain _that_ sucker causes. Playing popularity contests with software isn't useful. People install software either because of social reasons ("I don't know how anything works, but I hear everybody is using X, so I'm going to use X too!") or because they need the functionality ("redis is AMAZING at 30 different things!").
If install base at well known companies were a valid measure, we'd all be using Windows laptops while talking into our Kin and squirting our Zunes all over each other.
@antirez Could you elaborate on what these properties are and your thinking behind why they're more important than AP?
Thank you.
I was hoping for a CP system with sharding + replication.
Without which we can't have things like multi-key atomic operations etc...
Will wait and see though, maybe things are better than they seem.
I have done something similar with Zookeeper recently (through I wrote it in Ruby so it's terribly slow).
You would be surprised how little CP stores are actually available. There is an abundancy of AP stores but very very few useful CP stores, especially ones with shared-nothing architecture.
You don't need for every query to traverse the consensus algorithm. It's sufficient to use the consensus algorithm to agree on the master shards for each chunk of data and have sync replication.
Considering Redis is designed for 100% in-memory workloads if you are operating on a low latency network this is more than fine. Especially because this behavior would only be required for writes, reads could be served from either the master or a sync replica (and in theory if you had configurable read consistency a possibly out of date async replica).
I guess what I am sad about is just the lack of CP stores that are useful right now, I probably just need to suck it up and start writing my own.