When to Redis

Related: Redis 1.1 (currently in beta but ready for some test) implements a sorted type supporting range queries. It's pretty useful when the need is to take million of items sorted with very fast top-N items operations or items with score between min and max, especially in presence of continuous updates to the items scores.

If you are interested check http://code.google.com/p/redis/wiki/CommandReference (All the commands prefixed with "Z" are about the Sorted Set data type). Btw both insert / remove / and update-score operations are O(Log(N)). Top-N items is O(N) since the list of items is already sorted.

Redis 1.1 will be released as stable in one or two months at max.

For data structures and algorithms geeks: the sorted data type of Redis is implemented with a skip list, hacked to actually build a doubly-liked list, with the backward pointers being only at level 0 (we have reverse range operations, ZREVRANGE). Every element is also taken inside an hash table, so that it's possible to update the score of an element already inside the sorted set in O(Log(N)) time because we can get the old score in O(1) from the hash table, and then find the element in the skip list.

Even when the scores are not distributed but there are large clusters of elements with the same score, the update is still O(Log(N)) thanks to some interesting trick.

Redis is particularly useful for implementing simple message queueing systems where the full functionality or power of something like RabbitMQ is not required.

For example, when parsing incoming data into a database, push incoming data onto one end of a Redis list, and have a seperate worker process popping them off the end and into CouchDB, MySQL or whatever.

The fact that Redis list operations are atomic mean that multiple worker processes (possibly on seperate physical boxes) can process jobs simultaneously.

Result: the internet is entirely decoupled from both the hard drive and the database, and extremely high performance is possible.

Twitter Streaming API, anybody?..

There was a post earlier in the week about why Redis was "awesome" and a "lifestyle" and I posted a kind of snarky comment about why posts like that scare me away from Redis.

After this post, I had an hourlong discussion with my lead dev on swapping out memcached for Redis so we could use it for our job scheduler. This post makes an excellent, compelling case for Redis.

Just an interesting example of how two different evangelistic approaches can play out. I don't know if I'm representative of "normal people", though. (Don't say it.)

Redis is really cool, I can think of lots of uses for fast and lightweight and there's at least one project where it would have saved me a lot of effort if it had existed then!

Watch out for the Python library though! It doesn't even try to sanitize key names so malformed names cause all sorts of problems (including executing arbitrary commands). Plus unicode strings with non-ascii chars cause it to blow up.

All very easy to fix of course, as soon as I get a chance I'll submit a patch if it hasn't already been done by someone else by then.

Is Redis stable enough to replace using a database as persistent storage. There seems to be some indication that it is, but then there are paragraphs in the documentation that makes it seem like it's not. Particularly if your database size exceeds the machine's memory then you could lose data.

Is one of the goals of the project to replace the need to use a database? Or will it always be more suitable cache or worker queue replacement?