Hacking Redis: Adding Interval Sets | Hacker News Reader

Hacking Redis: Adding Interval Sets

starkiller.net

90 pointsjacobparker8 comments

Just to add, the redis code base is pretty clean and straight forward. I was able to digest enough of it to implement a multi-user system based on the redis databases[1] (1 db per user; I've never seen more than one used anyway). antirez and co do a good job at keeping it readable and understandable, which is awesome and I'm sure is part of the reason it's as reliable as it is.

[1] https://github.com/jimktrains/redis/tree/multiuser

That was absolutely our experience as well, the straightforwardness of the code base made it such that we spent almost all of our time writing code and very little figuring out how stuff worked.

The test suite is also great (and I got to learn some TCL), which was a huge help in getting us to a place where we were pretty confident that the somewhat tricky tree-balancing algorithm worked correctly. Basically, I wrote a fuzzer which output redis-formatted tests when it found failing sequences, whittled them down to minimally failing examples, and went back to the code and fixed the test.

Rinse and repeat until the fuzzer would run infinitely, and I had a good test suite and code I was confident in.

edit: the fuzzer https://gist.github.com/llimllib/5527441

I completely agree. One of my more immediate projects is to possibly find a way to refactor some of the work I outlined in the article to make adding datatypes even more straightforward, but I am very concerned on adding complexity to the Redis codebase rather than removing it. Sal has done a great job of making it easy to understand the code of relatively powerful utility.

I thought this thread might be a good place to write about a little project I've been working on for the past few months (in the shadows so far):

I've written an experimental abstract database loosely based on redis (or its base concepts) that is entirely based on a plugin system; and it's written in Go.

I love redis to bits, but it always bothered me that it doesn't have and doesn't seem like it will ever have a proper plugin system. So what I did was take the redis protocol (you use it via redis-cli or any client) and a few of its basic ideas: key=><data struct>, snapshot persistence, the overall command syntax idioms, master/slave replication, pubsub - and stripped everything but the very basic system commands. Oh, and also the concept of single threadedness seemed to be wasted on Go.

The rest is implemented inside isolated plugins that do not need to know about network or disk, and are just responsible for managing creation, serialization and manipulation of data structs. PubSub and such are achieved via Go channels, and locking is done on a key level, with reader/writer locks. The rest is pretty straight forward.

Besides some basic commands (strings, hashes, pubsub, monitor), I've added prefix search and json storage with sub node manipulation, with (relatively) very few lines of code. Although Go doesn't support dynamic loading of libraries, the idea is that you compile the database with the plugins you want, since Go's compiler is amazingly fast.

I haven't opened it yet, and pressure at work prevents me from progressing lately, but if anyone is interested in collaborating on this, I'd be glad to write about it in more detail and/or put it out there.

Sounds interesting, to say the least. Bill (llimllib above) was actually considering writing a Redis implementation in Go as a learning exercise right after we finished with the interval tree stuff.

I personally would be interested in the progression of the project. But yes, finding time for yet another project is always fun...

Thanks. Yeah, it was a great learning experience about Go, and it also turns out to perform pretty well (50%-90% of the throughput of redis, using redis-benchmark). My hope is to use it for a work related project soon to put more development time into it.

But I'll try to write up some docs and put it out there, incomplete as it is, it beats having it sit in a closed repo.

Very cool. Did you guys consider achieving the same functionality with Lua scripting that Redis 2.6 supports? I wonder what are the tradeoffs and if that was much slower.

Yup, Ken didn't mention it but we didn't undertake the work until we had shown that an implementation with sorted sets and an implementation with lua were too slow.

Reply on news.ycombinator.com