HyperDex is designed for bigger-than-memory data.
About benchmark "E", this is a primitive that Redis lacks, so it is in my opinion a bit misleading.
Another question is, in the benchmarks with Redis, does the data set fits inside Ram also in HyperDex DB (or OS cache)? In such a case you are comparing two in memory databases in practical terms, but at the same time you say that HyperDex is suitable for datasets bigger than RAM (in a portion of text near the Redis benchmark), but the numbers that the users will experiment in actual production setups will be different if the data set does not fit in RAM.
Finally, in the benchmark it is not clear that Redis can perform a lot of work in a single operation, for example a variadic LPUSH can insert 10 elements inside a list in the same time an LPUSH does this with 1 elmenet, more or less. If you multiply this for N cores in a stand alone Linux box you get 1 billion operations per second or alike, but I would be not comfortable with writing this in the Redis home page. I mean, if you don't specify very well the methodology this unfortunately is just marketing material without real interest for the field.
I agree that Redis does not have good support for "E", so anyone looking at the results should look at A-D,F.
Both systems performed entirely out of main memory to avoid touching disk.
The Redis benchmark results only came in last night. We'll be writing up the methodology soon. We used the YCSB Redis binding from branch master. I'd be interested in your opinion on how well the bindings map to what is truly best for Redis.
EDIT: (I checked that Hyperdex actually uses threads and multiple cores) If you want a fair comparison you should run Hyperdex on a single core as well, or you can run N instances of Redis (one per number of core) and write the benchmark so that it uses all the instances.
I'll check the YCSB Redis bindings, I never looked at them before. Thanks for the reply.
EDIT2: there are also problems with the YCSB Redis bindings:
1) It basically forces an object-store data model on Redis, so only uses hashes to store objects, and every time an object is stored or deleted, a sorted set is updated as well. This is a possible use case of Redis but not a very idiomatic / representative one.
2) Even for benchmarks not involving searching, the sorted set is anyway updated.
3) There is no pipelining used to alter the object and store the sorted set. Every operation pays 2x the Round Trip Time in the Redis bindings.
4) Even worse, there is no pipelining in the "search" operation, so you may the RTT a lot of times when you do a scan operation with this bindings.
The minimal change to the YCSB bindings is to modify the bindings to use pipelining when possible (almost always, actually). Still I think that an intermediate layer to turn Redis into an automatically-indexing object store does not make sense. Another big problem is that you are comparing multi-cores vs single-core.
So if you really are interested in a comparison between HyperDex and Redis you should pick an use case and model it accordingly with the best tools of both the databases.
What I would recommend is to use the following use cases.
1) Populate the two DBs with 10 millions of hashes, then write a benchmark where 50 clients simultaneously get and set specific fields.
2) Like "1" but increment a field by 10 at every write query.
3) Simulate a leader board where 50 clients simultaneously update the scores of the different "players" in one operation, and ask the top 10 users in the leader bord with another operation.
4) Simulate a capped collection where you always add the latest news in a web site, and you can get the top-10 to show in the home page. Every of the 50 clients should write a single item and fetch top-10 items.
And so forth. Always use 10 million objects and mixed reads and writes with 50 clients at the same time. Write the best code for both the DBs (I can help with Redis).
This time you are truly comparing the two DBs in a real world scenario.
Also, It's extremely difficult to get speedup of 4x on 4 cores, so I'll believe that argument when I see it. It seems to me that with the current implementation of Redis, you'd run into some serious problems with memory management if you run 4 redis nodes on one 4 core machine.
Also, you say: "This is a possible use case of Redis but not a very idiomatic / representative one." Can you elaborate? What is idiomatic? From the front page of the Redis site: "Redis is an open source, advanced key-value store."
Also, on a slightly different topic, I can't seem to find any real documentation about the consistency guarantees of Redis, so I thought you might be able to point me in the right direction. It appears that the master/slave replication scheme in Redis just backs up onto the slaves eventually, but the master immediately returns. Is that true? If a master goes down (disk and all), could a write have been confirmed to the client which is missing on a slave? What replication protocol does it use? I find the documentation lacking in this regard and it'd be great if you could point me to some actual technical specifications. Thanks!