ElephantDB: A database specialized in exporting KV data from Hadoop
github.com
github.com
The main thing is it gets rid of the painful step of having to set up another datastore, run MapReduce, take the result and store it in the database. Hopefully due to it being a purely read-only datastore solution it can also be highly performant.
The main downside is that currently they don't allow updating the data simply (though they mention they're planning to in the future). Currently to update the data you either need downtime or switch between two rings of data and update them one at a time.
I'm wondering if this was made for use in production somewhere (as many large data websites could appreciate this sort of DB {Cloudera's example site TrendingTopics[1]}) or if this is just a side project?
Edit: The primary author of ElephantDB seems to work at BackType so I can see how they could theoretically use this sort of tool.
[1] http://www.trendingtopics.org/ [2] http://www.backtype.com/
Will have lots more on our tech blog next week.
Adding background swapping support to the db is high priority for us, so that downside should go away soon.
ElephantDB is comparable to Voldemort's read-only mode. ElephantDB has additional support for incremental updates, and ElephantDB has support for consuming an ElephantDB datastore on the DFS in a MapReduce job (without touching any EDB servers serving random reads).
ElephantDB has an extremely small code base (about 2K LOC), and its highly specialized nature makes it simpler to configure than Voldemort.
I don't consider ElephantDB and Cassandra to be comparable. ElephantDB disassociates the creation of a datastore from serving it, whereas with Cassandra you'd be writing to a live database. You can create, update, and read from ElephantDB datastores using only Hadoop. This disassociation prevents your batch jobs from affecting the performance of reads to the db.
The core difference is that Voldemort read-only storage uses a custom storage engine, based on memory mapped files (see the blog link jbellis posted) which allows a very high data to memory ratio (without having to worry about HotSpot's garbage collection, while excellent, still has issues with heaps larger than 20-30 gb). The data and index files are both built as part of the Map/Reduce job in Hadoop. The files are fetched in parallel from HDFS to Voldemort serving nodes, swapped "atomically" (atomic relative to a single node, not the cluster), and the process of fetching these files warms up the page cache as these files are written to disk. However, that makes implementing incremental updates tricky.
Of course, being part of Voldemort adds additional features e.g., being able to collocate it with read-write stores, using Voldemort's pluggable serialization support, failure detection, existing clients (for Java, C++, Python, Ruby), etc..
We use the read-only storage engine heavily for such features as People You May Know (Voldemort push being after around 100 chained Map/Reduce jobs), Skills, parts of our recommendation engine (the more real time features use read/write store) and more. The average latency is consistently within a few milliseconds: upper bound being the time it takes a disk to perform a single seek (if the data isn't in the cache). We've current improving ability to expand clusters with read only stores without re-pushing the data: my coworker Roshan was able to even make use of sendfile() when shipping data from one node to another to make this expansion fast with large volumes of data. We're also working on changes to the index format which will make it easier to implement incremental updates.
In addition to LinkedIn, the read-only storage functionality / Hadoop integration, is also used by several interesting startups, namely DeepDyve and Mendeley as well as at least one big company (not sure if I'm allowed to give out their name).
Bottom line: look at your requirements and see which solves your problem best. Distributed storage systems aren't a zero sum game.
So why would I use ElephantDB instead of Voldemort read-only storage?
Incremental updates is the current feature Voldemort lacks. If that's a must for you, use that. We are hoping to that feature at one point, but it required changes to the index format (which are a recent feature).
That said, a complete push of data into Voldemort is quick: the whole process is network bound (data is read sequentially from HDFS and written sequentially to Voldemort).