LevelDB for Ruby | Hacker News Reader

LevelDB for Ruby

github.com

40 pointsDAddYE14 comments

Interesting.

The snap! feature is something that could be potentially useful, along with the batch feature.

We currently deal with millions of rows a minute (about 10-16k/second) on our platform and need to store, queue and process hashed data across multiple systems. Redis is our main system for it, but some of the data needs to be archived and doesn't need to remain in memory, but still needs to eventually be processed.

Could be good for that. Thanks for posting!

One thing to keep in mind with LevelDB is that it has been created with web browsers and mobile devices as the main use case, NOT as a high-performance data store. If the features of LevelDB appeal to you, however, you may want to check out Basho's fork of it that they have tuned for use in a server environment: https://github.com/basho/leveldb . If this is interesting to you, check out Matthew Von-Maszewsky's recent talk at the Ricon East conference about his work tuning it: http://www.youtube.com/watch?v=vo88IdglU_8

Thanks for posting the follow-up info on tuning!

There's also the HyperLevelDB fork which outperforms the Basho one by quite a bit for random write workloads. Here's a write up about it: http://hackingdistributed.com/2013/06/17/hyperleveldb/

basho and hyperlevel should be two easy targets for this gem, this week I'll plan to adapt the code (if necessary)

This is such a silly meme. It IS a high performance data store, forget what you THINK it is/was designed for or what people say about it in that respect, rely on actual data to see if it works for your data & load. There's no competition if you want to compare it to an external data store it'll beat the alternatives easily. Comparing to other embedded data stores and the 2 main forks (basho and HyperDex) and alternatives like MDB is a different matter and this is where the nature of your data, the way you access the store (single thread? multi?) and your real load are much more relevant than heresay and benchmarks that bear no relation to your actual needs.

OP, I've been using leveldb-ruby[1] for some time. Is yours just an alternative implementation or are there any other improvements I'm missing?

[1] https://github.com/wmorgan/leveldb-ruby

Is an alternative with some features, i.e. snapshots (IIRC), then I'm planning to add custom comparators and pluggable builtin serializers. Another thing is that leveldb-ruby is built in c++, instead mine uses ruby builtin ffi, should be a little slower (benchmarks will come) but easier to maintain and to extend.

Since it uses ffi it should work on jruby, have you actually tried to see if it works well?

Thanks for the link, though I find the information about the project a bit sparse. For example, I'd be interested to know what the expected/intended use cases are for such a database and what exactly does "embedded" mean in this context.

This is the closest I could find: https://code.google.com/p/leveldb/

Thanks for the feedback, I'll add more links in the readme. In the meanwhile I suggest this reading: http://skipperkongen.dk/2013/02/14/having-a-look-at-leveldb/

From the article you linked, the teaser for LevelDB is really:

"It is an embedded database, i.e. one you use directly in your programming language and it runs in the same process as your program. The interface is a key-value store."

Also: http://www.igvita.com/2012/02/06/sstable-and-log-structured-...

Thee embedded DB model can be great for smaller websites. I've run Tokyo Cabinet as the sole DB for a Sinatra webapp for several years now and it's been hassle-free. It also made it easy to migrate servers.

Reply on news.ycombinator.com