Tokyo Cabinet DB: Beyond Key-Value Store | Hacker News Reader

Tokyo Cabinet DB: Beyond Key-Value Store

igvita.com

135 pointsigrigorik21 comments

I have built a distributed key-value store on top of Tokyo Tyrant (which is a network interface on top of Tokyo Cabinet). I have also tried to use memcachedb (that's based on Berkley DB) and Tokyo Tyrant outperforms it - especially with large amounts of items. memcachedb has also problems with log (it grows furiously large and db checkpoints stalls the database for several seconds).

One can also script Tyrant via Lua (which is pretty amazing) and there's master-master replication. Here's how to implement incr via Lua: http://paste.pocoo.org/show/103848/ I have currently extended it with a fixed list and Mikio (the dev of Tokyo *) has implemented an inverted index for search.

Currently I have my key-value store in production (it has around 1 million key-values) and the performance is pretty good.

I will release it open-source, it's only Python now, but the code is around 200 lines + a consistent hash library implementation ( which I implemented in <50 lines of code http://amix.dk/blog/viewEntry/19367 ) so it should be possible to port it over.

Is there something like CAS of memcached? It's fine to live without locking but low-level atomic functionality is essential.

There's global locking and key locking. I currently use key-locking when updating.

You should be able to implement CAS in Lua I guess, but it isn't supported from the start.

Key-locking makes sense, I'll give it a try. Thanks for the info!

Sounds impressive and cool. I wonder about Table engine performance though.

My experiences with Berkeley DB (using the native C API) were positive, from a performance point of view, largely because BDB has no built-in query support. When I wanted to look something up, and I had no index on the desired value (BDB calls indices "secondary databases"), I always knew I was doing a linear table scan, because I had to write the loop myself. No SQL meant that inefficient queries could not hide in plain sight.

Berkeley DB trades off the flexibility of SQL queries for (1) excellent lookup performance in simpler cases, and (2) the ability to mix relational and object-based data storage, reducing the famous impedance mismatch of relational data modeling. For some applications, this trade-off works beautifully. I daresay that it holds true for most web apps which really don't need a relational data store, and either outgrew pure in-memory storage or just sensibly want to persist their state to disk. (Hint: if you use an ORM, then you probably don't want a relational data store.)

For some other apps, the lack of SQL queries hurts. I once worked on a trading system implemented on top of a proprietary engine which did not support arbitrary queries. It made writing even trivial reports ("list all transactions done two days ago by trader X using instrument Y") unnecessarily painful. (Of course, on a typical RDB with a non-trivial schema and a large dataset, these arbitrary queries would take 20 minutes to run, so catch-22.)

The Table engine for Tokyo Cabinet seems to do everything implicitly. Fast and arbitrary queries. If it works on the kind of schema-less free-form data the author refers to, and has working master-master replication, that sounds almost too good to be true.

Some performance benchmarks: http://gist.github.com/64391

They're not conclusive, but give you an idea of relative performance.

For Tokyo Tyrant there's a Python library: http://code.google.com/p/pytyrant/

It uses the binary protocol and it's written in a very simple and very clean way - and it should be production ready.

I would not recommend using the memcached protocol with Tyrant (it offers less operations and it isn't binary).

I'm currently using Tokyo Tyrant with Rails and the Memcached client (in production). It works great for my needs though I don't need to story binary data. Overall, TT has been extremely fast and stable. The dual master replication is a nice bonus.

The article is mostly about the Ruby bindings, but Tokyo Cabinet is written in C99 and has bindings for most of the usual suspects. English language specifications are here: http://tokyocabinet.sourceforge.net/spex-en.html

Just as an aside, using Ruby to access the db server via Net::HTTP should limit the performance to at max a few dozen http requests per second per (Ruby) process.

I think the more http is used to bridge web architectures, the more critical this aspect will become for Ruby.

It's safe to say there's some reasonable replacements already, and there are some smart guys working on better ones.

http://www.pauldix.net/2009/01/ruby-http-client-library-perf...

But, if you didn't know better, yes, Net:HTTP is going to negate the amazing speed of Tokyo *.

Haven't used it, but Erlang bindings: http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and-mnes...

Those erlang bindings are very immature. I wouldn't use them in production.

Good to know, thanks.

So, this is a BDB clone without the years of testing of BDB? That's exactly what I want for my critical data.

I am really confused as to why one would implement this in C. One pointer math problem, and boom, gigs of data are corrupted. If you are going to reimplement BDB, you might as well do it in Haskell or OCaml and buy yourself a little type safety. (Incidentally, this is on my TODO list, but BDB works well enough for us at this point.)

The table DB type and Tokyo Tyrant for network access seems like a pretty big deal to me. The full text search seems pretty damn awesome too. With regular backups it doesn't seem that risky for certain things.

This is really easy to add on top of BDB, though, that's why I ask.

(As an example, we implemented KiokuDB on top of BDB: http://www.iinteractive.com/kiokudb/. We haven't added full-text search yet, but we plan to do it soon.)

Note that this is a pretty old project really...

It is at least a second generation db project by the same author. It's a continuation of QDBM by the same author. I think QDBM has been around since the year 2000. Tokyo cabinet has been around since 2006.

At this stage it is getting quite a lot of testing, and has been around for years.

As someone bitten by lots of BDB bugs I think tokyo cabinet rates fairly highly in comparison.

There is also a full-text search system for Tokyo Cabinet called Tokyo Dystopia (http://tokyocabinet.sourceforge.net/dystopiadoc/)..Has anyone written Ruby API for this?

Was wondering whether it be a superior option for Rails search ahead of Xapian and Sphinx.

rufus-tokyo has the ruby interface: http://github.com/cheapRoc/rufus-tokyo/blob/7e524947825b6f7f...

any PHP API??

Reply on news.ycombinator.com