Hacker News Reader

Comment by RyanZAG | Hacker News Reader

10x insert performance improvement seems very small considering this simple benchmark someone did:

http://stackoverflow.com/questions/15151554/comparing-mongod...

  MongoDB: 0m0.618s
  RethinkDB: 2m2.502s

Although as stated in that post, this is likely because of the different fsync() policies between the two databases.

View on HN

    /dev/null: 0m0.001s

/dev/null is clearly the best database.

(If your performance numbers are too good to be true, they might not be true.)

One aspect of micro-benchmarks that is most of the time ignored is that they reveal different default settings various systems come with. And unfortunately after seeing the results, not too many dig deeper to figure them out.

Indeed. And the default settings reveal something about a system's priorities. If you care about your users' data, then Mongo's default settings should be a giant red flag. The revealed priority is seeming fast.

In noSQL world, any DB's default settings should be a giant red flag. Don't get me started about HBase... I lost 2 hours of data this way. :(

That was actually one of the explicit design goals of Rethink -- pick defaults such that users never have to wonder about the safety of their data. I know the folks at Riak are also in this camp, so there are definitely NoSQL dbs that do this well.

I'm not sure Riak is a good example as it's another example of an extremely slow database.

There's also the angle that if it's to offer no performance benefit, perhaps a classic relational database will do it?

I'm curious to hear your thoughts about it.

As someone that uses hbase in production, I'd be interested in hearing the setting you are talking about.

Honestly, I've been very positively impressed with CouchDB so far.

How about "they should be an invitation to learn more about the system" :D? As far as I can tell, in many cases these defaults have almost never been reviewed by devs because most of the early users already knew what tweaks they needed.

I dunno. Personally I'm a big fan of the "sane default or no default" approach. I think that mongo's former approach is irresponsible engineering (I understand they have fixed it recently).

Well, on a PG 9.1.9 instance, which has fsync=on and an older Intel i7 on an Intel SSD, my 8-process program inserting a 120-byte TEXT value into a (SERIAL, TEXT) table, with transactions each doing 10 inserts, manages to finish inserting 100,000 groups of 10 "documents" in 37 seconds. So that's roughly 27,000 documents inserted per second (not sure if they are measuring individual inserts documents or groups of 10).

According to iostat, this does 5000 write transactions per second to the disk.

If you are logging throw-away data, you can use an UNLOGGED table. This brings it to 49,000 inserted rows per second.

If you don't care about your data integrity, and set fsync=off, it goes up to 65,000 and seems to be limited by CPU.

Very cool, thanks for the numbers! A few bits of info on the benchmark in the post.

It uses a single process/thread/connection. Adding more concurrent clients would improve performance, though there is still some scalability work to be done.

Ops/second in the graphs measures number of documents, not groups.

One fundamental aspect of the benchmark is that it uses random keys, which is significantly slower (in any database) than using sequential inserts. I'm not sure how you insert into Postgres wrt to keys.

I suspect postgres has better latency, and since the benchmark in the post is very much latency bound, this would account for the factor of four performance difference. It's something we still need to address. EDIT: 1 million documents / 37 seconds / 8 connections = 3378 ops/second, which is about a factor of four off from 800 ops/second that RethinkDB does on this. (This is obviously back of the napkin math.)

There's lots of work to be done, I'm very much looking forward to publishing authoritative, scientific comparison benchmarks.

For postgres I've been using several threads, each with its own connection, and all going through the same transaction (using the number exported by pg_export_snapshot()) - with this the speedup is quite big (c++ project using libpq).

I expect that if I move to binary format for inserting it might be even faster. I'm basing this since my reader is binary and that made x3-x4 improvements for my data set/computer/etc.

According to that page, the Mongo test was using unacknowledged writes, so not exactly a useful comparison

This is a bit misleading -- the performance difference between 1.6.1 and 1.7 ranges from 2x to >1000x depending on the circumstances (size of documents, durability, whether the client waits on network reply, etc.) This also doesn't account for 10-100x performance improvements on the batch insert algorithm made prior to 1.7.

I suspect that RethinkDB 1.7 would be comparable to Mongo on insert performance workload described in this stack overflow thread.

It was not deliberately misleading - it's simply the top result when doing a google search on rethinkdb and mongo performance. It's a shame that decent, general case database benchmarks are so hard to find.

Sorry, I meant to say that the benchmark in the release announcement is misleading (since it doesn't address all relevant issues), not the link in your comment.

We're working on getting authoritative benchmarks out, but unfortunately good benchmarks are extremely time consuming (much like any science experiment).

it would be great to see comparison with tokutek benchmark on mongodb / version with fractal tree indexes http://www.tokutek.com/2013/05/sysbench-benchmark-for-mongod...

Ok, now try doing 1000 inserts at once on Mongo. OH WAIT, global write lock.

Reply on news.ycombinator.com