RethinkDB 1.8: date support, querying nested objects, 8x disk usage improvements | Hacker News Reader

RethinkDB 1.8: date support, querying nested objects, 8x disk usage improvements

rethinkdb.com

128 pointsmglukhovsky50 comments

Kudos to the team; some great releases this summer! (10x insert performance, regex queries, and now date/time and better disk space--all in ~90 days!)

I had never looked in to how rdb used disk space and off the cuff attributed its high usage to something similar to Mongo and allocating blocks ahead of usage.

This was something that went in to the negative column for me when I evaluated using rdb vs Postgres for a JSON dump. 12k somewhat light (but still larger than 250b) records ate up over 70MB of disk space in just over a couple days and I've not looked back. Time to give it another spin!

There is no technical reason why Rethink needs to have high disk usage. We got much better with this release, and will continue to get even better in the long term. (Squeezing more optimizations does get a bit harder, though)

How does rethinkDB's disk usage compare to MySQL and Psotgres?

This is good to hear and will make me try it too. High disk space utilization was something I experienced with mongodb back in the day and that caused me to lose interest (not sure if they're better about it these days). There's probably a trade-off between speed and having blocks allocated but it starts becoming a problem on smaller vps environments.

Mongo got a lot better about reusing disk space in v2.2, if you are deleting from a collection. Still no compression or document field name optimizations though.

Couldn't resist: try tokumx if you want "mongodb with compression". There isn't a speed/space usage tradeoff. Our latest blog post is about long field names: http://www.tokutek.com/2013/08/tokumx-tip-create-any-field-n...

Interesting product. They also have a mariadb engine (different use cases).

Look, I don't have enough experience to be sure on this kind of thing, but it seems like RethinkDB is being developed at a blistering pace. Docker is another one like that. Both impress me, both on their vision and rate of improvement. It makes me happy to know that there are successful projects like this out there.

Yeah, same with the ember.js guys. Churning out releases and RCs under such restricted timelines.

Being a heavy PostgreSQL user I really would appreciate a "How RethinkDB compares to PostgreSQL" FAQ entry.

That's a great idea, we'll be adding more comparisons in the future-- thanks for the suggestion!

Mike, I would love to see that too.

Can you please include the entry on connection management model. PSQL is known for taking process-per-connection + shared memory approach, knowing the canonical and "mothership approved" approach to handle Rethink connections would be awesome.

Thanks!

Thanks, that in the pipeline. FYI, it's quite similar in Rethink -- a thread/process per connection is the best way to access the db.

Can someone who ditched MongoDB for RethinkDB share his/her experiences?

I did a small project so maybe someone that did something larger can weigh in, but I worked with RethinkDB back in the 1.4 era, so a bit has changed, but the biggest difference IMO is that you have to be prepared to fall back to the RQL syntax to do queries more complex than simple get etc. vs. in Mongo everything was in JSON, and when I was using Mogno you couldn't join tables at all, don't know if that's changed and here you can. Otherwise it is pretty similar, since I didn't do any clustering.

Here's a good explanation for why you want to fall back to ReQL rather than write queries in JSON directly: http://rethinkdb.com/docs/introduction-to-reql/

ReQL for the win. It's so expressive, it's composable, I love it.

I feel more confident that the m will be easy to scale if/when my app needs it.

My code that interacts with the API feels cleaner.

I've been watching the great progress since 1.4, but I continue to be disappointed by the overall performance, makes me wonder if I'm not groking something. Here's a gist of a simple ruby script to insert 1000 trivial docs into rethink:

https://gist.github.com/wiremine/6245910

1,000 inserts are taking around 20 seconds. Hardware is a 1.8 Core i5 Mac Air with 4GB Ram and a 128 SSD.

Is this sort of performance expected at this point?

Batching inserts make them faster https://gist.github.com/wiremine/6245910#comment-887640

13.618112323 seconds vs 0.591895712 seconds (on my laptop -- X1 Carbon)

Using soft durability make things faster too, but that's cheating.

You have something wrong there. I just ranthe test suite I ship with my java driver against and I am able to do 2048 documents in about a second. Most of that time is spend in serializing the data to go over the wire and each insert is being done individually and in raw protobuf form. If you send in JSON and you queued up your inserts I imagine you could do even better.

This sort of thing isn't expected. Please check the following troubleshooting entry for slow inserts: http://rethinkdb.com/docs/troubleshooting/#my-insert-queries...

Thanks for the info! Makes sense. Is this performance you're expecting in the long-term, or will this be tuned at some point?

Performance tuning never stops. We have one more release coming up that will squash a lion's share of low-hanging performance fruit. After that, we'll be tuning continuously, but it will be slow, incremental improvement.

Well done guys,I'm especially happy for the Date Object! That was a easy but annoying thing to have to work-around.

Date/time was a hard one but worth the effort. Looking forward to the next updates!

I keep thinking that I should try RethinkDB, but most of my projects involve geodata / geospatial indexing. For this use-case PostGIS and MongoDB have benefits which I can't find elsewhere.

Same thing for me!

We have a github issue open for it FYI (can't find it now because github issue search seems to be down), but unfortunately it won't get done before 2.0.

maybe:

https://github.com/rethinkdb/rethinkdb/issues/1158 ?

Great Job and this is cool to see such progress. Now the long process of implementing all the new verbs in the drivers begins. I am really looking forwards to when RQL is marked as stable.

Actually, ReQL is basically stable as of 1.8. There may be very minor changes before 2.0, but there are no big changes planned any more.

So, is there too much overhead for each table (to create per-client tables since custom_field sharding is not yet available) ?

Tables are a little bit heavy now. I don't know the exact overhead (we should measure), but a table is certainly heavier than a single document. I'm not sure if a table/client is tenable -- if end up doing some tests could you please report the results?

So no new verbs going into the Query Language ? I have been waiting to do much more work around my Java driver until the pace of change slows down a bit.

Yep, that's right. We might make very minor tweaks to existing commands, but there are no new commands planned as of now.

Same I had been working on a Scala one, but had slowed down as I couldn't keep up with the Rethink team's releases.

Lack of native Date support was holding me back previously. Now i don't see any reason for not using rethinkdb. My next project will be using Rethinkdb! It will be nice if you can fix "groupby field names" issue https://github.com/rethinkdb/rethinkdb/issues/386 in your 1.9 release

Keep up the great work. It's a very interesting project, one I'll continue to keep an eye on.

One thing I wish would get added are more aggregators. This would add a lot of valuable use cases, in my opinion. At least one of my mini projects recently would have been greatly simplified had I had the ability to compute max's and min's in the same way as avg's.

You can currently compute max/min through map/reduce. We will still add a command to compute it since it's a common operation.

Thanks for the feedback : )

Are 2D geospatial indexes/queries planned for 2.0 or do they come later?

It probably won't get done before 2.0, but I wouldn't be surprised if it will be one of the first features once we get production ready.

Another great update. Keep it going!

They fixed my bug - yay.

Nice. Can you guys please put a wiki page now?

Do you mean a Wikipedia article? If you mean a wiki for users, we use the Github wiki (https://github.com/rethinkdb/rethinkdb/wiki) to track things like community contributions.

Yeah, a wikipedia article. That is where I go to first, when I hear about a new technology. I know I am not the only one.

What.. version 1.8 and now there is Date support? Shouldn't this kind of stuff be aournd at version 0.01 when we are talking about data stores? You know, the only non-easily swappable piece of our tech stack?

It's really comments like this that make me scared to death of posting a project to HN these days (so I don't). Seems to be so common now.

It's getting quite common for me to just skip reading the user comments after reading the article. This was never the case before; the comments were -almost always- better than the article.

As a sidenote, I cannot even recall when I actually read the user comments on Slashdot. On the occasion where something pops up in the RSS feed; read TFA and move on to happier things that actually make me feel good. :)

I guess I'm not saying anything new. I really wish I had a solution other than asking -you- to please imagine sitting at the receiving end: You get excited to see your project on HN, you click through to comments, and the first thing that was said was ... that.

PS. You can call me a chicken for being "afraid" of these things.

Sometimes people get angry and irritated at stuff. That's ok. The tone is uncalled for, but irritated/angry comments often have legitimate gripes. Rethink should have had dates earlier, we just couldn't get it done.

I agree that people should try and be more kind with their tone, but I wanted to try and encourage you not to be afraid of criticism. People have good memory for good work and no memory for bad work. So if you post an atrocious project here, the absolute worst thing that might happen is that some people will write some negative comments and forget about you ten minutes later. It seems like a worst-case scenario most people can live with :)

It should have been, but date support is way, way trickier than it appears. Check this out if you want to learn more -- https://gist.github.com/coffeemug/6168031. It's bad that dates didn't get in earlier, but it would have been worse if we rushed a bad implementation. Sometimes one needs to pick the least of two evils.

Reply on news.ycombinator.com