danielmewes

Thanks for the info. I'll look into this. The fact that we are running out of memory suggests that we're doing something wrong for this query.

Daniel @ RethinkDB here. We'll release the details shortly. This was running on 12 core Xeon servers with 2 SSDs each in software RAID 0. There were also additional read queries running at the same time as the write queries, and the read throughput that coffeemug posted is the sustainable increase in reads/s that you get when adding an additional server to a cluster. Single-server performance is much higher due to missing network / message encoding overhead.

I realize these numbers alone are still not very meaningful and there are many remaining questions (size and structure of the data set, exact queries performed etc). Rest assured that all of these details will be mentioned in the actual performance report that should be up soon.

We did a couple of scalability improvements in 2.0, but didn't optimize groups and counts specifically.

Would you mind writing me an email with your query or opening an issue at https://github.com/rethinkdb/rethinkdb/issues (unless you have already?)? I'd like to look into it to see how we can best improve this.

We're planning to implemented a faster count algorithm that might help with this (https://github.com/rethinkdb/rethinkdb/issues/3949), but it's not completely trivial and will take us slightly longer to implement.

Understood. We're planning to test with Jepsen soon. This will happen once we have implemented fully automatic failover (at the moment it still requires manual intervention, even though it's usually straight forward). We have a first working implementation, but are still working on the details. It should become ready in the next ~2 months.

See the issue you mentioned https://github.com/rethinkdb/rethinkdb/issues/1493 for progress on this.

Daniel @ RethinkDB here. As you mention, RethinkDB is fully open source so RethinkDB is always going to remain freely available.

Uncertain. The issue is still in the planning phase. We'll only remove range sharding if we're confident that hash-based sharding provides an adequate replacement of course. It's possible we'll keep both.

We'll start developing one pretty soon. You can follow the progress at https://github.com/rethinkdb/rethinkdb/issues/3930

We currently support range sharding by primary key.

Both compression and custom sharding are things we definitely want to do (https://github.com/rethinkdb/rethinkdb/issues/1862 , possibly https://github.com/rethinkdb/rethinkdb/issues/364).

I can't give an ETA on these features yet, but it will probably happen later this year.

Not sure if we will have benchmarks available at launch, but they should follow soon after.

Great to hear! Let us know if you run into an issues or have questions.

Daniel @ RethinkDB here

We're still optimizing the performance of certain operations. There are definitely still a few rough edges, especially with analytical queries as you'd pointed out.

As far as basic operations (inserts, updates, retrieving documents etc.) are concerned, RethinkDB already has competitive performance and we hope to publish official benchmark results soon. As you know, good, comparable benchmarks aren't easy, and we're currently spending most of our time on improving RethinkDB rather than working on those. But they're definitely coming.

This is just a guess, but a query involving COUNT(*) might execute faster with PostgreSQL since it can use an index for making the count basically a constant-time operation, while RethinkDB doesn' currently support this. That is definitely a missing feature on our end.

If you get a chance, I'd be very happy to hear from you about the specific queries and the data you'd been testing, so we can look into those specific performance problems. My email is daniel at rethinkdb.com .

We're generally open for this. However it's a bit of extra work since RethinkDB doesn't currently have a pluggable storage backend API like MongoDB or MySQL do.

How many users are you expecting?

If you go over a couple 1,000 active changefeeds in 1.16, I recommend setting the `maxBatchSeconds` optional argument to `run` to something like 10 (the default is 0.5).

That change should significantly reduce the CPU overhead on the server if you have lots of idle changefeeds. Note that this does not affect how quickly changes get delivered - you will still get changes instantly.

The exact performance will of course depend on what queries you're going to run exactly, the rate of writes etc.

RethinkDB 1.16 doesn't have auto-failover yet, but ReQL admin still helps because you can integrate it with external cluster management tools that you could use to initiate the failover.

We are actively developing the foundations for integrated auto-failover at the moment, so auto-failover is going to come soon.

Changefeeds are currently bound to a database connection, so they will get terminated if for example the application server goes down or there's a networking issue.

We feel like for many queries (especially the ones you find in web applications) that's not a big deal, since you can efficiently re-run them after reconnecting. In other cases it definitely matters, and we are going to add what you describe in a future release. You can follow the progress (or chime in if you like) at https://github.com/rethinkdb/rethinkdb/issues/3471 .

This document explains our basic replication architecture and might be helpful http://www.rethinkdb.com/docs/architecture/

Let me know if you want to know more about a specific aspect.

Hi Chris, great question. There are a couple of advantages actually.

- You don't have to write logic to figure out which clients need to be updated with some data. You just run the queries that you need to to generate the data for the client, and then RethinkDB sends you an update only if the result of that specific query changes.

- Having the thing that modifies the data send the updates to the connected clients becomes increasingly difficult if you have multiple application servers. Then you would need to set up some separate message passing / broadcasting infrastructure if you also want to update clients connected to other servers. RethinkDB takes care of "passing the news around", even in a distributed environment.

- RethinkDB supports changefeeds not just on the raw data, but also on transformations of the data. Not all transformations are supported yet (for example map reduce queries are not), but our goal is definitely to support changefeeds on pretty much any query. Just knowing that the underlying data has changed isn't enough. In a traditional setting, you would still need to either recompute the whole query, or implement your own logic for incrementally updating their results for every type of query you want to run. RethinkDB updates query results incrementally and efficiently.

(I work for RethinkDB)

Have you tried http://php-rql.dnsalias.net/ ? It's pretty much equivalent to the official drivers, and well maintained (usually gets updated within a week after new RethinkDB releases).

Nothing special. We use S2 to compute a covering of a polygon (like a town) by a number of grid cells. If you insert a polygon (or line) into a geospatial index, it will actually be inserted multiple times, with one entry for each grid cell of its covering. Then when you query by a point (e.g. to answer said query, assuming you have a table with all the town polygons), we find any polygon that has a grid cell that intersects with that point and then do a post-filtering check using the actual detailed geometry of the candidate polygon. This is fairly efficient and can be extended to a lot of different cases, though I'm sure there are more efficient specialized data structures for some specific types of queries.

Hi cwmma. RethinkDB actually does use Hilbert curves at multiple scales for indexing geometry. The implementation for this mapping comes from Google's S2 library. You can find the details documented in Google's code: https://code.google.com/p/s2-geometry-library/source/browse/...

We looked into some other data structures before settling for our current approach. Using a space-filling curve turned out to be the least error-prone to implement given that we already have a highly optimized and well tested btree implementation in RethinkDB.

Ok, got it. Guess I mis-represented what WGS84 in my earlier post. What I should have said is that in RethinkDB we only mean a specific reference ellipsoid for computing distances between latitude/longitude coordinate pairs when referring to WGS84.

If you have data in other projections you will definitely have to convert it first.

On a side-note, in case it's of any help: There is a currently undocumented feature that allows you to specify an oblate reference ellispoid other than the one from WGS 84 in ReQL. The way you do that is by passing an object `{a: ..., f: ...}` to the `geo_system` optional argument. The value of 'a' must be the major radius of the ellipsoid in meters, and 'f' must be the flattening of the ellipsoid. This feature is not heavily tested though and should be used with caution.

> The implementation is based on the S2 library (https://code.google.com/p/s2-geometry-library/) also used by PostGIS, which AFAIK does spherical linear interpolation.

Note that RethinkDB actually doesn't use S2 for computing distances. S2 is used for computing intersections and for indexing though. (also as another poster already pointed out, S2 is not used by PostGIS)

> Also, does this support many projections, or just a few.

RethinkDB doesn't do any planar projections of spherical coordinates (if that's what you are asking for). There are many independent tools available to perform such projections if you need to draw a map or convert to a Euclidean coordinate system. As coffemug noted, WGS84 is not a projection but just a reference ellipsoid used for distance calculations along earth's surface. (Edit: Ok, I guess that can be considered a projection as in projection from earth's actual geometry onto an ellispoid. It's not a projection with respect to latitude and longitude though, if that makes sense.)

> Along those lines, are distance calculations done on a geodesic

Yes, along the geodesic between the two points on an oblate ellipsoid. We use the algorithm from Karney 2013, see http://charles.karney.info/biblio/papers/karney13-geod.pdf

> geometry.distance(geometry[, {geoSystem: 'WGS84', unit: 'm'}]) Does this do a correct interpolation of degrees to meters along the path (again based on the projection)

Yes, again using Karney's algorithm (but not by doing a projection first).