RethinkDB 1.13: pull data via HTTP, push data via changefeeds
rethinkdb.com
rethinkdb.com
`r.args` isn't very flashy but is a huge step forward for ReQL.
Removing protobuf makes deploying to certain platforms (e.g. heroku) so much easier.
Promises support for the Javascript driver brings it to the modern age.
And of course changefeeds and `r.http`!
If you've been hearing about RethinkDB and thinking about trying it out, now's a great time to spin the wheels.
FYI the upcoming 1.14 release is scheduled to include a distributed file system and geospacial indexing (probably the most requested RethinkDB features of all time).
If you have any questions about Rethink, I'm here all day to answer questions. (We can also grab lunch or coffee if you're in the Bay Area).
RSVP to the meetup here: http://www.meetup.com/RethinkDB-Bay-Area-Meetup-Group/events...
We'd love to see you come hang out with the RethinkDB and Firebase teams. Even better, give a lightning talk on how you're using RethinkDB! Christina is finalizing the schedule, if you'd like to do a lightning talk shoot her an email (christina@rethinkdb.com).
The relationship between RethinkDB and Firebase is based on conceptual/philosophical agreement about where the world is going (realtime json sync), and that developer tools should be delightful. That's how we know each other, and decided to co-host the event. There is no software integration or anything.
I don't think the companies are really competitors -- Firebase is a service, Rethink is a product, Firebase has much more hands off ops and smaller query language, etc. So the use cases where you'd pick one or the other tend to be quite different.
We just want to give demos about building realtime apps, and get to know people who care about the subject!
I'll bump it up in the roadmap -- sorry it's taking so long to get this fixed. It turns out that everyone only needs 5% of RethinkDB features, but it's always a different 5%. That makes product roadmaps really hard. I think in this sense databases are a bit like word processors :)
The particular issue in question is using an index to filter documents, and then ordering them by another index.
I don't know how I feel about the http command but I am digging the other changes. Keep up the good work.
Seriously guys, thanks for the great product and awesome database experience (PyRethinkORM author here).
r.table('games') \
.changes() \
.filter(r.row['old_val']['score'] < r.row['new_val']['score']) \
.run(conn)
r.table('games') \
.changes() \
.filter(r.row['new_val']['user_id'] == X) \
.run(conn)
(BTW, we have a lot more planned for this)
- It fits! It just seems to work magically well with the rest of
ReQL on so many levels! JSON fits, streams fit, lazy evaluation
fits, even batch prefetching fits! Everything works so
wonderfully and provides such a great interactive experience,
it almost would be silly not to add!
- It makes a use case that's really important to me
personally (adhoc analytics) an order of magnitude easier. Over
the past few months I ran into a few other people that also do
a lot of interactive adhoc data analysis with Rethink, which
redoubled my resolve to add r.http.
- It makes example datasets *much* more elegant. Reading
RethinkDB docs? Just call this command to get the dataset in so
you can play around with a command. (We haven't updated the
examples yet, but we will). That makes the first experience
with the product better, and makes the learning experience much
more pleasant.
(On an unrelated note, I really love your use of 'frippery' -- I'm a bit of a word nerd, and this is a really neat word!)
The disconnect is that we see Rethink in different ways. You're looking at it, in this context, as Rethink + ReQL combining into a "tool for data analysis".
When I look at Rethink I see "Place to put my data".
My viewpoint pretty much ignores ReQL as anything more than "Means to get my data out of the thing". Which, thinking harder about it, is wrong. You guys are looking to make ReQL more than that. My viewpoint is too heavily influenced by the way I think about things like CouchDB.
I normally don't go in for value judgements about people's viewpoints, but I think in this case yours has to be declared objectively correct. Given that it's your product and I'm just a guy that's been an interested observer for a while, but has never actually used it in anger.
So, thanks for helping me get it. Word nerds unite!
RethinkDB looks better than MongoDB in many respects. But how does it compare to Postgresql (latest versions with document support)?
If I were to start on a new project with modest scalability requirements, what should I choose and why?
PostgreSQL also has built-in streaming replication, which is useful and works nicely. But the scenario it covers is scaling reads, not writes. To scale writes, you should shard the data manually.
There are efforts to bring write scalability to PostgreSQL. One of them I'm keeping my eyes on recently is Postgres-XL, which uses statement-based replication that focus on writability and availability. It is a premature product that started recently, though. And also, some easy-to-administer features like automatic failover and configuration propagation (adding/removing nodes) are not the objective of the project. You should adopt another solution, or write it manually.
However, instead of getting the scalability, you lose ACID in RethinkDB - whose impact can be huge on some use cases. But personally I think it is worth losing to gain scalability easily. And there is two-phase commit which is at least doable to imitate transactions, though it isn't elegant at all.
If your application is read-intensive, I think staring with PostgreSQL and adopting Postgres-XL or pgpool later is a good strategy. But if you are to shard the data to scale writes, or you want an all-on-one solution that helps the administrative works a lot, I think RethinkDB is highly recommended.
Postgres is an amazing product, and incredibly stable by virtue of being very mature. You can't go wrong with it and will almost certainly be just fine.
However, you can't quite compare Postgres document support with RethinkDB. You'd really need to play with the product to understand the difference, but think of it this way. You could sort of treat Java as a dynamic language by casting everything to object, it would work just fine, but it is night and day compared to how it would work in Python. The difference between the experience of using Postgres and RethinkDB for JSON data is just as vast. You'd really how to try both before you can viscerally understand how amazing a dedicated JSON environment can be.
RethinkDB is behind Postgres on raw performance and we're still shaking out scalability quirks (see http://rethinkdb.com/stability/) but if you take the long view (a year or so), these will be worked out in on time. I'd encourage you to play with the environment -- it's easy and fun, and see if you like it. Feel free to shoot me an email (slava@rethinkdb.com) and I'll be happy to help you out if you have questions.
Thanks again for your awesome work, guys!
I am a huge fan, keep up the outstanding work!
Feeds are great because you can use them to integrate with other pieces of the infrastructure like RabbitMQ or ElasticSearch, or write reactive apps where clients instantaneously react to changes in other clients. Incidentally, you can use them to easily get pubsub, but it wasn't the original intention.
There are much better pub/sub services out there, so we decided to stick to having a really good feed API and avoid implementing channels for the time being.
You can grab JSON data out of APIs in seconds, filter and manipulate it, and enrich it with more APIs. I've been using `r.http` for some time (since an internal beta) and I now find it invaluable for ad-hoc analysis.
So it's almost as nice as SQL, except bespoke and not specifically designed for interactive data exploration?
Someone who wants to periodically pull in feeds in production code can easily write a short script in their language of choice to grab it and add it to the DB.
It's not the role of the database to be an HTTP client, cron job scheduler, etc. Can it set the Accept header? The User-Agent header? Can it support different HTTP methods? How does it handle retries, and streaming downloads? Does the timer have to be initiated by a script each time or will it remain persistent in the database server? How can debugging info like the last time it ran and the next time it'll run by displayed?
It's not worth building supporting for all of these myriad possible use cases when someone can accomplish this easily in their general purpose programming language.
I understand adding r.http as a basic prototyping utility, but I think it should be left as it is. A timer primitive has no use for prototyping or experimenting. Developer time should be spent on improving core functionality, driver support and performance, etc.
We added `r.http` because it just fits magically well with the rest of ReQL on many many levels. JSON fits, streams fit, lazy evaluation fits, even batch prefetching fits! Everything works so wonderfully and provides such a great interactive experience, it almost would be odd not to add.
Now, we won't add timers just for the sake of adding timers. If we add them, we'll make sure they fit and make the product tighter. There are good reasons to consider a timer implementation in the server:
- Many users have asked for "capped" tables for logging, which
can be implemented as a combination of timers and secondary
indexes (i.e. delete everything older than X on a timer).
- There is already internal timer functionality that we use for
various tasks. It makes sense to evaluate whether it's worth
exposing to the user.
- Before we add timer support, we'd have to add more robust job
control (which many people have already asked for). If we do
add job control, timers feel like they might fit very
elegantly.
On closer inspection of the video: I'm jealous of any company that lets their employees walk around the office with their shoes off. Not only are they super-smart, they're cool too.
(Sorry for this shallow, superficial observation. I'm still learning about databases and programming.)