Call for comments - RethinkDB wire protocol | Hacker News Reader

Call for comments - RethinkDB wire protocol

rethinkdb.com

37 pointscoffeemug13 comments

See the "required is forever" warning here: https://developers.google.com/protocol-buffers/docs/proto

You may want to make even your conceptually required fields optional.

As an example, if you tried to proxy a newer version of the protocol through an old proxy (assuming it's deserializing), unknown enumeration values for required fields could cause errors.

Thanks!!

Why bother trying to encode the query syntax into the wire protocol? Why not just accept queries as a string, then have some simple way to pass parameters? It seems like encoding the whole query language syntax makes the clients far more complex.

Transmitting a string over the wire would pose a wide range of problems:

1. it's not efficient

2. we'd need to design a complete new protocol (keep in mind that you'd need to include query parameters) which for advanced query language like RQL is challenging

3. with a new protocol we'd need also versioning (for ensuring backward compatibility)

And the list is probably much longer.

alex @ rethinkdb

Thanks for entertaining me :). I'm only talking about the actual encoding of the query itself, not parameters, return types, etc.

Looking at the github page, it seems all the names of things are still being transmitted as strings (of course), so "r.add(1,2)" is unlikely to be smaller decomposed into discrete items (each with at least a 1-byte pb tag overhead).

I'd think it'd be passing a main query string, with placeholders for parameters. "r.add(@a,@b)" or something, then have a parameter block where you can efficiently encode non-string parameters.

Basically, keep protocol buffers and the rest, just don't encode the syntax tree - let the server parse it.

So, basically using SQL prepared statements. Which are fine for simple queries where the behavior is always exactly the same, but become very complicated when you outgrow the "one hour blog" demo phase. It's much easier to build queries out of data structures in most languages, and if you're already building a data structure, there's no reason to convert it to a human-editable text form just so it can be parsed again.

Oh, I see. So in the ReQL docs all those examples aren't really a "language"? There's no underlying "RQL", it's just using a language with some nifty client libraries to build up an internal state that gets serialized. That's a neat approach.

Right. Technically we could write a "language" by just writing a string parser that converts strings to protocol buffers (some day we might do that). But embedding into host languages is so nice, there was no need to do it so far.

Constructing the query as a string is actually much more difficult for clients. Basically it comes down to the difference between having to splice little snippets of code into a string, being careful that you get it on the right side of the parentheses. And just setting a field on a structure where you have no chance of misplacing the value. Not to mention, assuming results came back in the same format client developers would need to write a string parser and with protocol buffers it's written for them.

It also makes it much easier for developers because the server can better detect errors and send more meaningful responses.

I dunno about you, but in my experience, queries eventually need to be generated dynamically, which means having queries that can be externally represented by data structures.

Having a data structure that represents a query in the first place is a pretty good idea.

Even better is being able to go back and forth between strings and data structures.

Hey all, slava @ rethink here. As the post mentions, the new protobuf spec is here: https://github.com/rethinkdb/rethinkdb/wiki/protobuf_rfc_raw...

Would really appreciate anyone's questions or comments!

Maybe consider re-using an existing wire protocol, such as that for Postgres or MongoDB (whatever is closest conceptually to your database)? Or use HTTP? That way you don't need to re-invent a client for every language.

We did (even had a few implementations). Nothing could come close to enabling an awesome user experience as a custom one, so we decided it's worth the effort.

Reply on news.ycombinator.com