Hacker News Reader

Comment by rurban | Hacker News Reader

> however parsing is likely not going to dominate your application, so other factors should influence your decision instead.

Exactly. If you need performance, you'll use Cap'n Proto or FlatBuffer, which use a native binary interface, so you don't need to create and copy objects, you just map them them in from IO.

View on HN

Sadly, capnproto's awesome RPC system doesn't appear to support streaming, though I do believe gRPC supports FlatBuffers.

> Sadly, capnproto's awesome RPC system doesn't appear to support streaming

Sure it does. You can implement "streaming" in Cap'n Proto by introducing a callback object, and making one RPC call for each item / chunk in the stream. In Cap'n Proto, "streaming" is just a design pattern, not something that needs to be explicitly built in, because Cap'n Proto is inherently far more expressive than gRPC.

That is, you can define a type like:

    interface Stream(T) {
      write @0 (item :T);
      end @1 ();  # signal successful end of stream
    }

Then you can define streaming methods like:

    streamUp @0 (...params...) -> (stream :Stream(T), ...results...)
    # Method with client->server stream.

    streamDown @0 (...params..., stream :Stream(T)) -> (...results...)
    # Method with server->client stream.

Admittedly, this technique has the problem that the application has to do its own flow control -- it has to keep multiple calls in-flight to saturate the connection, but needs to place a cap on the number of calls in order to avoid excess buffering. This is doable, but somewhat inconvenient.

So I am actually in the process of extending the implementation to make this logic built-in:

https://github.com/capnproto/capnproto/pull/825

Note that PR doesn't add anything new to the RPC protocol; it just provides helpers to tell the app how many concurrent calls to make.

Interesting. I do tend to favor protocols that are a bit lower level and more flexible for composing higher-level functionality. Though this does sound pretty complicated to implement using only what capnproto offers right now. Would there be a way to jury rig "request(n)" backpressure as described in reactive streams[0] (also implemented by RSocket[1]) on top of capnproto? That's what I'm using for omnistreams and it's proven very simple to reason implement and reason about.

[0] https://github.com/reactive-streams/reactive-streams-jvm

[1] http://rsocket.io/

[2] https://github.com/omnistreams/omnistreams-spec

Actually the more I think about it I don't think it would work, since request(n) assumes the producer can send multiple messages for each request.

Reply on news.ycombinator.com