Comment by kentonv | Hacker News Reader

Comment by kentonv | Hacker News Reader

Parent Full thread

> A silly question: do you forsee any issues getting Cap’n Proto 0.8 to work over a completely locked down Docker environment that only allows HTTPS proxy as the connection between nodes that use Cap’n Proto RPC?

If it supports WebSocket, it should be relatively easy to layer Cap'n Proto RPC on top of that. Alternatively, some proxies allow full-duplex HTTP (request and response bodies streaming simultaneously), which could also be enough to bootstrap a connection on top of -- but in practice that tends to run into a lot of problems.

Otherwise, that's tough. HTTP is fundamentally a one-way, FIFO request-response protocol, whereas Cap'n Proto is multi-directional and asynchronous. Starting a separate HTTP connection for each call -- with connections initiated in both directions -- would be pretty ugly and have lots of issues with synchronization and routing.

> PS: Thoughts on why Cap’n Proto did not win over MessagePack? MessagePack had a better JS implementation? Is it the convenience of not needing to define a schema when using MessagePack vs Cap’n Proto schema requirement?

I don't really consider MessagePack a direct competitor to Cap'n Proto. It's more of a competitor to JSON. Schema-driven vs. non-schema-driven changes everything about how you use a serialization.

A more apples-to-apples comparison in Protobuf. Protobuf is much more popular for a simple reason: It has had a lot more engineering investment, leading to mature implementations in more languages and lots of great tooling that capnp doesn't have (yet). No amount of clever design can beat that.

> If it supports WebSocket, it should be relatively easy to layer Cap'n Proto RPC on top of that.

How would the new streaming functionality be implemented over WebSockets? WS has no flow control. You can check bufferedAmount but I found it to be fairly useless[0]. Maybe it's improved in the last 1.5 years or I was using it wrong.

> HTTP is fundamentally a one-way, FIFO request-response protocol, whereas Cap'n Proto is multi-directional and asynchronous. Starting a separate HTTP connection for each call -- with connections initiated in both directions -- would be pretty ugly and have lots of issues with synchronization and routing.

I think you could get a long way using HTTP/2 and server-sent events.

[0]: https://github.com/websockets/ws/issues/492

EDIT: added link to bufferedAmount issue

> How would the new streaming functionality be implemented over WebSockets? WS has no flow control.

Sure it does. WS is just a framing protocol on top of a regular TCP connection. Though it sounds like you're not talking about the protocol so much as the JavaScript API, which perhaps doesn't give you enough visibility into the underlying TCP socket state.

But a BBR-like algorithm could still work. Basically (massively oversimplified):

1) Determine the connection latency based on the fastest response you've seen.

2) Determine the connection throughput by tracking the highest throughput you've ever seen.

3) Set your window size to be a little bit more than latency * throughput.

This "should" saturate the connection with just a little bit of buffering. If it doesn't saturate the connection, then your measured throughput will go up until it does.

This way there is no need to ask the OS or browser to tell you how much is buffered...

Disclaimer: I have yet to actually implement something like this. Obviously, it gets tricky in the details.

> Sure it does. WS is just a framing protocol on top of a regular TCP connection. Though it sounds like you're not talking about the protocol so much as the JavaScript API, which perhaps doesn't give you enough visibility into the underlying TCP socket state.

Every WS implementation I've ever seen is nonblocking on both sides. Are you aware of any that aren't? I'm not even sure the spec allows for that.

But yes you are correct that you can implement more advanced algos on top.

EDIT: Sounds like blocking implementations do exist, but unfortunately I'm constrained to a browser environment, and all the browsers are nonblocking.

Heh. The WebSocket implementation I wrote for KJ does in fact provide backpressure (the send() method returns a promise that resolves when it's a good time to send the next message). I guess I'm surprised to hear that most don't...

I think it's more likely I've just been too deep in JS-land. Does KJ send() do any internal buffering, or wait for the OS to tell it to send more?

I've been looking into streams in detail recently because I'm preparing for the OpenJSF exam... maybe this is relevant to you too: https://nodejs.org/es/docs/guides/backpressuring-in-streams/

Thanks for the link. I'm actually quite familiar with that article. It's been very useful for me when designing omnistreams. You may find some of the links on the bottom of this page useful:

https://github.com/omnistreams/omnistreams-spec

It waits for the OS.

off the top of my head https://godoc.org/golang.org/x/net/websocket has a synchronous read/write api

Does that block until messages are actually transferred, or just shove them in a golang buffer and return? Now that you mention it I feel like maybe I have seen blocking implementations in golang and Python a while back, which would make sense.

Skimming the implementation, there is buffering, but it looks like it's bounded, so tight looping Write() you should still end up blocking on the OS at some point.

Makes sense. Thanks for correcting my misconception!

> HTTP is fundamentally a one-way, FIFO request-response protocol...

I have to disagree partially with this... HTTP allows a server to start streaming a response back as soon as it has received all header fields from the client, and since HTTP/1.1, it can stream back an infinite sequence of chunks, which the client can be doing at the same time - effectively resulting in a two-way communication channel as long as the underlying TCP connection allows that (which normally is true, unless you run into bad proxy implementations in the middle). Server-sent events is a incomplete implementation of this idea. I have used this in the past , before websockets, and I fail to see the advantage of using websockets over this.

You're quoting me from two posts up, but I also specifically called out full-duplex HTTP in that post.

HTTP/1.1 does indeed support full duplex. However, most proxies don't. E.g. if you put nginx in front of your application server, you might lose the ability to do full duplex streaming right there. Or, if your client is behind a normal forward proxy, they might not be able to get full duplex. But, it can be pretty hard to figure out what's wrong. So, in practice I don't recommend relying on HTTP/1.1 full duplex. To be honest, I don't recommend HTTP/2 full duplex either, because HTTP/2 is just an optimization applying to individual network hops, and you still have the same proxy software in between screwing things up (and maybe even downgrading the request to HTTP/1.1 for the next hop).

Personally I recommend using WebSocket when you expect full duplex. That way, all intervening proxy servers either know that you are expecting full duplex (because they recognize WebSocket), or connection setup will fail fast.

I realized I replied to the wrong comment :) but I had some issue with the suggestion regarding server-sent events... anyway, thanks for answering, I really appreciate your work and it's nice to hear your input on this.

Reply on news.ycombinator.com