Vice: Go channels across many machines

As jerf points out in the /r/golang thread [1] and this [2] older thread, Go channel semantics don't match network semantics. In particular, Go channels have exactly-once delivery, which is impossible to do over the network.

Go channels are designed for communicating between goroutines; they work more like a signaling mechanism than a data pipe mechanism. They have too many design trade-offs to be practical for anything else. They're terrible as general-purpose queues and pub/sub-style topologies, for example. It's tempting to use channels are "iterators" because of range{}, but that's also a terrible idea. Buffered channels should be avoided unless you know exactly what you're doing. And since unbuffered channels block on send, you have to be super careful about goroutine interdependencies; the only way to avoid causing receivers to block senders is to wrap the send call itself in a goroutine, but if you do have slowness, you can accidentally fork millions of goroutines this way. Channel ownership is tricky to get right. And so on. Channels seem trivial on the surface, but they aren't. They look like Unix pipes, but they aren't. (That only one party to the communication can close a channelI consider to be a wart. Trying to close a closed channel panics.)

The reason we see people abusing Go channels like this is simple -- channels are generic and support type-safe type matching (with select{}). They sound like a perfect match for certain things. But they really aren't. Channels are really a great argument for why Go could benefit from generics.

[1] https://www.reddit.com/r/golang/comments/6q4p1j/comment/dkv6...

[2] https://www.reddit.com/r/golang/comments/2gecvq/netchan_go_c...

We have libraries in Ruby, Go and Java, and the concept is that developers should not have to worry about transports or how they work: you make it as simple as possible for most developers in the org to work with distributed systems, and they'll start working with distributed systems, and the arcane understanding of exactly what the trade-offs are between different vendors and messaging technologies can be centralised to a few devs with ops support instead of it being bikeshedded to death.

This is nice in that it becomes scalable and transparent with little refactoring in Go, but having multi-language support is useful, otherwise we'd probably drop drumbeat and use this instead.

It does suggest we should think about how best to do this in Go again though - I like the pattern a lot.

An alternative solution, which requires a modified compiler but avoids the need for a context and errors channel, are the channels used by Clive:

https://lsub.org/export/golsub.html

I'd really like these channels (or some modified version of them) made it into Go2.

[1] http://github.com/rapidloop/nrpc

also see: heka, deprecation of