Topo – A library to create in-process topologies of goroutines

I think you misunderstood the blog post and go channels. Shuffle is absolutely not want you want to use (in the README example). Sending to a channel blocks until the message is accepted by the receiver (unless you're using buffered channel).

What you want to do instead, create 1 channel, and make multiple goroutines (Sinks) read from it, then whenever goroutine is finished with the task it would take a new message from that channel.

By picking a channel to dispatch yourself you don't take into account busy-ness of it, so you might wait on a channel even though there are others that are idling waiting for messages to be accepted.

The library handles a lot of what is covered on this golang blog post: http://blog.golang.org/pipelines.

I've been using this library which helps with composing channels, and also has some nice implementations of useful patterns:

https://github.com/eapache/channels

Unfortunately Go's lack of generics hurts in letting me use these helpers in a way that guarantees the data types going in/out of the generic channels.

Hi @epimenov thank you for the feedback. The shuffle method starts a goroutine for each inbound channel, so one busy outbound channel will not block the whole system. That being said I probably should add two more features, one of which is to add a channel size rather than always using a 0 size channel, and the other is to change it to select the first non-blocking outbound channel. I don't consider the blocking part always negative, because handling back-pressure is need at one point or another, but you are right, the method should not introduce starvation.

You probably know this already, but "topo" means mouse in Italian.

Looks like an interesting project. It'd be interesting to see someone who knows both Erlang and Go well explain this in comparative terms. I know Erlang reasonably well, but not Go.