Docker libswarm
github.com
github.com
What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).
TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!
[1] libswarm is built on a small library called libchan: https://github.com/docker/libchan
The idea is to standardize the way that administrators/users/developers "talk" to the various underlying containers. The intent is for it to have plugin support for etcd and fleet and all the other tools in that layer.
libswarm exposes a standard interface to systems like Fleet, Mesos, Consul etc. It also has adapters for popular infrastructure providers, and for the Docker API itself. That way you can compose your architecture from standard building blocks, and change the building blocks over time without being locked in.
There was huge demand for this in the Docker system because there are so many orchestration tools being announced and competing for the attention of developers and sysadmins. Just at Dockercon, New Relic announced "Centurion", Spotify announced "Helios", Google announced "Kubernetes". Having a standard interface allows for these tools to continue competing without the risk of fragmentation and lock-in.
Of course the interface has to be thin enough to not get in the way. That's why we're focusing on keeping the codebase small, and the API as simple as possible.
libswarm has a future as a nice way of doing composition of networked services that are represented as objects in pipelines. The first demo is a `swarmd` daemon that exposes a docker API endpoint on one end and aggregates between a lot of "real" Docker endpoints.
The big open question is how this sort of composition works when the objects have different interfaces. So, how do we expose a "swarm" API that can be shared between different consistent data stores like etcd, zookeeper or consul. Or how do we encapsulate things like a runtime configured load balancer like nginx, apache2 or vulcand using a unified interface.
My hope for the project is to look at the interface model of Go or the DBus protocol and try to get consensus around the interfaces that make sense and hide complexity.
Seems a little counterintuitive
To do this we take advantage of a low-level communication library called libchan: https://github.com/docker/libchan
What if most of my services (postgres, redis, etc.) are not even written by myself?
2: The idea is to use socket passing, so you can use libswarm for service discovery. Place your postgres service behind a libswarm proxy, which implements the "connect" verb: that allows a client to use libswarm for discovery.
Is the idea for the project to be a rallying point to normalize protocols for different services that already talk over unix sockets?