LXC container networking deep dive
flockport.com
flockport.com
That said, note that this is a very bad idea in most cases because you are increase network complexity, expose additional code (both points bad for security) and create startup latency. The only excuse for this normally would be an application clearly requiring layer two connectivity that you are prepared to test, maintain and debug across flaky somehow-encapsulated WAN infrastructure.
As the maintainer of lxc-gentoo and an early LXC adopter (5 years), I applaud the author for bothering to look at things instead of jumping on one of the PaaS bandwagons. However, one should be careful of designing spherical cows.
Personally I recommend the following for container networking:
(1) Use veth device pairs. These are host-to-container and provide you with per-container isolated networking and accounting very cheaply.
(2) Preconfigure your interfaces with static IPs from the host side by generating an appropriate lxc.container.conf and network up script to configure the host-side interface. This removes any latency in interface configuration due to layer two issues such as spanning tree protocols, and layer three issues such as DHCP. Note that you may have to set resolv.conf using a similar automated process.
(3) Provide outbound (internet) or inter-container connectivity to the various container veth interfaces on the host via explicit rules on a per-guest iptables chain that can be conveniently and precisely destroyed with the termination of each guest system. Treat in-kernel bridges as a hack with STP hassles, like they are.
- Sometimes people dedicate a whole physical interface for the sole use of a container, by moving it into the namespace. The downside is that the interface is completely unavailable to other namespaces, including the host.
- macvlan is pretty popular, it allows multiple containers to share access of the same physical interface, without conflicts and without the performance overhead of a veth+bridge combination.
- you can also connect your container to an overlay network using a variety of tunneling systems. vxlan is the flavor du jour but there is a flurry of competing standards and implementations in that area.
By far the coolest part about container networking on Linux is that it's just regular linux networking. The entire stack, and all the tooling that comes with it, is available to containers. You just need to glue it together. And of course these options are not mutually exclusive: a namespace can contain as many network interfaces as you want.
Disclaimer: I am not a networking expert, but a lot of networking experts want their favorite toys natively integrated into Docker, so they take the time to explain to me how they work.
A general-purpose product is harder to design well than a special-purpose one. - Frederick P. Brooks, 'The Design of Design: Essays from a Computer Scientist' (2010)
10th Fundamental Truth of Networking: One size never fits all. - RFC1925 (1996)
The comments are spot on, most networking gurus appear to be strictly against extending layer 2 as we note in the post but we wanted to put the options out there. You definitely need to address latency and mtu issues across remote hosts that will crop up, but with standards like VXLAN based on extending layer 2 perhaps we will soon start seeing interesting workarounds.