Is the Network the Limit? Dealing with Weave, CoreOS and Azure | Hacker News Reader

Is the Network the Limit? Dealing with Weave, CoreOS and Azure

arjanschaaf.github.io

23 pointsSanderMak13 comments

I suggest you also look at GCE for network performance. It is something that the Google networking team has been working hard on and, because of the unique network fabric at Google, should scale to lots of VMs.

Also, you can get equivalent "IP per container" using advanced routing with GCE. Flannel has a GCE specific back end where it inserts routes for each subnet assigned to each range. That will eliminate the extra level of encap.

(Disclaimer: I started GCE but am no longer at Google.)

Hi Joe :)

Conveniently, Amin Vahdat recently shared some content publicly about why it should scale to lots of VMs: https://www.youtube.com/watch?v=FaAZAII2x0w

(And, to others, disclaimer: I work on some bits of GCE networking at Google)

+1 to @jbeda, but let me offer some comments for the avoidance of possible confusion among potential Weave users.

Weave aims to deliver a completely portable network. In other words if you create an application using Docker container and a Weave network, that should be able to run anywhere. And all this should be 'magically simple'. Once you have decided where to run your app, you may wish to trade portability for performance gains. For example by using GCE networking (or Azure, or ...).

To date Weave achieved portability by sending some packets (inter host) through user space. This has big benefits in terms of ease of use eg. dealing with wide area networks, multicast and firewalls. But under load it performs worse than kernel-only models.

We now have Weave "fast data path" - http://blog.weave.works/2015/06/12/weave-fast-datapath/ ... This aims to deliver close to line performance with portability and extreme ease of use.

There may of course be yet more optimisations that end users wish to investigate. If you are willing to sacrifice some portability you could certainly make use of fast networks provided by a specific public cloud. We haven't yet seen a strong need to support this, but it is certainly a reasonable thing.

(Disclaimer: I work at Weaveworks)

Is the poor performance of Weave really surprising? Userland networking will always be lesser performing than that of native kernel networking (the show is somewhat better with technologies such netmap/PF_RING/DPDK). Weave itself would probably improve in performance if it utilised a TUN/TAP device + static routes as opposed to its current mangling of packets with pcap and BPF.

We did try TUN/TAP. It seemed to be slower than pcap.

Our answer to weave performance concerns is on its way: http://blog.weave.works/2015/06/12/weave-fast-datapath/

We're really interested to hear how weave fast datapath works out in all kinds of environments, which is why we put the preview out. It's a shame that Arjan's post did not include numbers for FDP, but maybe he'll be able to include them in an update.

Indeed tun/tap performance is terrible, it seems to take about a whole CPU at gigabit.

That, in general, is not the behavior I've observed on native hardware. Without getting into hard numbers, I routinely see multiple gigabits/second/core.

With which options? Standard tap (or tun?), or macvtap? The last test I did (macvtap to an external interface but none of the offload options), a gigabit used a lot of CPU, like around a whole CPU just copying data.

regarding fast data path: Arjan's colleague Paul posted some new numbers: http://paulbakker.io/docker/docker-cloud-network-performance...

User space networking overlays impose a penalty just like and for much the same reason as dynamic language, garbage collection, etc. -- it's a trade-off between versatility and performance.

The best networking performance is achieved by binding directly to the metal using SOCK_RAW, but that also gives you the least flexibility.

oddly, there is a growing trend of going userland and skipping native kernel because, well...the kernel is slow. userland can be made fast.

I think seeing Weave implemented in something like say, Snabb Switch, would at the very least be interesting -- from a performance standpoint.

A major point of weave is that you don't have to change your code, so it has to work with the kernel network APIs.

Indeed. Kernel-bypass networking is great when you are building something that just receives and send packets (and maybe does other things that can be moved to other threads within the same process using lightweight synchronization mechanisms). It's not great when you want to interact with other processes, because then you have to go through the kernel anyway. And if you have to go through the kernel anyway, the fastest approach is to do as much as you reasonably can in the kernel.

Reply on news.ycombinator.com