Hacker News Reader

Comment by justinsb | Hacker News Reader

User-space overlays are slow, Weave more so than flannel. The really interesting data-point for me is that flannel with VXLAN has negligible overhead.

View on HN

Well, the latency is far from negligible. However, the bandwidth really is impressive. Having userspace as the control plane while keeping the data plane firmly in the kernel has long been the way of things, and this blog post's not-so-subtle message is "your problem is not novel enough to justify the cost of your solution".

To be clear, there is nothing fundamental (to the best of my knowledge) that would prevent Weave implementing a VXLAN backend. It is simply a poor implementation at the moment, and even then, only in this regard. Who knows, maybe Weave's use of a gossip protocol vs flannel's CP etcd could make it more suited to some deployments. Though, that does need to be tested.

Sorry, yes, you're right - the latency overhead is not negligible (as a percentage, though in absolute terms it seems pretty tolerable).

Is this added latency just VXLAN overhead, or is flannel userspace still involved in per-packet-processing? I'd love to see a test comparing flannel/VXLAN to a manually configured VXLAN tunnel.

(I'm off to read the flannel code...)

Actually, userspace packet switching can be fast (10GbE linespeed-fast) thanks to approaches like DPDK [1], where a userspace process has zero-copy, direct access to NIC ring buffers.

1 - http://dpdk.org/

The problem with DPDK is (as I understand it) that it doesn't handle multiplexing the connection between multiple cores/processes/containers (i.e. lack of polling makes scheduling hard/impossible). The nice thing about using the Linux kernel as your data plane is that you can still have all your bog standard routing, in addition to this fun VXLAN/GRE/etc stuff. That said, I haven't ever implemented anything using DPDK, so I may be talking out my arse.

You are correct however that userspace networking is awesome, and I look forward to it becoming more and more prevalent in applications that can benefit from it.

It depends on how you use DPDK. If I use it from the container directly to the NIC, you certainly do loose all of the kernel capabilities. However, we believe (but have not tested) that you can use a DPDK virtual interface in the container/vm (memnic or virtio) that connects to the DPDK driver in the kernel, so the path from the container/VM is 0 copy. The kernel then does it's processing, and then, another DPDK path could (potentially) be used to 0 copy the traffic to the NIC (really uncertain about that last stage). Basically, you are just using DPDK to save on the copy cost.

This is all academic until tested, btw. As of yet, we (on Calico) haven't had anyone stand up and say that they need more performance than what the native data-path we use today is capable of delivering.

Now that sounds interesting. I'd love to read about that if you ever do move the idea from paper to production :)

We'll let you know.

The problem is Docker containers expect a system network endpoint, while dpdk gives you a userspace network endpoint.

Yep, DPDK is great for NIC-to-NIC traffic and maybe for VM-to-NIC now that vhost-user exists. But that's leading people to suggest DPDK as the answer for everything, which it isn't.

Is there any solution based on openvswitch ? If using vxlan to encapsulate containers L2 frames, the openvswitch kernel module should be able to route packets very well with much lower overhead than any userspace solution that do not access the nic memory directly.

SocketPlane or OVN.

I haven't seen those used in the wild yet, but fwiw, weave is working on a 'fast data path' that uses the same kernel path as ovs

OVN is 0.1 fairy dust and unicorn horns at the moment, looks very promising but you're most likely going to be waiting another 12 months before its solving your problems.

SocketPlane was bought by Docker and is now DEAD, to become whatever Docker decides is good enough to silence their critics. "See we bought these network guys, we swear we're taking networking and plugins seriously!"

Using a VLAN (VXLAN), as the post shows?

Reply on news.ycombinator.com