Benchmark results of Kubernetes network plugins over 10Gb/s network
itnext.io
itnext.io
We have been trying to reproduce the performance results ever since the article was published as this is not in line at all with what we measure daily in our CI. We can easily do a multiple of these numbers.
There are some obvious flaws in the benchmarking scripts [0] such as using the "used" column of `free` without taking into account cached file buffers.
However, it does not explain why HTTP and FTP are worse compared to the TCP benchmark which is doing wire speed at ease. None of the Cilium datapath is HTTP or FTP specific unless HTTP specific security policies are in place in which case HTTP traffic is actually parsed.
We have requested more information on the scripts used by the author and continue to investigate. We will publish results as soon as we can reproduce this.
As stated by other commenters as well, most of these benchmarks are measuring the same Linux kernel code except for Weave (OVS) and Cilium (BPF). However, at the specified MTU of 9000, the bottleneck for all plugins will not be the forwarding datapath but the actual client and server code copying the data in and out of the kernel as there are very few packets actually being created and forwarded.
[0] https://gist.github.com/AlexisDucastel/ebb884831aeec5827e4df...
Do you have some more detailed info on the configuration and commands you used? Nginx, for example, doesn't have sendfile() turned on by default (just one example of a configuration that might change benchmarks).
This, 1000x this. I'm afraid too many people treat their CNI plugin as 'magic' whilst many of them really aren't. 'Host' versus Calico is basically benchmarking the impact of a Linux bridge device, and maybe some more iptables rules than the host has (depending on whether the host benchmark has iptables enabled at all, whether there are K8s network security policies in place and enforced by Calico,...).
Also, configuration details are lacking. E.g. in the Calico benchmarks, was ipip enabled or not?
But you setup your CI. This guy’s numbers are a lot closer to what I or another CNI n00b would get trying to set something up.
OTOH if you’re already a CNI expert, you wouldn’t be reading this article.
As someone wondering what CNI to choose I found this article helpful
Every time I've done a performance comparison an expert pops up and says the result is invalid because of X. It takes 10 seconds to write the comment but perhaps a few hours to redo the tests and update the blog contents.
The blogger doesn't want an inaccurate blog and the software authors don't want bad benchmarks left up which constantly crop up in search results. As a blogger you feel a little duty bound to work on updating a blog you know probably won't be re-read by the majority of people who have already opened it anyway.
My conclusion is that fault should fall on the side of the software developer in most cases. Having created a startup I understand the time pressures and motivations driving the roadmap. There is a natural tendency to work on the differentiators and high value complex features. Blogs like this should act as a reminder that there is massive value in prioritising sane defaults, tests, documentation and building logic into the application that makes incorrect settings that effect performance unlikely.
From reading this blog I get the sense the author is quite technical. A positive public relations move would be to spend your time replicating the results and then when the problem is found make it difficult for the next person to have the same issue. Preferably with logic in the software, but worst case scenario with some bold text towards the top of the readme so it's not buried somewhere obscure.
Real world traffic has lots of nasty small packets.
"With SCP, we transfer the 10GB random file over scp using OpenSSH server and client. We can clearly see that even bare-metal performance are much lower than previously."
SCP is just notoriously slow, and it should be expected to not be able to max out the connection with it.
Encrypted WeaveNet handled plain TCP traffic at 1,330 Mbit/s. Without the SCP comparison there's no way to tell whether that's really really good or really really bad. Seeing that the unencrypted version of WeaveNet handled SCP at 1,594 Mbit/s shows that it's not horrible, but could probably be better. It's good for WeaveNet folks to know they have some room for improvement and it's good for you because you might want to consider alternatives if you require encrypted traffic and are very bandwidth sensitive.
When we ran a similar test we got 3.75Gbit/sec; the figure in this article is closer to what we got in our "slow mode" which is used as a fall-back when it can't get the in-kernel path to work.
https://www.weave.works/blog/weave-net-performance-fast-data...
https://docs.google.com/spreadsheets/d/1qCOlor16Wp5mHd6MQxB5...
In the Calico-without-ipip case, you're basically comparing host networking (let's assume with some iptables enabled) with host+iptables+bridge networking (yes, there may be more iptables rules involved). Let's assume the impact of iptables is the same between both, then it'd be interesting to measure the impact of the Linux bridge being used. As a colleague of mine mentioned, this may be barely noticeable on a 10Gb interface, but could be on faster networks. How about running these tests on a 20Gb or 40Gb network? These are quite common in datacenter networks, which is where you'll be using these CNIs (unlike GKE, EKS,... where you can integrate with the 'native' SDN).
Finally, I'd be interested to get some results when using the macvlan CNI plugin (though then one loses network policy support, sadly enough :( )
https://docs.projectcalico.org/v3.2/usage/configuration/ip-i...
This is used when there is source/dest filtering happening on the network (such as in certain cloud providers where you can't send packets from 10.0.0.1/32 to 192.168.1.2/32 because 10.0.0.1/32 is not valid, so you pack it into another IP packet that makes the source seem like 192.168.1.3/32).
https://github.com/coreos/flannel/blob/master/dist/extension...
For instance in this post itself we learn about all these CNI wrappers but learn nothing about Vxlan, BGP and Linux networking tools that these are using underneath. And those are actually easy to use directly. This does a disservice to both developers and users.
For instance true game changers in this space is tech like Wireguard that can deliver encryption at near line speed and would kill these benchmarks. Everyone can benefit from Wireguard today. Initiatives like CNI encourage wrappers and those then wrapping tools like Wireguard end up getting the recognition and success while the underlying tools like Wireguard fade into the background. This does not seem like a healthy or sustainable development model.
If Wireguard doesn't write the wrapper themselves they have no one to blame. People need to understand that the hard/cool part and the valuable part aren't the same and if you are too cool to create value then you can't complain about other people taking it.
All these wrappers still require time investment in knowledge, setup and configuration. The cost for users knowing the wrappers but not the tech is being locked in a platform, if they know the tech even if Kubernetes crashes and burns their knowledge and time investment remains valuable.
Before Wireguard it was extremely difficult to get near line speed with encryption, now accessible to all. If this is not 'value' what is? Given this post is about networking benchmarks this is a huge deal. If people stop creating these kind of networking tools and technologies then there is going to be nothing to wrap.
But in general most companies can't do automation or self-service in their physical network so overlays are the only way to get any network agility. And BeyondCorp is much harder than that.
None of them actually handle packets, connections, routing etc directly I assume?
Given that, it seems disingenuous to benchmark them... You're simply benchmarking the Linux Kernel vs the Linux Kernel.