Userspace isn't slow, some kernel interfaces are
tailscale.com
tailscale.com
For normal UDP sockets UDP_GRO and UDP_SEGMENT can be even faster than sendmmsg/recvmmsg.
In Gvisor they decided that read/write from tun is slow so they did PACKET_MMAP on raw socket instead. AFAIU they just ignore tap device and run a raw socket on it. Dumping packet from raw socket has faster interface than the device itself.
https://github.com/google/gvisor/blob/master/pkg/tcpip/link/... https://github.com/google/gvisor/issues/210
I did a lot of work on QUIC protocol efficiency improvements in the last 3 years. The use of sendmmsg/recvmmsg yields maybe a 10% efficiency improvement - because it only helps with reducing the system call overhead. Once the data is inside the kernel, these calls just behave like a loop of sendmsg/recvmsg calls.
The syscall overhead however isn't the bottleneck - all the other work in the network stack is. E.g. looking up routes for each packet, applying iptable rules, applying BPF calls, etc.
Using segmentation offloads means the packets will traverse also the remaining path as a single unit. This can allow for efficiency improvements from somewhere between 200% and 500% depending on the overall application. It's vastly useful to look at GSO/GRO if you are doing anything which requires bulk UDP datagram transmission.
With TCP, they are mostly done in the kernel and network hardware. TCP syscalls can transfer many packets worth of data per syscall, and the kernel and hardware have done segmentation offloads for TCP for many years automatically, depending on the hardware and kernel version. Over time these optimisations have been improved in the kernel, and over time newer hardware provided more capabilities for the kernel to use.
There's not much need for complex application or library support in userspace, though there are some syscall patterns that help the TCP stack more than others. The main thing is, send and receive in larger sizes to let the kernel handle buffering as well as reducing syscall count. Don't do silly things like a write() syscall for each HTTP header and then the body, for example. Buffer it up into a single write.
Note that this is not necessarily the right thing to do, for a VPN.
Encryption: Wireguard uses the destination IP of each packet to pick the outgoing tunnel.
Decryption: The destination addresses of the encapsulated packets are arbitrary.
Where the sequence is required to share things like routing lookups, a VPN generally can't just take a sequence of packets from one side and send the same sequence of packets, individually transformed, out the other side.
Batching in general is still a common trick for performance, but the VPN gateway acts as a multiplexer/demultiplexer. It's doing routing after all!
There is a v1 patch posted on LKML, and I think they're hoping to get a v2 patch posted by January. If you are interested in a chat with the team, email me rjones redhat.com.
I'll email you.
Also, the "slow" things about kernel interfaces (if you aren't doing IO which is nearly always the slowest thing) usually isn't a given syscall, it's the transition from user to kernel space and back. Lots of stuff going on such as flushing cache and buffers due to security concerns these days.
We certainly didn't try to make it click-baity. The point of the title is that people assume that Tailscale was slower than kernel wireguard because the kernel must be intrinsically faster somehow. The point of the blog post is to say, "no, code can run fast on either side... you just have to cross the boundary less." The blog post is all about how we then cross that boundary less, using a less obvious kernel interface.
What's really crazy is that we're talking about userspace and 10Gbit, which shows that CPUs, busses, and the interface protocols have all been scaling well with interface speeds!
Personally I don't ever increase MTU, even if there's a significant performance win, since I prefer to not place our oncall in a situation where they have to debug an outage due to MTU incompability.
The title gave me a reasonably decent idea of what to expect, and the article delivered.
When Josh et al tried it, they hit some fun kernel bugs on certain kernel versions and that soured them on it for a bit, knowing it wouldn't be as widely usable as we'd hoped based on what kernels were in common use at the time. It's almost certainly better nowadays.
Hopefully the Go runtime starts doing it instead: https://github.com/golang/go/issues/31908
Besides kernel supplied offloads, the thing which helps further is actually bypassing the kernel with AF_XDP or DPDK. But those techniques will have other challenges and limitations.
Most everyone nowadays seems to try to avoid kernel as much as possible, since it's vastly slower than a userspace forwarding plane (although eBPF/XDP will likely change this).
Also I don't think Tailscale could use DPDK in 99% of cases.
Just thought it'd be interesting to see the actual throughput along with the rest for the benchmarked environment.
What's interesting is that the cost for x/crypto on these system classes is prohibitive for serial decoding at 10gbps. I was ballparking with 1280 MTU, you have about 1000ns to process a packet, it takes about 2000ns to encrypt. The fan-out is critical at these levels, and will always introduce it's own additional costs, with synchronization, memory management and so on.
If yes, then you should be able to get the kernel to spread your incoming traffic across cores with minimal contention and coordination, with multiqueue tuntap:
https://www.kernel.org/doc/html/latest/networking/tuntap.htm...
I wanted to give a concrete example of this. Crypto or such code using SIMD can drop some Intel CPUs into lower frequencies, and this effect can also apply to all cores. It takes a short moment for the CPU to "speed up" again. Then there's icache pressure etc.
The usual rule is, microbenchmarks are microuseful. Systems mixing micro-workloads just can't reach the same throughput, and trying to guess the final performance is very tricky.
https://stackoverflow.com/questions/19722950/do-sse-instruct...
Note that the numbers include networking overhead, if you would just measure crypto performance the numbers would obviously be higher - as would the difference between the algorithms be.
We have a concept of userspace for safety. Systems without protected memory are very unstable. The tradeoff is speed.
Trading speed for safety is extremely commonplace.
* Every assert or runtime validation
* Every time we guard against speculative execution attacks (enormous perf hits)
* Every time we implement a "safe" runtime like java
* process safety models using protected memory
Efficiency is one of many competing concerns in a complex system.