Making Tailscale Faster
tailscale.com
tailscale.com
For really high bandwidth cases, things like DPDK are the long term best choice and are primarily userspace, for good reasons. Kernel mode is not the pure benefit it once was (if it ever was).
Separately, wireguard itself has a problem that the crypto suite it uses is not supported by hardware accelerators. So if we want to get into the hundreds of gigabits range, we will possibly need to switch packet formats entirely. (But, wireguard also needs to update to support post-quantum so maybe they’ll fix both problems at the same time and we can join in.)
Might as well switch to IPsec. Everything is already in place, including hardware acceleration. But most people won't, and we'll live in a world with duck-tape hacks built around a compromised Wireguard-ish layer.
Things like Google’s PSP already achieve that while still being hardware acceleratable: https://github.com/google/psp
Post-quantum negotiation will kinda suck but you don’t have to sacrifice everything.
Storytime: 15 years ago I co-founded a startup to build easy-to-use infrastructure management for AWS. At that time, it did not have cross-region VPC peering or routing, so you couldn't easily and safely have apps that communicate between regions.
So I started working on creating an overlay network. My idea was to use IPsec, it even has an RFC that documents its kernel interface. So that when a host wants to send a packet to the secure network, the kernel goes to my userspace daemon, that in turn goes to the central server that provides it the key for the given host pair.
And it turned out that the interface lacked a crucial part - on-demand key negotiation for incoming packets. It had this for _outgoing_ packets, but not incoming. The only sane way to make it work was to create a proactively updated database of all the hosts.
Well, I did that. It also did not work (tm). I found so many issues with broken MTU handling, broken NAT traversal, etc.
I eventually gave up on that idea and started working on a simple TUN/TAP-based overlay. I almost made everything work, but our startup got acquired by AWS, and this line of work was abandoned.
My implementation relied on the extremely finicky Linux kernel IPsec and would never have worked in userspace macOS or windows. But yes, a Tailscale control plane with a per-node switchable data plane, one of which is IPsec with PQC, is a real option that would work.
By locking down the control plane it becomes incompatible with classic IPsec, but also avoids most of the security gotchas that plagued IPsec.
It could have worked in macOS! It implements RFC-2367 (PF_KEY socket type) and I even made it work on my laptop.
I traced the SA database operations on Windows, and it could have been done. Probably. But I abandoned that to preserve my sanity.
I don't think there's anything necessarily complicated about MLKEM that would make this too difficult.
Switching to IPSEC loses you other WireGuard benefits; the wins don't end at "just one carefully curated set of cryptography primitives", but extend into things like DoS prevention and a design that admits to processing incoming frames without dynamic allocation.
Didn't know that WG can't use hardware crypto accelerators. I hope it will someday
It's quantum-resistant if you define a PSK. You still have to distribute the key out of band, but you already have to do that anyways for the public keys of the peers.
In practice, Alice and Bob will often be two machines that are under control of the same entity, and that entity will transfer the key material from a third machine to the Alice and Bob machines over SSH or HTTPS. In those cases, the out of band mechanism is "SSH/HTTPS via trusted relay machine".
I don't think this is quite equivalent: with public keys you 'just' have to establish authenticity, but with PSK you also have to worry about secrecy.
In someone ways it's similar to the PGP/GPG situation: initial secrecy is not the issue, trust is. If you trust someone's Twitter/Bluesky/Mastodon account, they could just the pubkey there; if you think e-mail is 'secure enough' to not be tampered with, they could e-mail you their pubkey. The threshold for 'PSK safety' is probably much higher.
SSH - the client pubkey has to somehow reach the server's authorized_keys file, and the server fingerprint has to somehow reach the client's known_hosts file.
HTTPS - The root certs have to be present in the client cert store in order for the client to authenticate the servers they connect to.
PGP - People exchange their public keys in key signing parties after verifying each other's identities in person.
This statement is mostly true for computers, but a human can look at a (e.g.) Twitter post and know that the account belongs to someone and copy-paste the key in the post, or via an e-mail that has crossed the Internet in a matter that they're confident has not been fiddled with. A computer (process) just has a string of bits that have come in via a socket: it has no other context and so a bunch of infrastructure has to be tapped into (as you listed).
* You trust the browser/OS
* Browser trusts the root cert store (either embedded in the browser installation or managed by the OS)
* Root cert authenticates the twitter.com connection
* Twitter validates the legitimacy of the account (anti-spam/anti-impersonation/verified user etc.)
* You trust that the person who made the post on the Twitter account is the person you want to communicate with
2) Tailscale's control plane model for peer distribution avoids every concern affecting IPsec in regard to protocol negotiation security and compatibility.
TL;DR, diverging from WireGuard & supporting AES (via protocol negotiation with hardware acceleration), would be relatively painless.
You lose the formally-verified cryptography guarantees underpinning WireGuard & its implementation, though. That's a bigger tradeoff: https://www.wireguard.com/protocol/
You used AI to write that, didn't you? Please don't do that, it's disrespectful.
Anywho, given your strong conviction about this, consider suggesting to Jason on the WireGuard mailing list to swap ChaPoly for AES-GCM, or perhaps consider a fork yourself. Good luck.
So yes, tailscale is compatible with wireguard, it's just not exposed by any of the major coordination servers other than mullvad integration in tailscale.com, but the clients will happily consume information about wireguard peers that do not run tailscale at all.
Now screaming internally because every router I've had that's supported Wireguard I wish I could have just joined into a tailnet directly for subnet routing... and I can't find anything else that just plays nice either.
To get other tsnet clients to connect to non-tsnet client you need to provide the connection configuration through netmap from coordinator server
Has hardware-offload (for AES et al) got faster still, or that keeping CPU busy in the data path for 100gbps workloads is not ideal, or something else? The WireGuard website claims ChaPoly is at least as fast as hardware-accelerated AES. And that it can be further sped up with SIMD.
https://www.wireguard.com/known-limitations
> now we’re on to the next order of magnitude together
Curious: Is this work currently in progress? If so, what's more that's still lined up? The previous GRO/GSO(/LRO, too?) improvements were incredibly impressive (even to u/majke, https://news.ycombinator.com/item?id=35567268).
Thanks.
I haven’t used WireGuard but I have easily doubled OpenSSH performance by switching back to AES.
What is referred to by the term "hardware-offload"? For NICs:
> ConnectX NICs offload and accelerate encryption/decryption at speeds up to 400Gb/s.
* https://www.nvidia.com/en-us/networking/ethernet-adapters/
There have been MACsec implementations at 800Gb/s for several years:
* https://www.rambus.com/blogs/rambus-launches-800g-macsec-mul...
* https://semiengineering.com/the-evolution-of-ethernet-to-800...
And 1.6T/3.2T as well:
* https://www.rambus.com/security/protocol-engines/macsec-ip-3...
Layer 2 VPN is where it’s at anyway. I want to be on my LAN not managing one device or app at a time, I never got the wireguard hype.
I was with you in principle until this part. You gave them ~30 minutes before claiming "no answer".
You can do that though? Tailscale can as well. A device can advertise subnets, and can route them through tailscale, so you just need a single node in a LAN.
This is still L3 layer though. One the main use case of L2 is proper DHCP propagation and avoid subnet collisions. I do not think that this matters in practices though. Only a limited amount of user facing service require proper L2 emulation (apple TVs ?)
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
And if any tailscale employees are reading this - https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
So it's not that simple: it's impossible for Tailscale to use any existing kernel or accelerated WireGuard implementation. They could derive inspiration, but a kernel module for Linux won't fix Windows & Mac. With that said, I feel they have enough funding to maintain a few platforms (:
That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)
1) the WireGuard kernel implementation, despite not even being zero-copy, exceeds the performance of the userspace implementation
2) implementations utilizing the userspace network stack have a maximum potential performance (context switch + memcpy is very slow, and that affects UDP disproportionately). It's the wrong approach for meaningful improvement.
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.
If they would have taken that advice, tailscale instances would have been pwned by copy.fail
>IPsec
Remember that at least one LPE CVE associated to kernel IPSec implementation has been discovered (copy.fail), which means that whatever gains you get from this vpn tunneling, is lost by breaking the basic user security system guarantee.
You are better off not using a VPN at all rather than using kernel crypto
A port identifies a process, an IP address identifies a machine. The machine receives the packet, and then passes it to the process, since it's the machine that needs to receive the packet before routing it to the process, it's the kernel with ring 0 privileges that needs to process the packet and map the port to the process.
I don't think it's a good idea to put more of the stack than necessary into each individual program though, because then you're reliant on each program implementing everything correctly and you have to worry about bugs and security issues in every single program you're running rather than just in the kernel.
Considering that even something as simple as opening a socket (which involves calling one function to give you three numbers which you pass unchanged to a second function) is largely a clusterfuck in existing programs, I don't trust them to implement the whole TCP stack.
I was thinking more along the lines of network namespaces -- although actually, I should have been thinking of TUN interfaces. Any packets sent into one of those are delivered to the attached program, which can do what it likes with them. Those are usually used by e.g. OpenVPN for L3 things, but nothing stops a program from handling the L4 headers.
I've even got a backup wireguard server running on a Pi 1b. Works fine. We currently run wireguard on our router (and it seems more and more routers are supporting it).
There are keys to configure for each client, but once you have the configuration for one client, the rest come very quickly and easily.
I should add that I don't have any experience with Tailscale, but compared to OpenVPN and other VPN solutions, Wireguard is lightweight, simple, and easy to setup/configure.
We use it on all our mobile devices (phones, laptops, tablets) to tunnel our traffic through our home network with all the filtering it offers (along side access to private services we host).
I find tailscale to be simpler, more reliable and cover my needs better. I am pretty sure that have i known about tailscale before, i wouldn't have got the unify gateway.
I still love my unify though, dashboard galore
That being said, I do understand the appeal of Tailscale and have used it.
Once you install it you can then do everything from a UI to onboard new users.
It supports OIDC so you can avoid mangling with password and delegate that to an iDP such as google, github etc.
WireGuard takes 30 mins to an hour to set up, you'll need to configure port forwarding, DDNS, create keys for each device, and add them to each device manually. But you have 100% control.
Performance-wise, I haven't noticed a difference. My internet connection maxes out way before Tailscale hits any performance limits.
Tailscale wins for convenience.
Tailscale allowed me to setup everything at home and just plug them to their network in 5 mins.
- I was able to get my partner onto the tailnet by telling her to install an app and login. She doesn't know or care what wireguard is, but she can now access some of my self hosted services on her phone.
- I'm able to easily dynamically register machines to the tailnet, such as CI jobs
- I'm able to self host a DNS resolver and have it just work for devices connected to the tailnet
I'm sure I could achieve these goals with plain wireguard, but I feel like I was able to outsource significant complexity to tailscale instead.
Just the other day I was able to set my sister up with access to my Plex server and the ability to piggyback on my UK internet connection to stream BBC/Channel 4 content from Australia. It took all of 5 minutes to get it working.
I have a few services I've configured static routes for on my router (so my wife can access them without needing to have her phone connected to Tailscale all the time), but other than that it's never been much of a pain point.
Assume we have devices a, b and c, which are basically in different segments of the same network and have nice pings to each other. We're trying to ssh from a to c, but NAT traversal isn't possible. Both a and c can do NAT traversal to b.
Instead of a going all the way over to the DERP in WAW and then back to c again, it could go a->b->c instead.
In my experience, DERPs are pretty slow and have high latency (compared to not going off-network at all), but there's no real way to avoid them if everything you have is behind some sort of NAT, even if some of them are trivially traversable (think "devices can do UPNP").
Note: I'm a tailscale employee
What I'm thinking of is something far more opportunistic; the way to piggyback on an existing peer (or a set of such) as a pseudo-relay, if and only if network conditions allow, and this is actually something that is worth doing in the given situation.
I'm not about to switch back to the configuration management hell that was pure wireguard but it would be really lovely to not have to connect/disconnect when I want to use tailscale
* https://github.com/luigirizzo/netmap
* https://www.usenix.org/conference/atc12/technical-sessions/p...
* https://man.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
Then again, that begs the question of what you’re heavily using exit-node-routed internet links for that Tailscale’s battery draw is noticeable. Most network-intensive internet tasks draw way more power to drive whatever the application is, such that VPN overhead is a rounding error.
That said, the architectural multiqueue improvements and memory optimizations are useful starting points to build on for every platform.
"lines"? "depth"? I feel like there is some context missing here. What are these terms they refer to?
It would be nice to have an easy way to define a multi node proxy. Right now i have several devices in my network which i can't install tailnet directly on. For each, i have to create a seperate container for each (since i tailscale node can't share network namespaces).