But one thing that Tailscale didn't do well (at least early on) is performance. It's user space Go, which seemed to cap the data transfers when I tested it out. I would prefer a really fast data transfer P2P so I could use Tailscale in between my web server and DB.
Compared to another VPN? I’d be curious to know whether the kernel mode byte shuffling solves that problem. But even so, a kernel module is a pretty big ask only for connectivity.
In my experience, UDP in general isn’t as performant in practice as one would think. Not saying you can’t push the limits, and even outperform TCP, but to do so with a reliable cross platform way isn’t exactly trivial today.
All my benchmarks (albeit user space) have shown that pushing bytes over udp has a higher CPU overhead, and that’s even if you omit retransmission, congestion control, etc etc (ie just push garbage bytes). And even if your cpu can handle the throughput, the congestion control can still bite you for god knows what reason. When I ran quic benchmarks they got deprioritized in the presence of tcp traffic. Don’t know all the reasons why (sorry, just didn’t have the time) but at least to me TCP wins the bang-for-the-buck-throughput-on-commodity-hardware category, hands down. Maybe this changes with platform-specific optimized vectored IO, but that alone would be a huge effort. No, the more time I spent on it, the more I appreciated all the things TCP gives me for free, and it’s remarkable resilience in complex conditions. I am also happy I don’t have to worry about bulky 3p libraries. This is what the OS is supposed to do, imo. So this UDP-hype-renaissance we’ve seen over the last years is a bit premature or at least not as obvious as people hoped (including myself).
Another fun fact (for anyone who read this far): contrary to popular belief p2p TCP isn’t harder to do than UDP, not really.
Another thing I wondered about is how much CPU overhead VPNs add, and how it performs when maximizing throughput. (Not Tailscale but a “regular” one with kernel packet switching). Do you have any experience with that?
> VPNs that use TCP as the transport layer could try and special case TCP handling and treat them as a flow and just transport the TCP data streams instead
Yes! An interesting observation is that TCP composes really well, ie relaying works excellent. However, for VPNs it’d be nesting TCP which melts down quickly.