GotaTun – Mullvad's WireGuard Implementation in Rust
mullvad.net
mullvad.net
That CPU is pretty much a toy compared to (say) a brand-new M5 or EPYC chip, but it similarly eclipses almost any MCU you can buy.
Even with fast AES acceleration on the CPU/MCU — which I think some Cortex MCUs have — you’re really going to struggles to get much over 100Mbits of encrypted traffic handling, and that’s before the I/O handling interrupts take over the whole chip to shuttle packets on and off the wire.
Modern crypto is cheap for what you get, but it’s still a lot of extra math in the mix when you’re trying to pump bytes in and out of a constrained device.
For reference, I have an implementation of ChaCha20 running on the RP2350 at 100MBit/s on a single core at 150Mhz (910/64 = ~14.22 cycles per bytes). That's a lot for a cheap microcontroller costing around 1.5 bucks total. And that's not even taking into account using the other core the RP2350 has, or overclocking (runs fine at 300Mhz also at double the speed).
An embarrassing slip, TBH. I’m gonna blame pre-holiday brain fog.
For testing I recommend starting from 1280 as a "does this even work" baseline and then tweaking from there. I.e. 1280 either as the "outside" MTU if you only care about IPv4 or as the "inside" MTU if you want IPv6 to work through the tunnel. This leverages that IPv6 demands a 1280 byte MTU to work.
The answer is MTU. The MTU on my network devices were all set to 1500, and my Wireguard devices 1420, as is customary. However, I found that 1340 ( - 80) was the maximum I could use safely.
Wait, though... Why in the heck did that only impact Wireguard? My guess is that TCP connections were discovering the correct MSS value automatically. Realistically that does make sense, but something bothers me:
1. How come my Wireguard packets seemed to get lost entirely? Shouldn't they get fragmented on one end and re-assembled on the other? UDP packets are IP packets, surely they should fragment just fine?
2. Even if they don't, if the Linux TCP stack is determining the appropriate MSS for a given connection then why doesn't that seem to work here? Shouldn't the underlying TCP connection be able to discover the safe MSS relatively easily?
I spelunked through Linux code for a while looking for answers but came up empty. Wonder if anyone here knows.
My best guess is that:
1. A stateless firewall/NAT somewhere didn't like the fragmented UDP packets because it couldn't determine the source/dest ports and just dropped them entirely
2. Maybe MSS discovery relies on ICMP packets that were not able to make it through? (edit: Yeah, on second thought, this makes sense: if the Wireguard UDP packets are not making it to their destination, then the underlying encapsulated packets won't make it out either, which means there won't be any ICMP response when the TCP stack sends a packet with Don't Fragment set.)
But I couldn't find anything to strongly support that.
This is why DoH (DNS over HTTPS) is a thing. It obviously makes no actual sense to use the web protocol to move DNS packet, but, this works and most things don't work for everybody so eh, this is what we have. Smashing the Path MTU discovery doesn't break the web.
Breaking literally everything so long as the web pages work even means you can't upgrade parts of the web unless you get creative. TLS 1.3 the modern security protocol that is used for most of your web pages today, would not work for most people if it admitted that it's TLS 1.3, if you send packets with TLS version 1.3 on them people's "intelligent" "best in classs security" protective garbage (in the industry we call these "middle boxes") thinks it is being attacked by some unknown and unimaginable dastardly foe and kills the data. So TLS 1.3 really, I am not making this up, always pretends it is a TLS 1.2 re-connection, and despite the fact that no such connection ever existed these same "best in class security" technologies just have no idea what's happening and wave it through. It's very very stupid that they do that, but it was needed to make the web work, which matters, whereas actual security eh, suckers already bought the device, who cares.
This situation is deeply sad but, one piece of good news is that while "This Iranian woman can't even talk confidentially to her own mother without using code words because the people in charge there intercept her communications" won't attract as much sympathy as you'd like from some bearded white guy who has never left Ohio, the fact that those people broke his network protocol to do that interception infuriates him, and he's well up for ensuring they can't do that to the next version.
However, there’s going to be a large discrepancy for all devices on battery usage based on whether VPN is on wifi or cellular, and additionally when on cellular how close to the tower they are. I live near cell edge and VPN’s roast my batts on cellular no matter the make, in city it’s almost not noticeable to have VPN on. Better to use wifi when far from towers, cellular more efficient if it’s strong signal.
[0] https://android-review.googlesource.com/c/kernel/common/+/14...
This seems like a very cool, useful project though!
From a quick search, https://blog.thea.codes/nordvpn-wireguard-namespaces/ sees to have at least the bones of a decent solution, though I've not had a chance to dig very far. A lot of results use root to set up the namespace, but I was pretty sure that shouldn't be needed with a new kernel and user namespaces enabled
https://docs.kernel.org/admin-guide/syscall-user-dispatch.ht...
https://blog.torproject.org/introducing-oniux-tor-isolation-...
Disclosure: I work at ZeroTier :)
Yes, lots of firmware runs on hardware where a GC doesn't make sense. Because of limited memory and performance constraints. Sometimes having predictable timings (i.e. not a GC with pauses) is nice. I believe compiler and library support is also just better for many embedded platforms in rust.
> networking type software
Rust is a much more aggressively optimizing compiler, and thus will typically be faster, in the places where that matters. GC pauses might also be a point against golang in some places here. Rust's idioms provide slightly less opportunity for bugs in places where reliability matters (e.g. having a type system that requires you check for errors instead of just patterns that encourage it).
So there's a difference, but generally go is a good enough language for networking software and it would be rare that I wouldn't suggest that "use what you know" is more important than the differences between the languages for non-firmware network software.
One thing others haven't mentioned that I like rust for in this space:
The typestate pattern makes it really nice to work with protocols that have state. You encode your state machine logic into types, and your transitions into methods with move semantics, and you have a nice way to make sure your higher level code is using your protocol library correctly.
Another nice thing is that you can keep the number of copies and allocations way down if you're careful about how you use your buffers.
Besides, engineers at Tailscale, I don't think, strike me as startled by any hurdle too tall to debug, improve Go-based libraries. In fact, they pushed wireguard-go past 10gbps on Linux-based platforms back in April 2023! https://tailscale.com/blog/more-throughput
Wait, isn’t UDP L4? Am I missing something?
(Btw, same sort of thing occurs with zfs combining raid and filesystem to close the parity raid write hole. Often strictly layered systems with separation of concerns are less than the sum of their parts.)
Makes it difficult to block by censors. Great video I saw here: https://youtu.be/pZiG8r-diTM?si=wy35elqMt1T6euq0
This also means wg is just doing one thing instead of a dozen it doesn’t “need” to.
WireGuard is a protocol that, like all protocols, makes necessary trade-offs. This page summarizes known limitations due to these trade-offs.
Deep Packet Inspection
WireGuard does not focus on obfuscation. Obfuscation, rather, should happen at a layer above WireGuard, with WireGuard focused on providing solid crypto with a simple implementation. It is quite possible to plug in various forms of obfuscation, however.
tl;dr Read the docs.
For example, multi-hop betrays the actual exit node to your ISP (or MITM) due to the port used.
AFAIK, at the moment your choices are AirVPN and ProtonVPN. AirVPN has static port forwarding and Proton has UPNP port forwarding.
I miss mullvad dearly, and I might try proton after my 3y sub is up.
It makes more sense when you know they're privacy activists first, businessmen second. But Mullvad shows you can be pro privacy and still offer great UX and a sleek site and client.
Btw, if you're managing things in CLI, you could take a look at their Hummingbird Suite. AFAIK it has a killswitch.
What sucks with Proton is that you can't share the VPN account with friends, because it is tied to your Proton account. They should create a vpn.proton.me subdomain that you can create a special managed account on that can only touch the VPN settings.
Hummingbird doesn't support wireguard iirc, which is a deal breaker
With DCO, OpenVPN can perform almost as well as Wireguard, sometimes even better. Although with more performance overhead so not the best choice for laptops and phones.
Tangentially related but I kind of wish Wireguard looked more toward the future and had included AES as alternative to ChaCha20. At the time of development, many ARM devices didn't yet have AES acceleration which is why ChaCha20 was needed for wide hardware support, but they do since ARMv8 which became widespread in 2015. Intel and AMD have had AES acceleration for a long time. And then ChaCha20 would have been the fallback on MIPS and RISC-V.
I do use Mullvad for most web browsing though. But Imgur for example is blocked on it, and it's blocked in the UK, so I need NordVPN if I want to see any images there.
Most people's VPN usage is literally just geolocation restrictions and Nord is really good at that.
System wide proxy configuration doesn’t actually always work system wide.
A VPN tends to have more success in encapsulating all application traffic (or all desired application traffic, if you’re so inclined to configure your system)
As someone who wants to see Wireguard succeed and in even wider use, this move makes sense from that perspective too. The more implementations we have available, the more we can trust that the protocol is secure and stable enough. Personally I also have about 100x more trust in Mullvad than Cloudflare both in terms of security but more importantly privacy, but that's just the cherry on top.
I work at Obscura VPN and faced with boringtun bugs a few years ago we evaluated a few of the forks and switched our client to be based on top of NepTUN (https://github.com/NordSecurity/NepTUN).
I am curious why Mullvad started their own fork rather than building on top of one of the existing ones. It would be nice if there could be reconsolidation somewhere.
Probably naively, I'm thinking:
- diversity: good
- doubling the attack surface: real bad
Issues in the protocol itself would need all implementations to change, but issues in the implementation would obviously be isolated to one implementation. For something like Wireguard, I'd wager a guess that issues in the implementations are more common than issues in the protocol, at least at this stage.
If anything this is a even a good thing, since it means that each individual vulnerability an attacker finds is less valuable to them.
ofc, thats a cynical view.
i personally think its a bad idea to duplicate efforts. better combine them. otherwise u risk making mistakes that were already solved. missing lessons already learnt.
It's unclear where to report problems, suggestions, etc.
if anyone else is more familiar with go (I only really do rust) is there no solution to preventing stack smashing on goroutines? https://github.com/mullvad/mullvadvpn-app/pull/7728 I understand that go routines have a smaller stack size (the whole green thread problem) but there's no way to fix this?
The official wireguard app also mentions wireguard-go
But you need to be rooted to use it: https://lists.zx2c4.com/pipermail/wireguard/2022-September/0...
But my app’s wireguard is natively implemented by fdio vpp plugin, so it’s based on C.
I tried downloading their Android app, but it's not generally usable for people who host our own WireGuard, which is fair enough.