WireGuard: next generation in-kernel modern VPN
wireguard.io
wireguard.io
I'm the author of this and would be happy to answer any questions you have.
I'm curious what will be your strategy with regard to working or not with upstream Linux ?
I wrote an email on LKML and netdev today to David Miller, the network subsystem maintainer. It's not ready for a [PATCH] set now, but it is ready to get initial feedback from them, so that I can start to get things ready for upstreaming. So: that's on the roadmap and a primary objective!
If your work is in the public domain, then at least the corporations would have much harder time hijacking and wall-garden it, and the chances of them simply giving up are higher. Or so I hope.
I've read a good chunk of your website. GREAT WORK! Do not give up, you're doing next-level work. =)
I do hope I am very naively wrong. But history hasn't given us much comfort in this area so far.
If I learned anything from those last 15 years is that they'll sue you if you don't agree the Moon is made of cheese if it suits their agenda.
So I remain cynical towards them. If they can hijack a good technology for their own profit, they will do it. I think we all knew that much at this point?
Blake2s is essentially their reference code.
Curve25519 is based on Adam Langley's donna.
Does the server get a separate wg interface for every client that connects? Or does it do something like multipoint GRE interfaces with next hop addresses in the arp table?
Does this do a DH key agreement to guarantee forward secrecy or does it literally encrypt the packet with the peers public key?
Also is it in the scope of this project for peers to discover each other for direct communication, like a Phase3 DMVPN?
No. Each wg interface has a public/private key. It then has a list of peers' public keys and associated allowed IPs. By tying each public key to the allowed IPs of that public key, you can be sure that if a packet comes from wg0 and comes from a particular src IP, it comes from who you want. So no need for a new wg0 for each peer.
> Does this do a DH key agreement to guarantee forward secrecy or does it literally encrypt the packet with the peers public key?
It uses a 1-RTT ECDH key agreement based on the Noise Protocol Framework, with identity hiding, perfect forward secrecy, and so forth. So, it does what you hoped it does. :)
> Phase3 DMVPN
I'm not familiar with this, but I'll read up on it. Just found a large cisco document describing it. I suspect that "automatic discovery" wouldn't be very much to my taste, but I'll read up on it before dismissing it wholesale.
If I understand wireguard correctly it seems NHRP could be run using wiregard for transport, but it would cool if a separate NHRP daemon wasn't required, especially since wiregard is already tracking the mapping of internal to public IPs.
> No. Each wg interface has a public/private key. It then has a list of peers' public keys and associated allowed IPs. By tying each public key to the allowed IPs of that public key, you can be sure that if a packet comes from wg0 and comes from a particular src IP, it comes from who you want. So no need for a new wg0 for each peer.
Does this mean that it is impossible for two clients to connect to the same VPN server from behind router doing NAT? Or does each client choose a random source port for their traffic and then the combination of source port and source IP identifies the client?
Also, one of easiest, formal methods is Design by Contract which can be emulated with asserts for pre- and post-conditions plus main invariants. Quite useful to encode operating assumptions in there. Such can also be used for automating test-case generation, fuzzing, dynamic analysis, and covert channel analysis. Oh yeah, definitely do a covert channel analysis on anything that gets near the keys to plug any leaks.
Yes this absolutely is the goal. I'm trying to keep the codebase as small and clear as possible.
> Design by Contract
Working out the precise pre and post conditions of every state would indeed be worthwhile. For all state transitions of the protocol, I've worked out by hand every possible combination and accounted it for it. I should publicize this work (i.e. type up my handwritten notes), and then let others join building on top of it.
That's actually... great. It's exactly what high-assurance security did in kernels and VPN's. I'm impressed. You should definitely write that up and share it for peer review. Also, since you've done it, the next step is to review each one to make sure security properties are maintained or it fails fast, safe, and loud. :)
This looks interesting and I will be sure to check it out when I get the chance. I see from the examples on the site that it's IPv4 everywhre. Is there IPv6 support ? If not, is it on the roadmap ?
Thanks for the documentation suggestion.
For example, there are a myriad of products based on BSD; everything from pfSense at the low-end, the Juniper routers at the high-end.
If the goal is to make money by licensing the code for non-GPLv2 use, then the GPL might make sense, if you're unconcerned about limiting market adoption by precluding integration by non-GPL products.
Couldn't it still be fast and written in a safer than C language considering the security context? (Or we'll still need to get grsec in there... :)
Cheers!
On a microkernel system, the standard way of connecting to the network stack might be through direct shared memory; and on such a system you could manage to run the VPN in userspace at essentially no cost.
However, on Linux, most of the network stack is in the kernel; the ABI for interacting with network hardware is stable. So while in theory you could write an efficient userspace VPN, it would require you to modify or at very least recompile your applications.
One disadvantage off the bat is that every operating system from OSX to Windows 7 to Linux to *BSD already has support for IKEv2 and MOBIKE caked in.
Since we appear to be bolted to the top of the thread, I'll repeat a point I buried deeper in the thread:
Maybe some readers of this thread are in the habit of casually bringing up random VPN connections between individual Linux machines. But in my experience, people tend to invest in VPN configurations so they can connect developers and ops to deployment environments and company networks. That's like, the big use case for serious VPNs.
In a serious VPN deployment, if you lose code execution on the VPN server, whether the attacker took the kernel or just a userland process is somewhat far down the list of concerns you have to deal with. A better first concern is "how many machines inside the VPN did the attacker just get code execution on because they can now make direct TCP connections to all our internal services".
From that vantage point, I'm a little less concerned about whether my VPN server is at CPL0, and a lot more concerned with things like "how complex is this code and how straightforward is it to audit".
Rather, I expect what the parent wanted to assert was that a novel malformed packet to a badly written VPN shouldn't bring down the entire VPN gateway machine, but rather just the VPN daemon itself, which can then be brought back up by init(8).
Or, to put it another way—even though the Linux kernel is monolithic, it can still pay to treat it as a microkernel and isolate fault-prone code outside of it.
This was one of the original big benefits of running e.g. the NTFS-3G in FUSE, when that support was experimental: the NTFS-3G driver did crash—often!—and your OS stayed up. The—usually external—drive would just get unmounted, and have to be fsck(8)ed.
Now, the argument you used also kind of answers this: in full deployments, we've got dedicated VPN gateways on either side of the connection, so if the VPN process crashes, the whole machine may as well have crashed, since it's not doing much of anything else anyway.
That's probably true for most networks, but I could think of one exception: a VPN gateway serving multiple customer sites. You'd really rather that one VPN "receiver" process crash on that host (like one webserver instance serving one request) than that the whole machine crash.
> But in my experience, people tend to invest in VPN configurations so they can connect developers and ops to deployment environments and company networks. That's like, the big use case for serious VPNs.
You're missing the other big use case: VPNs used to get one's internet access through a different network, in order to evade the Great Firewall, limit NSA surveillance, increase security on public Wi-Fi networks, or just watch Netflix in the 'wrong' country. These are often commercial (not that commercial VPNs are going to use WireGuard as long as it's Linux only, but hopefully it'll be ported to other OSes before long). In this case there are two ways the client could be attacked: the network itself could be modifying packets or doing a MitM attack, or the server could be hostile (either after being hacked itself [maybe with the same exploit] or, mostly applicable to government attacks, because it was set up as a honeypot).
With a network attack, the VPN code is one of the only bits of attack surface available to the attacker, especially if they don't know your IP on the other end. A hostile VPN server you route all internet access through, in comparison, already has quite a lot of power over you (hopefully you access all important services over TLS), and at the very least makes it easy to deliver (easier-to-find) browser exploits - but in theory the VPN implementation provides some additional attack surface which bypasses all sandboxes. Also, paranoid users may have a setup where their browsing is done inside a VM, whose connection is routed through a VPN on the host end. This helps ensure that they're not accidentally passing traffic through the wrong interface or leaking their real IP, and provides some protection against attackers who get code execution with the goal of finding said IP (see FBI Tor malware). In such a setup, any networking code running outside the VM is an interesting target.
Linux isn't all that secure against unsandboxed user processes, but VPN code has a simple enough interface that it could in theory be sandboxed effectively using seccomp-bpf.
The Great Firewall tries to block VPNs anyway, so a Chinese attack isn't all that realistic as things stand, but who knows, it might be nice if WireGuard grew some kind of obfuscation support. And that leaves the NSA and others.
Anyway, I don't actually think there's a problem with WireGuard living in the kernel! - because I don't think it presents much attack surface. It's a very small codebase, and performance is important for these applications. Actually, I wholeheartedly approve of WireGuard: it seems like a really nice design, and I've experienced the pain of just getting other VPN implementations to work, never mind securely. But if there is a vulnerability in WireGuard, then for the reasons I've stated, I think its placement in the kernel matters.
I doubt WG is a real candidate for the case you're talking about.
Web proxies are suboptimal because they only work with web traffic, or even if you use HTTP CONNECT, only TCP. SOCKS is mildly better at a protocol level, with SOCKS5 supporting both TCP and UDP, but still suboptimal if the server end is able to give each client its own IP address (not so common with IPv4 but easy with IPv6), due to the existence of protocols other than TCP and UDP. More importantly in practice, SOCKS clients are typically implemented in userland rather than the kernel, so each individual application needs to be configured to use the tunnel. There is software that lets you use any program with SOCKS by hooking socket API calls, but that's essentially a hack, and has many practical issues. Much nicer to just have the tunnel as a network interface like any other, and let the kernel's network stack work as it was designed - which is usually done with VPN software. VPN also lets one host transparently handle proxying for another, e.g. a router for a home network, or the host in the VM setup I mentioned in my original post.
You're talking about two completely different use cases with very different goals and potential issues.
You mean like lvs/ipvs? [0]
> How about some TLS termination?
You mean like the KTLS project from Dave Watson/Facebook Engineering? [1][2]
[0] http://www.linuxvirtualserver.org/software/index.html
WireGuard, on the contrary, is around 4000 lines of core code. This is super short, and allows for a single individual to audit it. By focusing on simplicity and ease-of-auditability, I hope to achieve something that can be considered secure.
Compare this to the massive codebase of OpenVPN/OpenSSL and the various other userspace (read: slow as mollasses) VPNs. WireGuard is considerably shorter and simpler than those too.
Also, keep in mind that on Linux, jumping from userspace to ring0 execution isn't exactly difficult. So in practice I'm not sure it matters.
I make a living by exploiting kernels. This makes me very aware of the questions associated with running this in the kernel. I've tried to write WireGuard using my experience breaking similar codebases; WireGuard is short enough that it should receive the thorough attention that this sort of software deserves. By focusing on simplicity, I think we can in time achieve something quite trustworthy.
Finally, there will be a (necessarily lower speed) cross platform implementation released that's written in Rust. At the end of the day I suspect that all parties will find themselves pleased.
https://lists.zx2c4.com/mailman/listinfo/wireguard
And if you're savvy and want to get involved with development, we need all the help we can find!
edit: Just read a little bit of its source, apparently you have to disable std, and he replaces it with his own linux_std that at this point has only printf implemented. So yeah it'd be a pretty intense project if one would attempt it :)
(disabling std is standard in environments like this; and you still get libcore, which is a lot of stuff!)
This is simply not true [1] [2].
[1] - http://dpdk.org/
[1] - http://dpdk.org/doc/guides-16.04/prog_guide/kernel_nic_inter...
For a VPN that intends end-user device deployment it's kind of silly to require specific hardware.
I know and I don't much care to see this crappy tradition carried on with this new protocol. :(
The less irrelevant code you have around in a security box, the better. The attack surface shrinks.
If the author actually meant to licence it under the GPL 2, he should read and follow the instructions in the GPL itself, more specifically the section at the end titled “How to Apply These Terms to Your New Programs”.
But any chance of LGPL? Or would we need a conplete reimplementation for BSD?
FreeBSD doesn't like GPLv3, but OpenBSD and DragonFlyBSD don't have a problem with it. I don't know about NetBSD off the top of my head.
IMO this seems like something that should be GPL and I think you needn't change it.
Congratulations.
How do you deal with MTU? OpenVPN's handing is particularly bad [1]?
Is the whole protocol in-kernel or just the data plane?
For admins who want to provision large number of clients, do you ever plan to implement some kind of certificate hierarchy?
MTU is dealt with the same way as other kernel space tunneling devices.
> Is the whole protocol in-kernel or just the data plane?
All of it, which is why there's so much emphasis on simplicity.
> For admins who want to provision large number of clients, do you ever plan to implement some kind of certificate hierarchy?
Rather, the goal is to make the kernel interface very simple and minimal, so that admins can then glue ontop whatever situation they want. This could be certificates, or pigeons, but the important thing is that it's a layer ontop (in userspace), and not intimately bound to the core.
If it could work better than ocserv, this vpn will be a milestone.
> So what's the point ? Linux is a niche market. Mac & Windows will be the vast majority of your users and won't enjoy your killer kernel-based feature...
Even when Windows/Mac are the vast majority of clients, each client only needs to process its own traffic. The VPN gateway is more likely the bottleneck, since it needs to process everyone's traffic. There are many organizations that would have no problem running Linux on the VPN gateway and could reap the benefits.
Also, the Linux in-kernel implementation is just the first(?) thing being released. WireGuard is a VPN protocol and its killer feature is good security that's simple enough to implement in 4000 lines of code. This simplicity makes it easier for someone to write a Windows/Mac kernel driver.
It makes a UDP connection, it sends and receives some packets and then exits again. It's by no means a functional VPN, just the handshake.
No doubt it'll grow into an actual client, but don't be too amazed by it just yet :D
It's not possible to do highspeed crypto from userspace while still integrating with the usual networking stack. Tun devices result in copies and context switches which makes things quite slow.
This also adds quite a bit of simplicity.
But sure, I agree, it's scary. But not nearly as scary as the obscenely huge xfrm/ipsec layer in the kernel. Or the dozens of other kernel tunneling deivces available. And I dare even say it's not as scary as most userspace VPN solutions like OpenVPN which are massive code bases. WireGuard, in contrast, aims to be quite minimal in its codebase, so that it can be audited by an individual.
I didn't make the title of this HN post, unfortunately. But do note: there _will_ be userspace implementations of WireGuard. It's most certainly not tied to the kernel. It's just that the OP saw me present at a kernel conf about it. :P
http://genode-labs.com/publications/mikro-sina-2005.pdf
Far as keeping it in kernel, it might be implemented in SPARK and Rust to see if it gets through static and dynamic (if necessary) checkers. Test and fuzz it heavily. If C, use strong, static tools to show correctness and/or compiler transformations like SAFEcode (or Softbound+CETS) to immunize it. I know Criswell runs Linux kernel on SVA-OS with SAFEcode but not sure if S+CETS can do it. Such approaches will knock out most of the flaws that lead to code injection.
Not trying to discount the author's work, and I haven't reviewed any of the documentation, but until this gets evaluated by a CC lab (or like independent group) I would not implement in anything you care about.
First, this is Linux, and if someone manages to get code execution in your unprivileged userland program, the odds of you keeping them out of your kernel are already pretty low.
Second, and more importantly, this is VPN software. If your VPN gets popped, you probably have bigger problems than whether the attacker got the kernel on the VPN server, as opposed to just being able to create arbitrary network connections from the "inside" of the VPN.
If you don't want to run it, don't run it. If you're going to ask me whether I'd trust OpenVPN more than WireGuard, no, I do not.
"If someone manages to get code execution in your
unprivileged userland program, the odds of you keeping
them out of your kernel are already pretty low."
https://pax.grsecurity.net/docs/PaXTeam-H2HC15-RAP-RIP-ROP.p...
Are you running the grsecurity patches? If not, everything they do that your kernel doesn't do is another reason your userland will inevitably cough your kernel up to an attacker at some point. Even with those patches, it's still an inevitability; it's just that interval is longer.
I don't run them! I do something easier: I assume that if I lose control of userland on a Linux machine, I've lost the whole box.
It is especially weird, though, to see security people engaging in this kind of risk reasoning. It seems to me that most of the times you lose code execution on your VPN server, the kernel security of the VPN server is pretty far down the list of problems you need to deal with.
So, despite severity of VPN-level compromise, there's advantages mainly in detection and damage limitations if it's a strongly-constrained process. Hell, I've done prototypes that used fixed memory with only networking privileges needed that logged their own behavior to immutable storage. You could compare the execution and logging to a profile. Outside that range, it was a crash or attack. Clive Robinson taught me that last part. Others were Orange and Red Book 101.
As a "kernel professional", I'm laughing incredulously at this. A lot.
The end goal ought to be getting more things out-of-ring 0, not adding more things to it -- I'm never going to say that "I'm not concerned" or that "It's a non-issue" (although I will say that I'm a lot more worried about the massive amounts of crap code in WiFi drivers than I am about a VPN implementation, generally speaking).
In the OpenVPN case, numerous vulnerabilities have been limited to only DoS exploits due to the zealous use of assert(); in the kernel case, that brings down your machine, and quite often, assertions are compiled out of the release kernel due to their overhead in critical paths (whether they should be is another question).
We do have ways to tackle the kernel->userspace performance divide when it comes to packet processing, e.g: https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
netmap is a framework for extremely fast and efficient packet I/O for
both userspace and kernel clients. It runs on FreeBSD and Linux, and
includes VALE, a very fast and modular in-kernel software switch/data-
plane, and netmap pipes, a shared memory packet transport channel. All
these are accessed interchangeably with the same API.
netmap, VALE and netmap pipes are at least one order of magnitude faster
than standard OS mechanisms (sockets, bpf, tun/tap interfaces, native
switches, pipes), reaching 14.88 million packets per second (Mpps) with
much less than one core on a 10 Gbit NIC, about 20 Mpps per core for VALE
ports, and over 100 Mpps for netmap pipes.
Userspace clients can dynamically switch NICs into netmap mode and send
and receive raw packets through memory mapped buffers. Similarly, VALE
switch instances and ports, and netmap pipes can be created dynamically,
providing high speed packet I/O between processes, virtual machines, NICs
and the host stack.OpenVPN uses TLS. The whole point of WireGuard is to shake off that complex baggage.
Granted, but this should generally be behind a shared secret HMAC, according to best practices, limiting the scope to those with the key in hand.
> The whole point of WireGuard is to shake off that complex baggage.
That's great, but also has nothing to do with running in-kernel. I'm not ready to give either product a clear pass on their risk profiles.
I sometimes think Apple removed /dev/tap from iOS because it provided too much potential freedom.
Not every user needs in-kernel performance for their daily routine but there's certainly an argument that every user could use a decent "VPN" that could run on all their computers.
Alas, OpenVPN mindshare is rather strong, to the detriment of existing or future userspace alternatives.
It kind of seems to fly in the face of the multiple layers of userland and kernel separation engineering we've seen go into other attack surfaces like browsers?
But all things are not equal. Any code execution on the VPN server practically implies that the attacker controls the VPN, which is game-over for most VPNs I've ever worked with. So my priority would be "not having those kinds of vulnerabilities at all". I'd rather have simpler, easier-to-audit code than userland, if I have to choose one.
You may be surprised, then, that most of the standard VPN tech — IPsec, GRE+PPTP, VxLAN, etc. — operate in the kernel (with some help from userland apps for things like IKE).
In advance of that, I'm curious: What's the tl;dr: on how this compares to Tinc? In particular, I'm wondering what WireGuard's mobile story looks like, especially in comparison with Tinc's (which is pretty rudimentary as far as I can tell), and about the extent of effort that's likely to be involved in ongoing configuration management.
Really?
1. RSA & ECC are both too slow for this to be performant
2. Padding???
Certainly you meant "encrypt using a negotiated symmetric key". And earlier you meant to say "negotiate symmetric key with peer"
Thanks!
I fully agreed that being in-kernel is the right choice for performance, but the chosen constructs excludes the possibility of using any type of existing crypto hardware accelerator that shines in the IPSEC use-case (cache cold data == no cache flush overhead, fully async processing of packets with DMA chaining). Time to start lobbying SOC vendors :)
Fortunately AVX2-accelerated (and soon AVX512-accelerated) ChaPoly is super fast in pretty much all hardware.
What are the advantages of doing UDP instead of IP?
It seems like, all else being equal, doing IP has the advantage of working with UDP and TCP out of the box.
Yes, our goals are much broader than VPN. You can think of ZeroTier as a virtual smart switch built on a P2P network; a "smart switch for Earth." We want to totally abstract away the physical network. A lot of ZT's complexity over WireGuard is about working around the underlying network (NAT-t, dual-stack support, dealing with timeouts and path failures, relay fallback, peer location and roaming, redundancy, etc.) as well as smart switch features and robustness against DOS attacks and similar things.
Nevertheless ZT's core code really isn't that big. The core protocol implementation is 23,000 lines of C++ code. This includes many very long comments documenting the protocol, etc., so total lines of actual code is probably more like 18-20k. It's comparable to an embedded TCP stack.
WireGuard does have some things in common with ZeroTier, such as the use of cryptography to identify endpoints and eliminate the hard-coding of endpoint addresses. This is one of many ways that IMHO cryptography actually simplifies networking. I really like the WireGuard design in general and I think it has a somewhat different use case from ZeroTier, namely fast long-lived provisioned links across WANs and insecure LANs. You could use ZT for that but this being in-kernel makes it likely faster. In any case I hate IPSec (over-engineered mess, terribly hard to configure) so any alternative to that is welcome.
(Edit: there are things you can do to speed up user-mode networking like zero-copy data paths, etc., so it's not guaranteed that this would always be faster. But that's another topic.)
Sorry for my English.
Is there any worry that someon might perform a DoS attack against a client by replaying a valid packet from the target from different hosts so that the other servers cannot correctly route to the real IP of the target? This is based solely off of the description on the homepage and it might not be possible due to implementation details I'm unaware of.
But an attacker with an active man in the middle position might change the outer (unencrypted) src IP of a packet. However, that src will need to be able to produce authentic replies; otherwise the session will quickly be deemed invalid, so that mitigates any potential very oddball amplification attacks you could dream up. And regardless, an attacker with this kind of active man in the middle can already drop packets at will, so this isn't a vulnerability.