Gravitational Wormhole: WireGuard for Kubernetes
gravitational.com
gravitational.com
(I had no idea of any of the Kubernetes products mentioned in the article prior to reading)
To try and summarize without speaking too much for projects that I'm not intimately familiar with, Wormhole takes the approach of automating the configuration of WireGuard to create a full mesh encrypted network.
Other approaches take a much more comprehensive approach, of trying to solve key signing, identity, revocation, offload to the kernel and intercept via sidecars (using ebpf), etc. These tools come with many features and an inherent complexity, where I believe there is room for a simpler tool that is easier to use and troubleshoot.
i.e. Reading kubeadm config, reusing system certs, and automating the configuration of a VPN overlay system: https://github.com/jameskeane/scrambler
--
FYI to readers, the above project was completed as part of a 'take-home assignment' that I was explicitly told was not a project being worked on.
The feedback I received indicated that this approach was not considered previously, and may indicate (also your commit dates) that this entire project may have been fabricated after my submission.
Here's the feedback from Gravitational around Oct 2018 (before this project was 'started'):
> * I do think re-using the kubernetes certs for IPSec is a compelling solution for the IPSec secrets
> * I like the re-use of the kubernetes keys/certs for ipsec as it avoids use of the PSK
At least it was released as OSS, but seriously you should be ashamed of yourselves.
I found some other resources/projects online while I was writing this which indicated that I certainly wasn't the first interviewee to be asked to do this - I evidently wasn't the last either as your project shows.
My point is that it seems Gravitational has had this 'idea' for quite some time as they've been using it as an interview question for at least the year 2018. I don't feel like that constitutes them using anyone's interview work as free labour.
Are you in contact with the Wireguard author, and how do you plan to support the Wireguard project?
No I'm not in touch with the WireGuard maintainers. I'd be happy to start a dialog.
As for contributing to the project, I'll likely take a crack at porting the netlink interface to the golang netlink project (https://github.com/vishvananda/netlink) so that the wireguard interface can be automated through golang. Right now, the project just automates this by execing the wg cli.
We're also integrating this with our commercial offering of gravity, which if we see traction I'll be making the case that we should contribute to the project financially as well.
I'm looking forward to reading the source of your WireGuard integration. Congrats on launching.
Feel free to contact us if you need - team at wireguard dot com.
TLS also has the nice property of working with all sorts of clients, even those outside the cluster that might not be using WG encryption at the network later.
To me, where WireGuard comes in is for services that aren't traditionally encrypted, or where an extra layer of security is desired.
The example case that we've seen a few times is cluster DNS. The DNS traffic internal to the cluster is just plain DNS, meaning it leaks internal information and can be tampered with on an untrusted network. Even though CoreDNS does support DNS over TLS, it's a large effort to update clients in some organizations, that is better spent elsewhere.
This is where we see the opportunity for WireGuard, is ensuring all internal traffic is encrypted. For taking applications where the security model traditionally requires a secure network, and allowing them to be forklifted into a kubernetes cluster that provides those capabilities. I'm not saying this is an ideal model over using mTLS for every connection, just not everyone has what's needed to deliver mTLS everywhere.
Unfortunately, we do lose the identity features that TLS offers in the x509 certs. Nothing prevents running TLS for services that need it though on top of the WireGuard network.
So like everything else, I think there are pros and cons to both approaches, and it depends on what applies to the specific application.
I hope that helps.
Presuming that by mTLS you mean mutually authenticated TLS, I don't see it as a good fit for internal systems - to me (as someone who has implemented mutual auth to considerable success) it's best where you have B2B applications, because you've got the three distinct parties that make a PKI worthwhile. Whenever I see a PKI in which all three of the parties are essentially identical I cringe, e.g. I keep seeing OpenVPN setups where one person runs the CA, runs the VPN server and mints all the private keys - the PKI is pure theatre in this approach, it makes no difference to anything.
I doubt in most cases that systems to be forklifted were truly secure originally...
Yeah, people generally understand that apps developed under a perimeter security mindset aren't considered secure by newer zero-trust standards.
...an opportunity is missed to fix more serious problems.
The security debt is so enormous that there will probably never be enough budget to fix it.
However, Flannel's WireGuard extension looked way too experimental, and unmaintained. I realise it is also early days for Gravitational Wormhole, but Gravitational Wormhole looks promising.
I see wormhole as a simpler / lighter weight solution. Like flannel, but building an encrypted network instead of a vxlan (or xx plugin) network. The trade offs are optimized towards simplicity and being a lighter weight option with less capabilities.
>Do you think Wormhole and Istio could be deployed together, using Wormhole for mTLS and turning it off in Istio, and some advantage in doing so?
I don't think I have a deep enough understanding of Istio to know for sure how easy it is to use them together and have never tested this sort of configuration. I wouldn't consider wormhole a replacement for mTLS, as wormhole isn't capable of embedding identity of either end into a connection. You can think of it like establishing ipsec tunnels between networks.
Where I think wormhole would come into play in this theoretical setup, is if you want a subset of traffic to be encrypted, which either doesn't play well with istio for various reasons or you benefit from additional layers to the security model.
> Do you see Wormhole's focus as being tuned to situations where the relative heaviness of solutions like Istio are not necessary and only the lightest solution possible is desired?
Yes, exactly.
> Does that mean that Wormhole should be viewed more as "include this with my application designed for small / lightweight environments" or "include this with my Kubernetes distro, and config applications running on that distro to best use Wormhole"?
Yes. Although using wormhole doesn't require any application involvement. Wormhole operates as the networking configuration for the cluster, and simply uses an encrypted protocol for doing so. So any applications that run on say a flannel cluster, should not realize wormhole is there, we just provide a kubernetes network. Of course there is always an implementation detail we may do slightly different, but that's likely a defect in the implementation we would need to address.
If Gravitational asks you to complete an 'engineering challenge' they are using you for free labour.
I think it's helpful if you (or anyone reading this) compare two projects side by side.
This is the project you have submitted:
https://github.com/jameskeane/scrambler
It is a prototype of IPSec CNI plugin using Bash. We are happy you are promoting your project online and as we've discussed during the interview, you could use it as you wish as it's your work.
This is the project that we have created at Gravitational:
https://github.com/gravitational/wormhole
It is using Go, Wireguard and is designed for production use under Apache 2 license.
I understand that similarities in the problem statement may have led you to believe that we could re-use the project idea, I'm assuring that you that we have not.
There are many OSS CNI plugins in the market solving the similar problem in a similar fashion (e.g. flannel with IPSec backend).
As a matter of fact that's the main reason we have picked this as a problem statement - to make it easier for candidates to look around for prior work.
For example, this is a submission of the engineer we've hired based on their similar execution of the assignment from April 2018:
https://github.com/webvictim/k8s-ipsec-cni
James, I wish you the best and hope that my response clarifies some confusion around this.
Sasha, CTO and co-founder at Gravitational
Were you lying when you explicitly told me the opposite or did you decide after my submission (going by commit dates) to make this a "real" project?
You didn't steal my code; you stole my _ideas_ and are now presenting them as your own original work. Shame on you.
Is there a good analysis somewhere about how typical kubernetes setups trust the network and what badness an advesary could do with kubernetes network access? How sound is this default deployment setup from security POV?
For example I think DNS is used internally for service discovery, and incoming TLS is often terminated and proxied onwards as HTTP - those could be both MITMed, right?
But requirement 2 confused me: > A Kubernetes cluster with IPAM enabled (--pod-network-cidr= when using kubeadm based install)
So, do node machines need to already be on the same network or not?
Kubernetes includes an optionally enabled IPAM, that can allocate subnets from a large subnet, into the /24 subnet assigned to each node. This becomes the spec.podCIDR field in the node object when enabled.
Instead of implementing another IPAM for the node cidrs, wormhole uses the built in IPAM within kubernetes. The only thing is, the ipam needs to be turned on. On a kubeadm based install, this is done by setting the pod-network-cidr flags during init. This sets --cluster-cidr / --allocate-node-cidrs on kube-controller-manager.
I set up a Wireguard /17 (iirc) mesh, and then kubeadm'd with --pod-network-cidr as that node's wg endpoint /24.
I suppose what I'm not understanding is how I can set up the cluster sufficiently to be able to kubectl apply wormhole if I haven't already setup a wireguard (or...) network between the nodes?
Because the kubernetes APIs are protected by mTLS between nodes, the API connectivity doesn't need to be over WireGuard, unless you don't have bi-direction connectivity for other reasons (like NAT). Although, I don't generally recommend splitting nodes between disparate networks, as it's not a good failure model for kubernetes.
How this is modelled, is when setting up a kubernetes cluster, you establish node trust using the installation method, and whatever network exists between the hosts.
This creates a kubernetes installation, but doesn't setup network. When using the flags to kubeadm, this creates a kubeadm config file, and turns on the kubernetes subnetting. So say you were to configure 10.100.0.0/16 as a pod-network-cidr. The would cause the kubernetes controller manager to assign a /24 to each node for pods. So node 1 gets 10.100.1.0/24, node 2 gets 10.100.2.0/24, etc. The addresses from the node cidr are then assigned the pods as they are scheduled by the cni plugin. The wormhole plugin, then takes care of setting up all the networking that covers 10.100.0.0/24, because kubernetes has effectively been granted management of this subnet.
So, yes, this assumes some sort of network exists between nodes, which is the network that has been used to create mTLS trust between kubelet/api on all members.
It's basically this, but with ~100 lines of bash. As simple as it gets.