Easily accessing all your stuff with a zero-trust mesh VPN
changelog.complete.org
changelog.complete.org
The gist being that you are no longer making unverifiable trust assumptions about the identity of nodes connected to your network, as you have encrypted links between all nodes (not just between external nodes and a VPN gaetway to a non-zero-trust internal LAN) based on their verified identity and allowed access.
A node's identity can be verified by its public key, so you no longer implicitly need to trust that only known and expected nodes are plugged into the switch in the office (for example). I do not have to care about whether the physical network infrastructure can be trusted (zero trust) because I can verify the identity of all nodes.
This of course does not mean your entire architecture is zero trust. There are pretty much always trust decisions and assumptions somewhere, but that is why it's described as zero trust [mesh] networking, not zero trust complete end to end systems.
edit/aside: some people prefer the term "trust minimising" rather than zero trust to emphasise that there's always trust somewhere and you are just pushing it outside of some component or layer, and I agree this is probably a better/more accurate term.
I would instead assert that Tailscale's/Wireguard's approach of connecting devices is not zero trust as our focus should be on protecting 'services', not 'devices'. This requries micro-segmentation, least-privilege, attribute-based-access control, authenticate/authorise-before-connectivity as part of the overlay.
Allow me to rephrase that for you ...
Tailscale, a company whose business model relies on selling Wireguard-based VPN products, tries to convince people/prospective customers that a VPN can be shoehorned into the Zero Trust concept.
If all you have is a hammer, everything looks like a nail as they say....
I'm not well-versed in zero-trust networking, so I may be missing something fundamental.
What you describe here sounds a lot like what we’ve been building at Ockam: https://github.com/build-trust/ockam
Zero trust isn’t about doing it in one place, and then leaving all other doors wide open (although that is one possible realisation).
From a security standpoint, it should be about layering, and making it and based on relevant security anchors. For most people, the fact of being on an authenticated local network is a huge positive indicator. (Or more to the point, being outside of it is the indicator.) This is why, generally, home routers don’t leave the admin interface exposed to the WAN. Are we to advocate dropping the “deny WAN for /admin” rule simply because the admin GUI already has password access control?
Giving benefit of the doubt, perhaps one of the tools mentioned is doing that implicitly, in which case the author hasn't mentioned it. I think it's ZeroTier but I'm not sure how it's being configured to do so.
Without benefit of the doubt, the author may have misunderstood what ZTNA is and the title is misleading.
So while I think you 1st sentence is a bit whooly, your 2nd is bang on. Wireguard (and ergo Tailscale) is focused on connecting devices, not services, and being on the mesh gives you access to other devices/endpoints. To achieve zero trust networking you also need micro-segmentation, least-privilege, attribute-based-access control, authenticate/authorise-before-connectivity as part of the overlay.
There are less obscure uses for broadcast in those networks. For example mDNS "just works". You may not have an automated DNS at the service level, but you can use avahi to discover your other services or use .local without any extra config.
Seems like a good value add-on for services that already require setting up encrypted tunnels.
Other than that, yggdrasil and tinc sound so much better than the alternatives.
There's really no redundancy. If your headscale node goes down it can take your tailnet with it. In particular, if you drop in some ACLs that have issues, your entire tailnet can drop until you get it fixed. The recent (0.21) "configtest" can help there, but it still feels a bit brittle.
Headscale can use a lot of system resources. ~100 nodes can saturate a t3a.small instance in cpu time and disc access. Reducing the update frequency can help, but there are hard limits here. I'm imagining much of this is database updates to sqlite, but I haven't tried switching to an external postgres server yet to see how much of the load is database related.
I mean, the usual access VPN solution. It’s not peer to peer, but you can set up or rent a server near clients, like in the same city.
Also, Netmaker is kernel level WireGuard which is much more performant. If you’re just using Tailscale to talk to clusters from your MacBook, for instance, userland WireGuard is probably going to be more than enough. When you need servers to talk to each other, though, you usually need the extra perf
At my previous company our VPN server was on a VERY good network, including InterNAP traffic optimizers.
One of our employees ran some testing of it and found that in most cases going over the VPN was faster than natively using his Comcast connection. Comcast would tend to route the traffic over their own network links as much as possible.
In the case of the VPN server, it was well enough connected that Comcast could get to it quickly, and the good connectivity it had (largely thanks to Level-3, but the InterNAP optimizer would help it pick great connections in any case), would get it to the final destination faster than Comcast's network could.
I'm just saying: It's not that simple.
But, to clarify, when you say "relay node", are you talking VPN traffic relays (DERP nodes in tailscale parlance) or relays to the public Internet (exit nodes in tailscale parlance)?
If the former, tailscale goes out of it's way to avoid the DERP servers and instead route traffic directly between the nodes (hence the "mesh"), so it doesn't gain the benefits of hub and spoke that I was speaking about, unless the src/dst nodes can't directly communicate.
If the latter, I don't know of any mesh that has smarts about optimizing the reachability to the public Internet and shifting traffic between exit nodes to get better reachability.
Can you mention a VPN that has the abilities you are speaking about, because I'm not aware of one, unless you somehow did something like integrating BGP into the exit node selection combined with something like a InterNAP traffic optimization appliance.
Your VPN server also basically becomes a network switch in this case, doing the backhaul between all clients, which is likely to be expensive and slow compared to peer-to-peer connections.
If you try to improve this by making the clients entire networks with switches/routers connected to each other only via internetwork VPN tunnels, then you also lose the property of the network that only known, verified nodes can be on that network. Not only can you not verify all nodes you connect to, neither can the VPN server (e.g. someone could plug a malicious or insecure device into the switch in your office, that would then have the same access to the network as other nodes).
Sure, the traffic is decrypted on the VPN server in the hub-and-spoke case (even if most traffic is already encrypted by TLS and applications) but in both cases the server in the cloud could compromise the connection (in different ways, but correct me if I’m wrong that malicious servers used initially to establish the peer to peer connection cannot compromise the network, even with the Lock feature which prevents addition of unsigned public keys). You have to trust the cloud provider.
Somehow people think mesh VPNs don’t need opening ports or are peer to peer. Not exactly: you just pass it to someone else to open ports for your network (used at minimum in the initial connection, and in the case of relays for the entire session). You have to trust the cloud in both cases, though in different ways. Further, at least your identity provider could act as an administrator with full privileges.
Yggdrasil, assuming nodes operate their own identify based firewall as described the the original article, and tailscale if you use it with self hosted headscale rather than their coordination server can improve on this further.
At work we run headscale on a physical machine we control for now.
Personally I’m very interested in mesh networks like Yggdrasil and secure, radically open overlay networks and virtualised services in general.
You do have to trust the control/coordination server, though you could also run this in confidential compute.
Also, if you are interested in mesh networks, check out open source OpenZiti - https://github.com/openziti
In my use case, I have a modest number of nodes. Although nodes learn of other nodes from each other, I use ansible to keep each node's config updated.
I use vpncloud (and previously, tinc) between docker hosts. So, you have to be careful about interface MTU's inside of docker, particularly if you use containers based on Alpine.
My gut is that tinc is going to be as good as anything else here -- specifically what this means is that his "for your friends" idea doesn't hold much water for me.
I.e. -- either you have friends techy enough such that you could get tinc going, or you don't, and at this level ALL of these solutions are going to be too annoying and complicated for them?
For the non-techy types, I don't think any VPN like thing has been built yet? Would have to be like "syncthing" level easy.
It's fully private and doesn't rely on public servers, like some of the mentioned solutions. Adding new peers is a minor chore, but if you have a network that doesn't change frequently, this is not an issue. Also unlike most of the alternatives, it runs absolutely everywhere.
I had it setup on my pfSense router, and on my Android phone, and it worked great. Not sure why the article says that mobile support is iffy.
I also used it professionally to setup a small Docker Swarm network over the internet. It also worked without issues, and was relatively simple to setup and maintain.
Performance could be an issue, as the article mentions, but unless you're transferring large files, in practice it's good enough.
I'm currently looking into setting this up again for personal use, and was hoping that there would be native mesh support within Wireguard, which doesn't seem to be the case.
I have a side project in mind where IoT devices (dropping in and out of 4G and/or ad-hoc wifi or bluetooth) need to communicate and send/receive events, reconciling with event sourcing or whatever
The tough parts of the architecture for this seem pretty generic so I'm hopeful that something already exists!
I had a lot of fun abusing this to make a serverless[1] video streaming platform. The video streaming ended up being shit with fundamental issues I don't think I could solve. but it was fun to put together the proof of concept.
1. actually serverless. not "the server is in the cloud" bullshit people usually mean when they say serverless.
How did they solve or work around what you couldn't solve?
Curious due to some deeply engrained dislike of anything centralized :p
The theory was to listen to an incoming video stream chop it into 2-3 mb segments and publish each segment via ipfs, the program then notifies everybody watching what the next segment is via a signed ipfs pubsub message.
To watch a stream you get your streamers public key(so you can identify their segment messages are valid) and subscribe to their ipfs pubsub channel. as the software gets segment messages it downloads the segments via ipfs and reassembles the video.
Theoretically it would scale automatically to the population of people watching the stream. with the watchers providing the scale infrastructure.
realisticly ipfs pubsub does not scale but the real problem was the lag. the lag started at around 2 minutes and went up rapidly. My impression is that streamers value interactivity with the watchers and would not tolerate such a large lag.
My proof of concept code. it is very primitive, no ui, designed (both client and server) for the user who enjoys a unix style environment. I think about trying to finish it every now and then but have not yet found the motivation.
For iOS there isn't. Last time I checked there's only one without a GUI that needs root.
PS tinc is not fully zero trust. Every node can connect to every other one. This includes vps nodes you'll probably use for firewall traversal. Other systems have a 'lighthouse' concept where the vps just coordinates traffic but isn't able to actually read it.
At the Tinc level, they can't connect to you unless you have their public key configured locally.
At the IP level, set 'StrictSubnets = yes' in the main config file to prevent nodes not explicitly configured locally to send packets to you.
You may even install ZeroTier directly in these routers to [re-]configure router from anywhere.