ZeroTier – Global Area Networking | Hacker News Reader

ZeroTier – Global Area Networking

github.com

76 pointsaphrax36 comments

I used ZeroTier (free) to allow my brother’s friends to connect to his Minecraft server (instead of port forwarding on the router due to CGNAT on our home internet). Based on my experience, I give it 5/5.

The client software is under “Business Source License” allowing you see what runs on your machine (unlike other options). It defaults to being dependent on a closed-source proprietary backend coordination server, to orchestrate the P2P connections.

It was easy to setup the virtual network, and it was very easy to get members to join. There’s a packet filter language that operates on every node in the network, so you can lock down traffic really well, down to a single protocol & port (e.g.: Minecraft).

The UX is elegant and non-intrusive, thought the network configuration interface isn’t the best for non-techy users.

The networking performance for our Minecraft usage was excellent. The stability, even during poor underlying connection times, was impressive.

Each node in the network is assigned an IP in a private use area (like 192.168.123.123, but you can choose the ranges yourself), allowing other software to use the virtual network effortlessly. No routing shenanigans are applied to your existing network interfaces.

I hope this product stays awesome (and as secure as it currently seems). I would be happy to migrate my own professional virtual network needs and pay for a premium service if I knew for sure the coordination/orchestration server wouldn’t disappear within the next 10 years, rendering the service dead. Will definitely use for temporary free stuff, though. (Tragedy of the commons, oops!)

> It defaults to being dependent on a closed-source proprietary backend coordination server

Isn't its source[0] available?

> This is our reference controller implementation and is the same one we use to power our own hosted services at my.zerotier.com.

[0]: https://github.com/zerotier/ZeroTierOne/tree/master/controll...

I haven't talked about this before, but I found a pretty severe vulnerability in the Central API (the thing that backs the web UI) back in 2018. The users endpoint for a network was returning everyone's API tokens, including the network's administrator's one. Obviously that lets you do a lot. I reported it and they fixed it very quickly, within about 24 hours. Nothing to fault in the response, but I have been a bit skittish since.

These days I use Tailscale for my home setup, which is similarly awesome. Both obviously have closed source components. I was convinced because (1) you can operate a Tailscale network entirely on OAuth2 with an external identity provider, no long-lived API tokens and less for them to mess up; (2) it runs over Wireguard and I like not having to trust more people on the crypto. I had been trying to basically build a mini-Tailscale when I discovered it already existed. Others in this thread have mentioned being able to operate your own control plane nodes for it, I'll have to look into that.

The primary difference between them is that ZeroTier is layer 2-ish whereas Tailscale is layer 3-ish. Ironically enough, ZT's design is more oriented towards scalability, whereas Tailscale is more of a fully connected graph with independent encryption on each edge. ZT rules are like configuring `pf` on your whole network at once. I miss that, especially the capabilities system. With Tailscale, if I were doing more than my own machines, I would probably rely on actual firewalls rather than their miniature JSON one. But the DNS features on tailscale are unmatched. You get `ssh myothermachine` with zero configuration. That's hard to beat at home.

I cannot recommend highly enough giving one of these a go. They make the internet fun again.

I'm also a happy TailScale user (coming from ZeroTier which I still use for some of my networks).

Unfortunately I now work for a company who is also a happy TailScale user, which means I cannot use the same machine to connect to two TailScale networks at the same time.

I understand that some think this is a feature, but I liked how ZeroTier worked in that respect.

We (Tailscale employees) all feel that same pain so we continue to think about it. It's not the highest priority, but it's a priority.

Hilariously one of the reasons we switch to zt is because tailscale is being an arse in sharing private networks when you're not using a business email (literally felt like a bait and switch).

In zt i had a colleague join the network in less than 5 minutes. Just told him download and install the client, type something on cli. What you see is what you get.

ZeroTier is amazing! And also support OpenWrt.

But bad news is that their license isn't GPL anymore: https://www.zerotier.com/pricing/

Personally as a free software zealot, I'm actually pretty happy with zerotier's licensing. The BSL license they've chosen reverts to MIT after 5 years, so there's very little risk of vendor lock in. As near as I can tell everything except their web interface has the source available, so there's nothing that makes you dependent on their infrastructure.

It means that you need to negotiate with them if you want to build a product on top of their work (unless it's older than 5 years) but other than that minor compromise it keeps in spirit with free software.

ZeroTier founder here: we don't love the BSL, but it's helped defend us so far from marketing shell startups and huge corporations just taking our stuff and rebranding it and saying "thanks, we'll take that now." Networking is a particularly bad market for this.

In the future we want to do something better for licensing, but we're thinking carefully about it and researching it a lot since we don't want to change licenses too much.

My ideal license would be free/libre for individual humans and something closer to AGPL or BSL for for-profit work and corporations, but that is virtually impossible to do with a license that will work in every major jurisdiction. It sounds straightforward but according to lawyers I've spoke with it is not.

We also want to document and eventually standardize the protocol, probably post-V2. I'm calling it GVSP: Global Virtual Switch Protocol. There's a few things we want to revise first.

Some questions about ZeroTier, particularly in comparison with Tailscale.

Does ZeroTier require a central coordination server for authentication and to distribute public keys?

That was a problem with Tailscale that prevented me from using it. The provider apparently could inject public keys into anyone’s network. I don’t know if the situation is any different in ZeroTier.

I see there are “moons” and you can install your own moon. How much is the effort for self hosted version?

Another question: How does ZeroTier bypass NAT if the connections are peer to peer? Is it similar to Tailscale (UPnP and STUN, followed by relaying)?

> The provider apparently could inject public keys into anyone’s network. I don’t know if the situation is any different in ZeroTier.

https://github.com/juanfont/headscale is a (not yet complete) free software control server for Tailscale if you want to self-host.

You can self-host ZeroTier controllers, in which case this is impossible. ZeroTier roots can be federated too, but that's less important for this concern because roots have no authentication authority. They're just like STUN/TURN servers.

>Does ZeroTier require a central coordination server for authentication and to distribute public keys?

You can run your own network controller, although that option isn't very well documented. If you did, that network controller would be the only thing that had the private keys that sign changes to the network configuration. The basic network controller is bundled with the zerotier builds, but there's no web UI or anything like that.

>I see there are “moons” and you can install your own moon. How much is the effort for self hosted version?

It's easy to set up a moon. By default every zerotier install can also work as a moon. The challenge is more in configuring all the other nodes to use that moon. Some platforms, like android/iphone, I don't believe can be currently set up to use custom moons. There's currently no way to remove the default zerotier moons, as they're hard-coded.

I think the only thing you really get out of a custom moon is that it can act as a relay server, and can work in an environment where there's no internet at all. I wish that it was easier to push moon config to other nodes, as right now the feature is kind of useless.

>Another question: How does ZeroTier bypass NAT if the connections are peer to peer? Is it similar to Tailscale (UPnP and STUN, followed by relaying)?

Yep.

Is there a Docker container for the referenced network controller by chance?

Every zerotier node ships with the CLI network controller, but someone wrote a simple GUI for it and that is on docker.

https://github.com/key-networks/ztncui

For this use case, the Nebula mesh VPN has the advantage of being rather more fully open source.

I use wireguard to provide access to my home network from outside. For that, nebula would have the advantage of being a single subnet and handling point-to-point communication instead of everything going via the tunnel endpoint. I've failed to get multicast routing to work over wireguard which would have been good for DLNA. But wireguard also serves the extra purpose of protecting my Internet traffic if I use untrusted free wifi somewhere.

Other things (I haven't yet investigated) in this space include: https://github.com/wiretrustee/wiretrustee https://blog.tonari.no/introducing-innernet https://gravitl.com/

I've used zerotier for a good 5-7 years now, and can honestly say it's pretty great. I use mostly for personal stuff, some business, mostly letting folks into my house to hit my storage because it's perpetual, and supports pretty much every device. I use this with rpi's as well with quad/octal serial cables as term servers with customers too.

This includes my synology filers, android, ios, pi, win, lin, mac, ddwrt, and tons more, but this is at least what I've used it with so far.

I saw Slack's Nebula release, and thought "oh, they just recreated zerotier", and recently someone told me of Tailscale much the same. Now Cloudflare seems to be doing like for their Quick Tunnels and treating it with the buzzword zero-trust to get enterprise attention.

Zero-tier is and has been way ahead of the game for a long time. I'm a bit surprised someone hasn't just acquired them by now. So many of the "enterprise" sd-wan and zero-tier solutions suck out there, but traditional infrastructure folks don't understand how this overlay networking works to even consider it. Their loss, but obviously Slack and Cloudflare do.

Same boat here. I've turned so many friends on to software defined overlay networks over the years and most can't understand how ZeroTier is free for everything you get. I use both Tailscale and ZeroTier for different purposes. With ZeroTier L2 bridging comes at less of a cost than with ZeroTier. I honestly wish Tailscale has a bit more flexibility and functionality in their routing capabilities. Both are fantastic though and I highly recommend both!

I'm also surprised that ZeroTier hasn't been acquired. I was working for a prominent network security vendor about 6 years ago and mentioned to some higher level product folks that ZeroTier is basically "next-gen VPN" and how it would be a fantastic addition to our product suite. I remember getting snide comments about we already did what ZeroTier was solving for. However DM-VPN type implementations aren't remotely like how ZeroTier solved for and now I think the light bulb has likely lit up for those folks given how all the Zero Trust / Perimeterless / SASE / SD-WAN have taken off. Totally agree that most of the "enterprise" solutions are just garbage built on OpenVPN or overly rigid IPsec implementations with poor performance and even worse functionality.

Probably a great idea, but pretty poor onboarding documentation.

I read the entire manual and still have no idea what the user experience looks like once connected. How do I reach the connected nodes? ZEroTier DNS? IP? Do I pass the ZeroTier ID directly to my network stack and it’s accepted and interpreted?

It's just another network adapter /ethernet interface with its own subnet, u can reach every network member by ip, u cna push routes to members in the network settings, and future versions may also include dns names, in newer clients u can already push dns server addresses to members though

I've used Zero Tier in the past but not associated with the project.

Have you set up subnets on a Network before? If you have it's like that.

It creates a subnet any IP within the subnet range will route to your virtual network. It shows up as a Network device on your computer, as if you had another NIC.

The ID is used to connect to the Network. It plays the role a domain would play when connecting to a VPN.

You could set up an internal DNS server like you can do with other internal subnets but it's optional. When I used Zero Tier it didn't have a DNS layer so you'd have to do it yourself with Bind/a DNS server/something, not sure if it does now.

I have found tailscale to be far more superior. For one, it actually works behind double NAT whereas zerotier just doesn’t

Not true. I used it behind double NAT without problems in the past. Even in 2014.

Success in traversing double NAT with anything depends a lot on the NATs. If one of them is symmetric, you're pretty much SOL.

I had to agree with this, quite shitty step by step docs lol.

But out of frustration from tailscale we tried it, and once you get it, it's easy

I keep hearing how great are Zerotier and Tailscale are, but from internet networking point of view it seems that are they not just encrypted overlay/underlay networks?

My questions are that, can we implement the similar using purely standardize protocols and provide a GPL alternative? Since this can work globally think this is like Linux but for the internet. Personally I'm a big believer of local-first software and me think it's the future [1], and these Global Area Networking (GAN) protocol will be the building blocks and enabler for that goal. This should be easy and autonomous like normal Web technology where you can use it as it is even there is no company supporting it, but of course you still need internet service provider (ISP).

I believe many of the building blocks are already existed for example zero trust architecture initiative for security and encryption [1], and GENEVE [2] for the networking. Once IPv6 take over the world (whenever this will be) this should be even more straightforward but IPv4 based networks should be feasible. For better delay and latency we can replace TCP with Homa protocol recently proposed by Ousterhout [3], while it's designed for data center it should be better for this type of GAN based local-first software application. Or perhaps a better version of global scale Homa type protocol will be in order.

[1]https://martin.kleppmann.com/papers/local-first.pdf

[2]https://csrc.nist.gov/publications/detail/sp/800-207/final

[3]https://www.redhat.com/en/blog/what-geneve

[4]https://www.usenix.org/conference/atc21/presentation/ousterh...

> I keep hearing how great are Zerotier and Tailscale are, but from internet networking point of view it seems that are they not just encrypted overlay/underlay networks?

I think the local-first dream (which I also share) is a bit different than the promise of these overlay networks. The overlay networks are a relatively small amount of infrastructure over current net technologies that allow folks to have private networks. I wouldn't underestimate the amount of value that brings to people. For example, it's much simpler to host web services and have them bind to an overlay network interface, and then just get privately accessible web services for free. The work needed to make applications work over a true zero-trust local-first network would be much greater.

Thanks for the tip about Osterhout's HOMA work. That's really interesting stuff. I've been playing around with NNCP [1] recently which is a store-and-forward zero-trust network that also works local-first, but there's a lot more work to go to make it usable beyond just hackers.

[1]: http://www.nncpgo.org/index.html

I used ZeroTier years ago. At least at the time, on one of my computers it did interfere with my network and I had to uninstall it but I also had a VPN and Docker networking layer on my machine so I'm not sure it was ZeroTier's fault.

It has a lot of promise for something like a distributed P2P network. It also has an SDK which I thought would be cool to embed in another app to create kind of a seamless Network.

How does this compare to Yggdrasil?

https://yggdrasil-network.github.io/

One critical difference is that it looks as though Yggdrasil publicly intends to stop development once they hit a 1.0 stable release.

"If we ever reach 1.0" is also important to note.

I would recommend installing a Wireguard server on a VPS + split tunneling to escape CGNAT instead of using ZeroTier/TailScale where you have to depend on third parties to work.

related discussions:

ZeroTier - Simple Software Defined Networking (111 points, 5 years ago, 38 comments) https://news.ycombinator.com/item?id=11699122

ZeroTier: Network Virtualization Everywhere (107 points, 7 years ago, 50 comments) https://news.ycombinator.com/item?id=8235702

ZeroTier 2.0 Status ( https://www.zerotier.com/2019/09/24/zerotier-2-0-status/ ) (98 points, 2 years ago, 47 comments) https://news.ycombinator.com/item?id=21066821

LF: Fully Decentralized Fully Replicated Key/Value Store ( https://github.com/zerotier/lf ) (191 points, 2 years ago, 46 comments) https://news.ycombinator.com/item?id=20579868 (related blog post: https://www.zerotier.com/2019/06/30/lf-announcement/ )

How we moved to Google Cloud using Consul and ZeroTier with zero downtime ( https://tech.channable.com/posts/2017-10-25-how-we-moved-to-... ) (105 points, 4 years ago, 21 comments) https://news.ycombinator.com/item?id=15548642

Ask HN by the zerotier founder:

Ask HN: Critique my startup: ZeroTier One (6 points, 7 years ago, 12 comments) https://news.ycombinator.com/item?id=8193250

Ask HN: Single founder startups: real data and experiences? (3 points, 7 years ago, 8 comments) https://news.ycombinator.com/item?id=8723931

(maybe api can now tell us about his own experience as a single founder)

the zerotier blog also had a few popular posts (with fixed urls):

The Horror in the Standard Library ( https://www.zerotier.com/2017/05/05/the-horror-in-the-standa... ) (830 points, 4 years ago, 216 comments) https://news.ycombinator.com/item?id=14275805

The IPv6 Numeric IP Format Is a Usability Problem ( http://web.archive.org/web/20160528080607/https://www.zeroti... ) (323 points, 6 years ago, 196 comments) https://news.ycombinator.com/item?id=11136739 (i could not find this article anymore. was it intentionally deleted? sorry, api)

since this post seems to be controversial, you may also also note the followup: Followup on IPv6 Notation Usability Concerns ( http://web.archive.org/web/20160305151134/https://www.zeroti... )

The State of NAT Traversal ( https://www.zerotier.com/2014/08/25/the-state-of-nat-travers... ) (189 points, 7 years ago, 96 comments) https://news.ycombinator.com/item?id=8229327

Introducing Network Containers ( https://www.zerotier.com/2015/09/29/introducing-network-cont... ) (82 points, 6 years ago, 22 comments) https://news.ycombinator.com/item?id=10319747

AES-GMAC-CTR (SIV) ( https://www.zerotier.com/2019/09/04/aes-gmac-ctr-siv/ ) (74 points, 2 years ago, 45 comments) https://news.ycombinator.com/item?id=20878207

Would not trust.

Reply on news.ycombinator.com