Comment by lenova | Hacker News Reader

Comment by lenova | Hacker News Reader

In the self-hosted space, I've been really enjoying playing around with decentralized encrypted overlay mesh networks like Nebula. Here's the current list of my faves (all Wireguard based).

Open-source projects not-quite-prod-ready:

- WebMesh: Golang, decentralized nodes https://github.com/webmeshproj

- InnerNet: Rust, with subnet ACLs https://github.com/tonarino/innernet

- Wesher: Golang, simple mesh with pre-shared key https://github.com/costela/wesher

- Wiresmith: Rust, auto-configs clients into a mesh https://github.com/svenstaro/wiresmith

Open source projects with company-backed SaaS offerings:

- Netbird: Golang, full-fledged solution (desktop clients, DNS, SSO, STUN/TURN, etc) https://github.com/netbirdio/netbird

- Netmaker: Golang, full-fledge solution https://github.com/gravitl/netmaker

Honorable mention:

- SuperHighway84 - more of a Usenet-inspired darknet, but I love the concept + the author's personal website: https://github.com/mrusme/superhighway84 https://xn--gckvb8fzb.com/superhighway84

No love for tinc[1]?

It's the granddaddy of mesh networking, long before Wireguard, and while it's not quite zeroconf, it's very simple to setup and maintain. It also runs on everything.

[1]: https://tinc-vpn.org/

Yeah it's still my go-to because it's fully self hosted. Nebula can be too but it uses certs that can expire whereas tinc just uses keys.

And really I've been using tinc for almost a decade and I didn't really see the benefit of changing. It's rock-solid.

With the exception of one thing: I use some central nodes on cloud VPSes and they can access everything. As far as I know a nebula lighthouse can't access any of the clients. So I've been meaning to give nebula another try.

But zerotier and tailscale aren't options for me because they rely on their cloud infrastructure. I only want stuff that's fully self-hosted.

There's a great tinc android client these days too.

> It's rock-solid.

Unfortunately, I cannot confirm. Sharing my experience:

I used tinc over multiple years on production servers and it would sometimes create netsplits that did not recover. I also suspect that there's a race or bug in re-keying, which also causes disconnects.

On the netsplit issue, it was me posting alone on the relevant issue [1] over multiple years without response. (I don't expect to get any from free-time maintainers, especially on hard-to-reproduce issues, but it's still important to know that such unsolvable hurdles exist.)

When I switched to Nebula, it improved this situation. But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].

I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.

I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.

Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].

So I'm currently looking to move the Ceph into WireGuard.

Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.

[1]: https://github.com/gsliepen/tinc/issues/218

[2]: https://github.com/slackhq/nebula/issues/637

[3]: https://github.com/google/security-research/security/advisor...

That's good to know, thanks.

My tinc usage was far less demanding. Just a handful of nodes and light traffic, so I didn't experience any issues.

I did migrate to WireGuard about a year ago, and instead of a mesh network, I ended up with a hub-and-spoke configuration[1] which worked fine for my humble needs.

[1]: https://www.procustodibus.com/blog/2020/11/wireguard-hub-and...

Hey there - sorry for closing your issue! I saw Nate had recommended some commands to run and didn't see a response 8 months later, so I assumed you had moved on.

I've reopened the issue so you can continue to document and look for solutions. :)

> No love for tinc[1]?

I actually looked into tinc, and really wanted to love it because of its simplicity.

Unfortunately though, it seems that the development scene around it has been stalled for several years now, and benchmark reports show that it can't keep up with cloud speed tests when compared with Wireguard:

- Wireguard: 390.3 Mbps

- Netmarker: 369.3 Mbps

- Tailscale: 62.5 Mbps

- ZeroTier: 56.8 Mbps

- Nebula: 38.4 Mbps

- Tinc: 34.7 Mbps

- OpenVPN: 22.3 Mbps

Source: [disclaimer, article written by Netmaker CEO, so implicit bias]

(https://medium.com/netmaker/battle-of-the-vpns-which-one-is-...)

Hhmm I don't remember it being that slow, but then again, I didn't need much throughput for my use cases. Frankly, I'm surprised that Zerotier and Tailscale are that slow as well, so I'm not sure how trustworthy that article is.

I found this other benchmark[1] that places them much closer to native WG performance, with Netmaker still in the lead, and somehow faster than a direct connection. This is probably due more to Netmaker using the in-kernel WG, instead of the userspace implementation. I'm kind of curious to give it a try, TBH. :)

Hell, even OpenVPN is _much_ faster than that[2,3].

Also, yeah, tinc development has been slow for years now, but I didn't experience any issues with the prerelease versions. It's certainly behind the times now, but I also enjoyed how easy it was to configure and use.

[1]: https://techoverflow.net/2022/08/19/iperf-benchmark-of-zerot...

[2]: https://www.zerotier.com/blog/benchmarking-zerotier-vs-openv...

[3]: https://www.wireguard.com/performance/

Tailscale is using GSO/GRO now. They blogged about reaching the 10Gbit/s milestone.

https://tailscale.com/blog/more-throughput/

Replying to my own comment as I can no longer edit it:

The folks over at the Nebula team had an interesting discussion regarding the the "battle of the VPNs" article published by Netmaker I sourced in my parent comment:

https://github.com/slackhq/nebula/discussions/911

I'll throw in INET256 https://github.com/inet256

It's a specification for identity based networking. There is a meshnet and a centralized implementation. You can layer IPv6, IPv4, or application traffic on top of any compatible implementation.

There's also Yggdrasil [0], an ipv6 based completely decentralised network. You can join the public network or restrict access to known keys for a private network.

[0] https://yggdrasil-network.github.io/

filling a slightly different niche, there's also yggdrasil, which can be used to create private overlay network taking advantage of public relay nodes.

Reply on news.ycombinator.com