Zerotier – multiple vulnerabilities lead to private network access | Hacker News Reader

Zerotier – multiple vulnerabilities lead to private network access

pulsesecurity.co.nz

135 pointsmiles42 comments

Zerotier seems a bit too clever about auto-discovery/peering.

For comparison: WireGuard, by design, is a fairly simple protocol that requires some other software or out of band configuration to provide the peer pubkey list (among other things). As a result, there are multiple mesh configuration tools built on top of it, some of which support dynamic (auto-discovered) peers/tunnel establishment. This separation of concerns for the well-designed mesh configuration layers on top of WG make it a little safer than integrated options like ZeroTier in my opinion, because the peer config/establishment and tunneling protocol are separate, independently auditable pieces.

I am wary of mesh VPNs that don’t require you to statically enumerate peers, especially when I don’t fully understand the underlying authorization/trust scheme. I’m extra wary of those that don’t have a well staffed team to respond to threats, and that don’t exhibit the appropriate amount of humility when making security claims. (This is not directed at ZeroTier specifically.)

On an unrelated note, there is a really fun infosec podcast named Darknet Diaries, worth a listen.

I think the intended use case for ZeroTier is six Android phones and two Windows laptops connected to a same L2 with ~500kbps UL/DL, that has to be bulletproof, dumbproof and as close as possible to zero setup and nannying, even with Android phones constantly roaming between a Wi-Fi router in a car and Starbucks Wi-Fi and its own metered WWAN.

It's a very specialized solution, not a generic VPN. They do seem to care about security but they also probably prioritize zero setup/maintenance/service restart over it.

Like, you can post that two apks and that zip and a ZeroTier key on Discord, let that trusted pseudonymous guy to approve connection requests on ZT, tell people to clear cache and retry if it's not working, and that's all you have to do. WireGuard would take more effort to deal with.

dc: No, I've never been involved in that kind of activity, but came across ZT from a different direction it is useful for, and got what the app is about.

What is the actual intended use case, not the implementation details?

For Wireguard mesh networks is there something in the similar level of maturity as Zerotier?

Tailscale perhaps?

Self-hosting is possible by adding headscale too: https://github.com/juanfont/headscale

It doesn't have all the features of tailscale hosted server, but they seem to be actively working the list of missing features.

AFAIK Tailscale is an alternative to ZeroTier, but it doesn't involve WireGuard.

Edit: I was wrong.

Tailscale uses WireGuard under the hood.

Slack's Nebula: https://github.com/slackhq/nebula

Does Nebula use Wireguard? I briefly looked at the readme, but did not see it mentioned.

There are several Wireguard managers, but this isn't an issue with the transport protocol. It's an issue much higher in the stack, and one that requires that planets align in precise ways.

Things usually are not broken by breaking encryption. Things are usually broken via bugs in higher-level logic somewhere else. Wireguard is fine, but if the thing deploying and managing and configuring Wireguard happened to be vulnerable then it doesn't matter.

This is why you want defense in depth. Don't trust just one layer of security, ever.

It's trivial to do that with Wireguard or IPSec when you are manually provisioning everything. As soon as you add any kind of UX or discovery layer on top, things get more complex and you are by definition delegating trust.

You either trust multiple third parties such some kind of SSO provider, multiple SSL CAs (to secure https), and a configuration service, or you have some kind of cryptographic discovery system with certificates. ZeroTier does the latter, and the certs are issued by something ZeroTier can host or can be self-hosted.

> WireGuard, by design, is a fairly simple protocol that requires some other software or out of band configuration

The core ZeroTier transport protocol isn't much more complex than Wireguard and is basically the same thing. Then you have the certificates, identities, etc. As you say to make Wireguard useful for large scale or auto configuring networks you have to build something analogous on top of it.

Vulnerabilities are almost never in the transport protocol unless you're dealing with some hairy legacy protocol like SSL that has a lot of modes and can be vulnerable to downgrade attacks or unless it's a totally incompetently designed transport that uses cryptography wrong. Vulnerabilities are almost always in higher layers, management layers, authentication code, etc.

> On an unrelated note, there is a really fun infosec podcast named Darknet Diaries, worth a listen.

Are these guys paying people peoples to advertise here?

> Zerotier promptly addressed the root server identity verification issues. This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity that collides with an intended target. Zerotier client’s still accept any identity learned via WHOIS packet as implicitly valid.

ZeroTier founder here:

Clients accept identities learned from WHOIS via root servers, but not from regular nodes.

As mentioned in the writeup, this was reported and fixed rapidly. I don't think there is any remaining issue here, but I am doing a re-read and will publish a blog post in a bit.

It's also important to note that the original (fixed) issue also required a node to be offline for >30 days. So the attacker would have to know your network and the identity of a node that was has not been online for a long time and invest >$500K worth of GPU compute power (at current cloud rates) per identity target.

The certificate thing mentioned is well known and hasn't been considered a vulnerability because certificates of membership are not secrets. They are akin to the SSL server and client certificates presented on SSL connection. That being said, it wouldn't be a bad idea (defense in depth) to make it harder to get these to make enumeration more difficult. We're looking at some mechanisms for this in new versions of ZeroTier.

It's a good point that the cost could (would always?) be high, because you could probably just park a van outside the target's wifi network, given the sad state of wireless crypto.

However, ZeroTier is designed to provide encryption on top of an existing network, so by targeting ZT, you gain access to everything within that network, even in other locations.

Is the ZeroTier root code open source? Can someone run their own root servers?

The root code is open source. That's how they found the issue.

We always encourage our users not to (just) trust ZeroTier, not because we don't think it's safe but because you should never trust just one thing. Never, never, never. One layer of security is no security.

There are people who will tell you their software could contain bugs. Then there are people who will lie, or who are delusional.

Anything you care about should be gated by at least two security systems, preferably ones with different code bases from different vendors. Bonus points if they use different mechanisms or algorithms.

As I read TFA, the attack would take ~128 GPU-years to break most or all ZeroTier addresses, not per target. And the author says that there is plenty of room for optimization. And fpga or whatever might be much cheaper than GPU. And "attacks only get better."

Absolutely true, which is why the root cause is now fixed.

The proof of work component of ZeroTier addresses is a defense in depth measure and was not intended to be a primary defense. It's meant to increase the cost of an attack if other defenses fail to reduce the pool of potential attackers.

It also says below that the main vuln is fixed... Can't now figure out if I should be worried or not...

> This attack is still possible (as of 20/09/2021); however, an attacker is required to invest significant compute time (many many years with a single RTX2070) to generate a valid identity

So it's entirely feasible if you have access to any cloud vendor then.

> So it's entirely feasible if you have access to any cloud vendor then.

At around $0.50 per hour, and assuming "many many years" means 10 years, then the attack needs to be worth those $43800.

Many many means 512 in the article. But that's with lots of room for optimizations.

That's... less than I expected? Certainly more expensive that a normal person could afford, but with the levels of income disparity in the US, that's frighteningly affordable for a large corporation and within reach for highly motivated rich individual. If you own a cloud (aka Amazon wants to break into your VPN), they get major discounts on time on their fleet. If, instead, it costs $0.01/hr due to using unused capacity during quiet times of day, that's works out to be like $1000 to crack it.

As to what makes it worth it. $50k is too expensive to attack all users of this particular VPN software, but there's way more than $50k worth of information companies want to keep secret (call it competitive intelligence) being protected by VPN software the world over.

In addition to the compute power, you also need som other things:

"An attacker would need to hijack the address for a peer that has been offline for 30 days or communicate only with other peers that do not have a cached identity for the hijacked address."

If you have access to the victim's ZeroTier dashboard (or API calls), that's trivial to get. Even if there are currently no disclosed vulns in the dashboard, that doesn't mean that none exist.

What worries me is that 2 months have passed since the vulnerability was fixed and yet there is no new version released which contains the patch... So you're exposed unless you build ZT from source.

If the problem is in the ZT root servers, then the clients do not need to be patched, I guess?

The patch was on the roots and was applied within 6 hours of learning of the vulnerability. A new release wasn't needed since the issue was not in there.

A new packaged version is now available.

zerotier-one (1.6.6) unstable; urgency=medium

  * Backport endpoint mitigation against address collision attack.

 -- Adam Ierymenko <adam.ierymenko@zerotier.com>  Tue, 21 Sep 2021 01:00:00 -0700

> many many years with a single RTX2070

Unless they're lucky.

If you’re lucky you can break any encryption in under a second by pulling the key from /dev/urandom. It’s a meaningless statement without defining how lucky.

If it takes 2 years to exhaustively search the entire keyspace, you'd expect to be 50% lucky a year in, assuming the keys are evenly distributed (which is a huge assumption). If you have a lot of keys to crack and it's a feasibly short time, mean time to brute-force key recovery becomes far more interesting to you than the worst case scenario.

Which is why saying “if your lucky” isn’t helpful, but quoting time to exhaust keyspace (knowing that half the time you’ll get it in half the time) is more useful

unless the number of years that compute time is measured in has a lot of digits (like 3 or more) I wouldn't be comfortable. a RTX2070 isn't even the fastest card available now, let alone in the next generation or two. Compute performance for video cards is still rapidly scaling and that means that if the "many many years" isn't large enough it may not even be years soon enough.

I kind of wish that the metric would be X hours on MSFT or Google's largest available cluster.

Cryptographic part of ZeroTier protocol was audited last year

https://www.zerotier.com/wp-content/uploads/2020/10/ZeroTier...

The code (not the protocol) has not been audited yet apparently.

I wonder if the public key handling was audited above!

Overall, I was surprised of such vulnerability in ZeroTier!

It's existed since 2014 (first alpha release). This is the first time we've had anyone find anything.

I wonder how many corporate networks have end users using things like Zerotier or Tailscale without the admins being aware. Especially running self-hosted where you can use well-known, allowed ports for the control pieces.

Security teams have way more visibility than you seem to be aware of.

My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention.

>Security teams have way more visibility than you seem to be aware of.

I imagine in the real world, this varies greatly. I've worked in many large, non-tech, companies, and saw a lot of false bravado. They watched some things, but not much. Big companies with zero "deep packet inspection", for example, do exist.

>My advice to you and others is to always ask yourself, "Is this really worth losing my job over?", when considering any sort of security control circumvention.

Good advice, but I wasn't wondering for the sake of trying it. I was wondering in the context of this vulnerability.

Don't really need DPI to detect services and protocol patterns, just a decent perimeter device which almost everyone has.

Ex. Tons of UDP streaming from an endpoint to a cloud box sticks out like a sore thumb using little to no special detection magic.

For those who are interested, this is the official ZeroTier response.

https://www.zerotier.com/2021/09/21/incident-response-to-sep...

Reply on news.ycombinator.com