Peer-to-Peer Communication Across Network Address Translators (2005)

I had always wondered how you guy's were doing it, but I had always thought that the router Vonage handed out was something special that used its own reserved ports or something that wasn't easily visible. I never looked into it deeper. It's interesting that instead something like this "hole punching" technique was used.

I've heard about the technique before, but never investigated it before...

I don't think NAT had an awful lot to do with that.

   > The main progress has been WebRTC

Some very good references on the topic I found recently: https://webrtchacks.com/ and https://www.html5rocks.com/en/tutorials/webrtc/infrastructur....

CORBA "et al" lost because they are fragile, not because of network issues. SOAP, for instance, is no more difficult from a connectivity perspective than REST; it rides on the same TCP/HTTP rails as REST. Yet hardly anyone, given the choice, will choose SOAP today.

Not that REST (really just HTTP APIs, mostly) is all that robust; it just asks far less of everyone involved to specify, coordinate and troubleshoot. You can hit an endpoint with your browser, pop open the browser developer tools and figure out what is going on. That _easy of use_, which CORBA, SOAP, "et al" completely forego, lowers cost, speeds development and causes an order of magnitude less nausea for developers. The market has proven the value of these benefits, despite what holdouts wish to believe.

There's the standard ways -- hole punching as described there, uPnP/NAT-PMP, etc. -- and then there is a long tail of hacks for getting to that 90%+ success rate you want. Here's an incomplete list:

* Port prediction for symmetric NATs that increment ports

* Creating multiple sockets to work around buggy NATs that can't hole punch if multiple devices have the same local port, etc. There are also buggy NATs where hole punching breaks if you also open a port with NAT-PMP or uPnP. There are a lot of buggy NATs. (Factoid: we have found zero correlation between the cost of a NAT device and its buggitude.)

* Getting the timing as tight as possible -- we use a triangular three-party rendezvous method to try to get it so both sides see a send before the "reply."

* Sending UDP packets with incrementing TTLs ahead of the real hole punch message to open multiple layers of NAT, and randomizing the few-byte payload of those low-TTL packets because some NATs care about this. Yes people plug NATs into NATs into NATs. It's pretty common. Ugh. As long as none are symmetric it works.

* If at first you don't succeed try try again... forever. For long-lived links (common with ZeroTier since they are virtual LANs) if you keep trying and include some random port trials you will eventually punch a symmetric NAT. There are only 65535 ports.

ZeroTier still has to relay though if both parties are behind hostile NATs, but we can provide free relaying for everyone since it's only a small percentage of traffic and cloud bandwidth is cheap.

Edit: there are more sophisticated methods that work more of the time, but those unfortunately can trip intrusion detection systems. We try to walk the line with ZeroTier between high success and not tripping your IDS.

Edit #2: wow there's a lot of outright nonsense on this thread. I continue to be saddened by how few developers (even top notch ones) know so little about networking and have so many misconceptions about it. Networking is still such a black art. Of course if it wasn't maybe there would not be a niche for us. :)

The client enumerates all the machines it can see on its local LAN, and for each one, assembles a list of information about it, probably something like IP address, machine name, assigned user, maybe MAC address, etc. Hash this using SHA256 or something (unlike the suggestion in the ZeroTier article about IP addresses, this approach has enough information that brute force reversal is out of the question). Assemble a list of, say, 20 of these hashes - the client's own hash is first, followed by the rest of the LAN. If you have less than 20 nodes, fill out the list with random numbers; if you have more than 20, include yourself and 19 others at random (it's important that you pick them at random, not just the first 19 in your LAN listing). Send the list to the server - this reveals nothing about your LAN to it.

The server compares the hash lists sent by clients. If any two lists have any hashes in common, send a message to each of the clients that sent those lists: "You seem to be on the same LAN as the client with hash XYZ." The client can then look up the hash in its local table, and try to contact the corresponding node. (The list intersection check is why it's important to pick your sample at random - the birthday paradox is now working for us, and there's a good chance of at least one match even if your LAN has hundreds of nodes.)