How NAT Traversal Works (2020)
tailscale.com
tailscale.com
The tribal knowledge seems to be that you shouldn't do TCP-based hole punching because it's harder than UDP. The author acknowledges this:
> You can do NAT traversal with TCP, but it adds another layer of complexity to an already quite complex problem, and may even require kernel customizations depending on how deep you want to go.
However, I only see marginally added complexity (given the already complex UDP flows). IMO this complexity doesn't justify discarding TCP hole punching altogether. In the article you could replace raw UDP packets to initiate a connection with TCP SYN packets plus support for "simultaneous open" [0].
This is especially true if networks block UDP traffic which is also acknowledged:
> For example, we’ve observed that the UC Berkeley guest Wi-Fi blocks all outbound UDP except for DNS traffic.
My point is that many articles gloss over TCP hole punching with the excuse of being harder than UDP while I would argue that it's almost equally feasible with marginal added complexity.
[0] https://ttcplinux.sourceforge.net/documents/one/tcpstate/tcp...
Hence the added complexity of doing a simultaneous open via TCP is fairly minor. The main complication is communicating the public mapping, and coordinating the "simultaneous" punch/open. However that is generally needed for UDP anyway...
One possible added complexity with TCP is one has to perform real connect() calls, rather than fake up the TCP SYN packet. That is becase some firewalls pay attention to the sequence numbers.
Also, wouldn't it be easier for stateful firewalls to block simultaneous TCP open (intentionally or not)? With UDP, the sender's firewall must create a connection as soon as it sends off the first packet, even if that packet bounces off the other firewall: the timing doesn't have to be particularly tight. But with TCP, the firewall might plausibly wait until the handshake is complete before allowing incoming packets, and it might only allow the 3-way SYN/SYN-ACK/ACK instead of the simultaneous SYN/SYN/ACK/ACK.
That's true. The actual problem are symmetric NATs where every peer sees a different port number. This makes traditional NAT-traversal impossible and you have to resort to port guessing/scanning. See for example https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d...
You're probably right that it makes less sense from today's perspective
If you are able to do that whatever security you got from NAT has been breached even before NAT hole punching enters the conversation.
NAT will block unsolicited incoming connections, that is a great boon for security but obviously not a silver bullet for all network related security issues nor outgoing connections. That has never been a trope.
Why doesn't it make sense to you? From my perspective the idea was that the NAT protects your devices - and your device is now punching a hole into this protection, making it vulnerable to the world wide web
This circumventing doesn't have to be done by a malicious actor, it just comes at the added risk of becoming "targetable" from the Internet
It has no bearing on security.
Please explain why the ability to establish an outgoing connection through NAT, when you already have a sidechannel, is a security issue?
If a NAT made a device unreachable from the Internet, it'd be more secure then a device that's reachable.
You could theoretically leave management ports unsecured etc, because there is no danger - they can't be reached after all.
It just turned out that techniques such as NAT hole punching were developed, which made this rationale invalid. Because your devices could still become reachable.
Yes, this specific technique requires a an active part on both ends, but this active part can be something completely innocent, such as an activeX applet in Internet explorer 6.
Or a Freeware they installed, and now opens up the whole network, not just the device is installed on.
This is all from the very beginnings of the Internet, things weren't as explored back then, and that's why NATs were considered a security feature back then.
Modern malware is obviously much more advanced so we intuitively know why it was never really providing any safety. Hackers were mostly doing things for bragging rights, cryptolockers weren't a thing, the current default malware setup of having command hosts that compromised hosts check for new commands to execute weren't even thought of. In these times, NAT looked like a security feature, because it kinda looks like it.
TCP hole punching is very fun. The way I do it is to use multiple NTP readings to compute a "clock skew" -- how far off the system clock is from NTP. Then the initiator sets a future meeting time that is relative to NTP. It honestly gets quite accurate. It even works for TCP hole punching between sockets on the same interface which is crazy if you think about it.
The reason I wanted to support this strange, local-based punching mode is if it works that efficiently to be able to succeed in host-based punching then likely it will be fast enough to work on the LAN and Internet, too. My code is Python and my very first attempt at this was eye opening to say the least. Due to how timing-sensitive TCP hole punching is I was having failures from using Python with old-school self-managed sockets. I was using threading and a poormans event loop (based on my C socket experience)... which is ah... just not the way to do it in Python.
The only way I could get that code to work was to ensure the Python process had a high priority so other processes on the system didn't deprioritize it and introduce lag between the punching attempts. That is how time-critical the code is (with an inefficient implementation.) My current implementation now uses a process pool that each has its own event loop to manage punching. I create a list of tasks that are distributed over time. Each task simply opens a connection that is reused from the same socket. I determined this code was the best approach (in Python anyway) after testing it on every major OS.
You are right about TCP and UDP hole punching difficulty being similar. The main difficulty to both is the NAT prediction step. I haven't written code yet for symmetric NAT bypass but I am starting to see how I'd integrate it (or possibly write a new plugin for it.)
This was sometimes an issue for underpowered home/SOHO routers in the mid-2000s, but most modern routers have enough memory to support decently sized connection-tracking tables.
In any case, both TCP and UDP require connection tracking; there's no inherent advantage to UDP.
Punching through NAT, and most associated state tracking filters, is very easy.
I've implemented such in a production corp environment, as a product to be sold. There is no magic here, it is all well understood technology by the practitioners.
If you actually want to have packet filtering (a firewall) then deploy a firewall instance distinct from any NAT, and with appropriate rules. However that only really helps for traffic volume reduction, the actual security gain from a f/w per se is now minimal, as most attacks are over the top: HTTP/HTTPS, POP/IMAP etc.
You can say that in general, network firewalls are not a security mechanism. They are at most a means to prevent brute-force attacks from outside of the network.
In reality, of course, the stateful firewall is doing all of the heavy lifting that NAT is getting the credit for. Tailscale does not get rid of the firewall in fact it has a much more comprehensive setup based on proper ACLs.
Though I’m definitely the first to admit that their tooling around ACL’s could be significantly improved
That’s simply wrong. NAT is, and always has been for the sole purpose of Network Address Translation, I.e. allowing a large IP address space to hide behind a much smaller IP address space (usually a single IP address), for the purpose of mitigating IP address exhaustion.
NATs were meant to be a stop gap solution between IPv4 running out, and the rollout of IPv6. But we all know how that panned out.
The “firewall” like aspects of a NAT are purely incidental. The only reason why a NAT “blocks” unsolicited inbound traffic is because it literally has no idea where to send that traffic, and /dev/null is the only sensible place to direct what’s effectively noise from the NATs perspective.
The fact that NATs shares many of basic building blocks as a very simple stateful firewall is just a consequence of both NATs and firewalls being nothing more than stateful packet routing devices. The same way any standard network switch is (they internally keep a mapping of IP to MAC address of connected devices based of ARP packets, which incidentally blocks certain types of address spoofing, but nobody calls a network switch a firewall).
Apple advertises their iPads as “computer replacements”, that doesn’t mean the iPad was originally designed to be a computer replacement, and it certainly doesn’t make iPads a good computer replacement for many people.
I would also highlight that Cisco PIX had a dedicated firewall layer in addition to its NAT layer, which provided much more capabilities than the NAT layer alone. The fact that these two layers intelligently built on each other is just good implementation engineering, it doesn’t change the fundamental fact that NAT isn’t, and never has been, a proper security tool.
People confuse the fact that NAT is not an especially powerful or coherent security feature with the idea that it isn't a security feature, which leads you to the crazy rhetorical position of having to argue that PIX, the first mainstream NAT product, was not a security product. I have friends who worked on PIX, for many years. I assure you: they were in the Security BU.
I think this position is pretty hopeless, though if you want to drag us around through the network security marketing of the mid-1990s, I'm happy to do so, just for nostalgia's sake. NAT is absolutely a security feature, and was originally deployed as one, in an era where it was still feasible to get routable allocations for individual workstations.
I'm sure you also know, that any stateful firewall can achieve the same result without having to provide NAT capabilities. Sure Cisco PIX may have been a security appliance, but that doesn't make NAT's a firewall. You don't need Network Address Translation to create a firewall that allows devices to connect to the internet, but makes those machines unrouteable to unsolicited requests. For your claim that NATs are meant to be a firewall, you need to provide an explanation as to why we don't use NATs with IPv6.
Why would increasing the IP address space so that it's once again possible to get routable allocation for indivual workstations, result in people not deploying IPv6 NATs, when apparently they're an important security tool for IPv4, in even in the days when "it was still feasible to get routable allocations for individual workstations"?
Nobody's reading this thread anymore, so why don't we leave our arguments where they stand.
I thought standard network switches kept a mapping of MAC address to physical network ports, and didn't concern themselves with the IP layer at all (other than things like IGMP/MLD snooping)? Mapping from IP to MAC addresses is a function of hosts/gateways, not switches.
For example: https://www.arubanetworks.com/techdocs/AOS-S/16.10/ASG/YAYB/...
One need only disable stateful firewalling and use that to see how completely dire the situation would be. As all outbound connections open up your host to the internet.
Networking has long been the toxic wasteland of security and misconfiguration. Now combine that with newer host-based networking models for containers. The Windows network stack is substantially different now due to that, and more complex. Since Wireguard has been part of Linux, everyone and their brother now has a VPN, somewhere connecting to a VPS. It's probably worse than you think because you don't know what you don't know.
Firewalling is a different concept, but since you raise that issue of connectivity wrt. security, I have to say that what makes /me/ sad and anxious is to see how internet security has always been hinging on bloquing paquets based on the destination port.
Doing what's easy rather than what's correct, exemplified and labelled "professional solutions"...
I also frequently stumble upon "connexion" and "trafic", in the same department.
It still needs something on the inside to talk to outside first, so the actual firewall should whitelist both outbound and inbound connections.
Than again, if you rely on perimeter, it’s a matter of time when someone figures out what’s your equivalent of high wiz jacket is.
At the end of the day, Tailscale works just like any other VPN, from the perspective of the type of data that can traverse between machines connected to the same virtual network. Tailscales use of a P2P wireguard mesh is just an implementation detail, it’s no more or less risky that having every machine connect to a central VPN gateway, and bouncing all their traffic off that. Either way, all the machines get access to a virtual network shared by every other machine, and misconfigured ACLs could result in stuff getting exposed between those machines, which shouldn’t be exposed.
If anything the Tailscale mesh model is much more secure, because it forces every device to use a true zero trust model. Rather than the outdated, “oh it managed to connect to the VPN, it must be safe then” model that traditional VPNs often end up implementing.
It's also worth considering that exploitability of ACL code is just one factor in comparing the risk and Tailscale or similar solutions allow security conscious setups that are not possible (or at least much more difficult) otherwise. For example, the NAT and firewall traversal means you don't have to open any ports anywhere to offer a service within your Tailscale network. Done correctly, this means very little attack surface for a bad actor to gain access to that stray VM in the first place. You can also implement fairly complex ACL behavior that's effectively done on each endpoint without having to trust your network infrastructure at all, behavior that stays the same even if your laptop or other devices roam from network to network.
Not to say I believe Tailsclae is bulletproof or anything, but it does offer some interesting tradeoffs and it's not immediately obvious to me the risk is worse than legacy networks (arguably better), and you gain a lot of interesting features and convenience.
Defense in depth.
How do you suppose they gained access to the kernel and userspace just by having a network connection to the laptop?
NAT traversal requires careful coordination between two devices to create a connection, it’s not like any random device on the internet can perform NAT traversal against a machine just because it’s running Tailscale (not to mention every modern browser has NAT traversal built in for WebRTC connections).
And if the issue doesn’t arise from using NAT traversal, then how does Tailscale expose anything more significant than what a traditional VPN will expose? After all the only difference between a P2P VPN and a traditional VPN, is that a traditional VPN bounces all your traffic off a common server, rather than attempting P2P connections.
Traditional VPN don’t usually put firewalls between machines on the network, because traditionally the whole point of a VPN is to avoid the need for firewalls to provide security between nodes on the virtual network, by assuming that only safe machines can connect to the VPN.
Am I missing something?
You are right. Tailscale nodes can send packets processed in any other node, irrespectively of ACLs. Essentially each node gets to “run code” in other nodes, which is normally dropped. I don’t know how deep the Tailscale packets go before being dropped (perhaps the coordination server distributes firewall rules).
But you have to compare with another access method, like, the hub and spoke VPN. The compromised and uncompromised nodes connect to a VPN access server. A compromised node sends packets that are processed in the VPN server, but can also connect to the uncompromised node, meaning, the latter has to process and drop the packets of the former. You have to trust the OS IP stack. To some extent, the same is true if the trusted node VPNs directly to the untrusted node. During an established connection, the networking stack of the trusted node has to block the other side.
Maybe someone familiar with the implementation of ACLs in Tailscale could chime in.
Update: The ACL rules are applied to the incoming packets over tailscale interface. The filtering is then done by tailscaled. The packet has gone past the interface and processes by tailscaled. So an unauthenticated packet indeed travels through the kernel space all the way to the userspace.
An external network firewall can only offer protection if you can somehow guarantee that every packet that hits a specific node is first routed via that firewall. Traditionally nobody has setup networks like that, because it requires routing every single packet via a single common bottleneck, causing huge latency and throughput problems.
As for packets going via the network stack, and then the go runtime. Do you honestly believe there’s set of vulnerabilities out there which would allow random external packets to be sent to a random machine, and cause an RCE by virtue of simply being process by the OS kernel, which somehow can only be exploited if you’re running Tailscale? Better still, if such a vulnerability exists, what on earth makes you think your firewall isn’t also vulnerable to same issue, given that pretty much every firewall out there is built on the Linux kernel these days.
https://tailscale.com/security-bulletins#ts-2024-005 https://tailscale.com/security-bulletins#ts-2022-004
Yes
> You would typically remove the default any to any ACL rule
This part doesn’t happen.
Defaults are rarely changed.
From the public internet: tens of thousands of internet routers have publicly known exploits right now, which the router vendors refuse to fix. Just scan the internet for the routers, use your exploit, and you're inside.
From the opposite direction: malware in a website can redirect your browser to the management interface of a router on your local LAN, where it can reconfigure your router. If there is a password but you have logged in from your browser, the active session token lets it right in, and CSRF protection is often disabled or incorrectly set up. And even if it has a password, many such routers have exploits that will work despite a password. Many people also fall for phishing attacks that can drop payloads on your machine directly.
In some cases, the ISP itself has shipped a firmware update to routers that included malware.
All of these things have happened in the past 2 years, to millions of internet users, that we know of. Many large attacks go unnoticed for years. Once the router is compromised, it can be configured to forward ports or enable UPnP, or simply persist malware inside the router itself. The network is wide open and at the attacker's fingertips.
And this is just one class of attack. There are many more that can attack private networks. So there is no place safe from network attacks. Not in a corporate network, not on your local LAN, nowhere. There is no network security. The only network services that can be somewhat trusted are ones which require strong authentication, authorization, and encryption.
Have you taken steps to validate the integrity of every single device connected to the network?
If a single device is compromised, how will detect its been compromised?
If a device is compromised, what prevents it from being used to launch an attack on other devices in your network, especially if your security model assumes that all devices on your local network are “safe”?
For a practical example of this happening, in a very impressive manner: https://arstechnica.com/security/2024/11/spies-hack-wi-fi-ne...
For a more boring everyday equivalent, just search around for one of the many botnets that are assembled from compromised SoHo routers, or IoT devices, around the world.
https://arstechnica.com/security/2024/01/chinese-malware-rem...
Assuming a local network is safe and secure is foolish. There’s nothing inherently secure about a local network, the only reason it offers any level of security is due to a local network being many-many orders of magnitude smaller than the entire internet. So the probability of a hostile device (whether intentional installed as hostile, or became hostile after a remote attack) being connected is smaller. But at the end of the day, is security via “being luckier than the next dude”.
"At a less granular level, the coordination server (key drop box) protects nodes by giving each node the public keys of only the nodes that are supposed to connect to it. Other Internet computers are unable to even request a connection, because without the right public key in the list, their encrypted packets cannot be decoded. It’s like the unauthorized machines don’t even exist. This is a very powerful protection model; it prevents virtually any kind of protocol-level attack. As a result, Tailscale is especially good at protecting legacy, non-web based services that are no longer maintained or receiving updates."
Source: https://tailscale.com/blog/how-tailscale-works#bonus-acls-an...
GRE tunnels exist and I actually use them extensively, but UDP hole punching is not handled so hub-and-spoke architecture is needed for them, no peer to peer meshes with GRE (ip fou).
Are there equivalent libraries out there which do UDP hole punching and unencrypted GRE tunnels following an encrypted handshake to confirm identity?
libp2p [4] may be what you're after if you want something geared more towards general purpose connectivity.
[1] https://datatracker.ietf.org/doc/html/rfc8445
[2] https://github.com/pion/webrtc
FWIW, libp2p also enforces transport encryption, quote:
> Encryption is an important part of communicating on the libp2p network. Every connection must be encrypted to help ensure security for everyone. As such, Connection Encryption (Crypto) is a required component of libp2p.
It's written in Python. Though its not based on using the default interface like most networking code. I wanted the possibility to be able to run services across whatever interfaces you like. Allowing for much more diverse and useful things to be built. Its mostly based on standard library modules. I hate C extension crap as it always breaks packages cross-platform.
This is where I wish SIP lived up to its name (Session Initiation Protocol, i.e. any session, such as a VPN one...) and wasn't such a complicated mess making it not worth the hassle. I mean it was made to be the communication side-channel used for establishing p2p rtp streams.
Its like http, but its also statefull, bidirectional, federated and works over udp too.
Just looking at the amount of stuff (tls over udp included) baresip implements to barely sip. And it isnt even bloated, the stuff has to be there.
and for the same reason: both were initialy designed to be simple...
Previous discussion:
(2022) https://news.ycombinator.com/item?id=30707711
(2020) https://news.ycombinator.com/item?id=24241105How NAT traversal works (2020) - https://news.ycombinator.com/item?id=36969018 - Aug 2023 (106 comments)
How NAT traversal works (2020) - https://news.ycombinator.com/item?id=30707711 - March 2022 (37 comments)
How NAT Traversal Works - https://news.ycombinator.com/item?id=24241105 - Aug 2020 (28 comments)
(p.s. your links weren't clickable because lines that are indented with 2 or more spaces get formatted as code - see https://news.ycombinator.com/formatdoc)
This may be the only way we ever have to build p2p apps. IPv6 doesn't have enough steam since NAT and SNI routing solve most problems for most people.
And ISPs are very much not incentivized for that to change.
It's a very good theoretical article. I wonder to what extent a software engineer could use this though. Because although it does describe many things I'm not sure there's enough detail to write algorithms for it. Like, could an engineer wrote an algorithm to test for different types of NATs on the basis of this article? Could they adapt their own hole punching code? I've personally read papers where simple tables were more useful than entire articles like this (as extensive as it is.) Maybe still a good starting point though.
Also, the last section in the article is extremely relevant. It has the potential to bypass symmetric NATs which are used in the mobile system. The latest research on NAT traversal uses similar techniques and claims near 100% success rates.
A read a bit about this space a few weeks ago after not knowing anything about it beforehand. My impression is that ip6 dices all of this and NAT traversal isn't necessary anymore. So why isn't ip6 more popular and how do I get started with it for my home network and tailscale VPN?
Not sure how much of a factor but human usability has always been challenging.
There is also the lack of business incentives.