NAT Is the Enemy of Low Power Devices
blog.golioth.io
blog.golioth.io
> Therefore, when you try to continue talking to the server over a previously established session, it will not recognize you. This means you’ll have to re-establish the session, which typically involves expensive cryptographic operations and sending a handful of messages back and forth before actually delivering the data you were interested in sending originally.
The blog post mentions Session IDs as a solution, but these require servers to be stateful, which can be challenging in most deployments. An alternative is Session Tickets (https://datatracker.ietf.org/doc/html/rfc5077), but these may cause issues when offloading networking to another device, such as a cellular modem, as their implementation may be non-standard or faulty.
incoming rant
These issues could be mitigated—or even solved—by using mature software platforms like Zephyr RTOS and their native networking stacks, which receive more frequent updates than traditional “all-in-one-package” SoCs. However, many corporations choose to save a few dollars on hardware at the expense of incurring thousands in software engineering costs and bug-hunting. It is often seen as more cost-effective to purchase a cellular modem with an internal MCU rather than a separate cellular modem and a host MCU to run the networking stack. It is one of the many reasons why many IoT devices are utter garbage.
This one isn't just cost--the compliance restrictions that the cellular carriers place on you are idiotic.
The big one we bumped into is "must allow allow carrier initiated firmware updates with no restrictions on scheduling" which translates to "the carrier will eat your battery often and without warning".
In addition, many IoT devices may not call home more than once every couple of months. And the carrier will happily roll out tower firmware that will kill those being able to call home.
If I use a module with my own firmware, the modem folks will simply point fingers at me. If I use a module with integrated SoC and firmware and it gets updeathed, I get the "joy" of yelling at the cellular module manufacturer.
(I had the wonderful experience of watching a cellular IoT project go gradually dead over 3 days as the carrier rolled out an "upgrade" across its system. We got a front seat as the module manufacturer was screaming bloody murder at the carrier who simply did "We Don't Care. We Don't Have To. We're the Phone Company.")
Normally US carriers require Firmware-Over-The-Air (FOTA) capability for the modem firmware. This is not the case for deployments outside of the United States, to my knowledge.
It would be interesting to hear more about your story!
Good luck litigating.
I am evaluating some Nordic semiconductor parts for a project. They seem to have an internal modem but Nordic uses zephyr. Any thoughts?
The US 3G shutdown required some rather expensive and unexpected upgrades. Vendors signed long-term contracts with 3G providers, and then "someone" was on the hook, to replace something, when the 3G vendors terminated their contracts prematurely.
The deeper the modem was integrated into a product, the harder it was to change. The shallower the modem was integrated, the easier it was to change.
For example:
One of my cars just lost its internet connectivity, and the automaker never offered any way to fix it. (I didn't care, I only used Android Auto in that car.)
My employer (IOT) sent out free chips to our customers. They had to arrange someone to go do a site visit and swap a chip while on a phone call with us. We're small and early enough that it wasn't a big deal.
My solar panel vendor wanted me to $pend big buck$ on a new smart meter and refused to honor their warranty. I told them to run a cable to the ethernet port in my meter.
The only feature I need to control remotely in my car is preheating during winter—I wonder how they could achieve that without using cellular connectivity as paying a subscription for such a service would make it less attractive to me.
Part of the reason is there's no delay waiting for my phone to connect, and no random disconnections while driving.
Personally, I’m not a fan of Tesla’s car computer. My main annoyance is the lack of physical buttons. All I want from my car is reliable navigation, seamless audio playback, and easy mobile call handling.
That's pretty much what the Tesla computer has. I do agree that the lack of physical buttons is a problem; and now that there are more EV makers on the marketplace the market can speak very loudly and choose other automakers.
When using a separate cellular modem, you can connect it to your MCU via either a USB or UART interface. In IoT applications, UART is the more common choice. Then you can multiplex the interface with CMUX, allowing simultaneous use of AT commands and PPP.
With frameworks like lwIP or Zephyr supporting PPP, you can get your network running really quickly and have full control over the networking and crypto stacks. Using Zephyr you get POSIX-compliant sockets which allows you to leverage existing protocol implementations.
In contrast, using a SoC's often requires reintegrating the entire network stack, as they typically do not support POSIX sockets. I've worked on SoC's that only support TLS 1.1 and the vendor refused to upgrade it, as it would require them to re-certify their modem. Switching to a different SoC can mean repeating this process from scratch as different vendors implement their own solutions (sometimes even the same vendor will have different implementation(-s) for different modems).
> I am evaluating some Nordic semiconductor parts for a project. They seem to have an internal modem but Nordic uses zephyr. Any thoughts?
It runs on Zephyr RTOS and can be built as a standalone modem (https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/appli...). You can either use Zephyr's native networking stack or offload it to the modem while retaining the same API. This means you get access to all of Zephyr’s protocol implementations without additional effort. The design makes it feel as though you have a completely independent MCU connected to an external modem.
That said, it does have quirks and bugs, particularly when offloading networking. It also has relatively limited resources, with only 256 kB of RAM and 1 MB of flash storage.
Overall, it is the best SoC I’ve worked with, but it is still an SoC. Whether it suits your project depends on your specific use case. If you plan to integrate additional radios in the future (e.g., UWB, BLE, Wi-Fi), I’d recommend using a separate MCU if your budget allows. This will provide significantly more flexibility. Otherwise it is definitely one of the better SoC's currently in the market, to my knowledge.
PS. It only supports Cat-M (& NB-IoT but I'm going to skip over it intentionally!) which is not globally supported, so you should make sure the region you want to deploy in supports that technology.
On one hand, licensing requirements and regulation often mean that modems are locked down in terms of firmware updates, reference documentation, source, and capabilities. This often translates into a larger "black box" area, and one embedded inside your SoC instead of physically separate and connected over a serial bus.
On the other, on-chip modems often (not sure about those Nordics) have DMA.
The combination of those two is scary.
Could you clarify what you mean by this?
If the case of my car I don't care - I have never found a use for the cellular connectivity it has (if any). However there are lots of other devices where the cellular connectivity is important and users will want to upgrade the modem to keep it working. If cellular connectivity is just a marketing bullet point nobody cares about then integrated is good enough, but if your device really isn't useful without the modem make that modem replaceable for somewhat cheap.
Doesn't this just move the 'state' into the operating system, or networking layer, in the form of an active TCP connection?
Nevertheless, with modern cloud providers moving the state into the OS/Networking layer is still easier to scale. You don't need to write your own services to handle it.
NAT gets the blame, and the intranet as a concept is generally a big corp term.
But I prefer my IoT devices not to need to reach out of my network. For me, NAT is an unwitting ally in the fight against such nonsense.
My university has a public IP for every computer, but you could still only connect to the servers, not random computers, from the outside. Because they had a firewall.
NAT/port forwarding, for all their faults make it rather difficult to write rules allowing traffic to a machine you didn't intend to expose to the world.
This is not in any way a benefit of NAT.
Just because NAT accidentally achieves some good outcomes doesn't in any way imply that said good outcomes are somehow exclusive to NAT.
Obviously, those average people have a suitable firewall provided by default on their routers.
Tailscale doesn't strictly need NAT traversal. They can run only their DERP servers and still continue to work. If your firewall tries to block two devices from communicating and yet allows both devices internet access, you have already lost.
They have all the upsides of NATs (i.e. an option to block inbound connections by default), with none of the downsides (they preserve port numbers, can be implemented statelessly, they greatly simplify cooperative firewall traversal, you can allow inbound connections for some hosts).
I could have all of my servers in public subnets and give them all public IP addresses, but I still prefer to put everything I can in private. Not only does the firewall not allow traffic in, but you can't even route to them. It now becomes really hard to accidentally grant more access than you intended.
I would hazard that most devices on there internet are in the boat of want to talk to the internet but not be reachable on it.
There is a distinction between being publicly addressable and publicly routable. You can have the former without having the latter.
If you want more private addresses, IPv6 has a solution too: use ULAs and not GUAs. Design your internal network so it has mostly ULAs for application servers, database servers and the like, except for the reverse proxy having both publicly accessible GUAs as well as ULAs for talking to the rest of the network.
I personally use ULAs and GUAs concurrently on my network, because I have a residential ISP where my GUA prefix is not fixed.
If it's a well-reasoned decision, sure, but I do suspect that more often than not it's a lack of knowledge about alternatives that makes people still opt for NATs, and that just makes me sad on top of being annoyed with the inconvenience of having to tunnel when a direct connection seems so close at hand.
> I would hazard that most devices on there internet are in the boat of want to talk to the internet but not be reachable on it.
I highly doubt that. One big example is VoIP: Incredibly common these days, yet so much of it is going through centralized relays, and often for absolutely no technical reason.
Being forced into a CGNAT on ipv6 is just a dick move though. And I believe that's the kinda NAT that has coloured the opinions of most NAT for ipv6 detractors.
But to your point about not cheering on NAT, well I will because I see NAT as useful tool.
It is not an opinion well aligned with the preferences of the IETF. But the purist model of transparent end-to-end networking has never sat well with me. It’s just not a thing we want.
Because that's what happens if you advocate for NAT by default. Conversely, with "IPv6 + inbound-blocking-firewall on CPEs by default", everybody gets the same behavior by default, and people that want something else can get that instead.
Security and privacy is of course critical to all this, but the concept of internet itself is not wrong.
Does that VPN use certificates or a pre shared key? Do they understand the different security implications between those two choices?
But theirs uses certificates (the router UI generates the openvpn client config files with the certificate embedded inside it) and no, they do not understand the security implications between those two choices.
Mine is a wireguard VPN with both the pub/priv keypair and PSK.
There's a reason IoT vendors try to do this all "in device."
The mobile platforms in particular continue to assume that you can live with their high level HTTP stuff, and it's just not good enough. The non-mobile platforms largely don't even approach the problem at all.
Seems to me that would be the lowest power, lowest developer trust, lowest number of connections, maximum gatekeeping method.
[1] Doing this of course kills any chance at server-to-device comms, you can only ever apply changes when the device next checks in. This does cause us complaints from time to time, especially for those with longer intervals.
Maintaining an active session for communication between the endpoint and the edge device is highly power-intensive. Even with (e)DRX, the average power consumption remains significantly higher than in sleep mode. Moreover, the vast majority of devices do not need to frequently ping a management server, as configuration and firmware updates are typically rare in most IoT deployments.
One way to get around this in cases where devices need to be fairly responsive to cloud to device communication (on the order of minutes) but in practice infrequently receive updates is using something like eDRX with long sleep periods alongside SMS. The cloud service will not be able to talk to the device directly after the NAT entry is evicted (typically a few minutes), but it can use SMS to notify the device that the server has new information for it. On the next eDRX check in, the SMS message will be present, then the device can ping the server, and if using Connection IDs, can pull down the new data without having to establish a new session.
In theory, you get all the power saving that the cellular network stack has to offer without having to maintain a connection. While on protocol layer NIDD is almost handled like an SMS (paging, connectionless), it is not routed through a telephony core (and hence sloooow). The base station / core will directly forward it to your predefined application server.
> This doesn’t solve the issue of cloud to device traffic being dropped after NAT timeout (check back for another post on that topic), but for many low power use cases, being able to sleep for an extended period of time is more important than being able to immediately push data to devices.
(edit: there was originally an unfortunate typo here where the paragraph read "less important" rather than "more important")
Depending on the device and the server, powering down the modem does not necessarily mean that a session has to be started from scratch when it is powered on again. In fact, this is one of the benefits of the DTLS Connection ID strategy. A cellular device, for example, could wake up the next time in a completely different location, connect to a new base station, be assigned a fresh IP address, and continue communication with the server without having to perform a full handshake.
In reality, there is a spectrum of low power options with modems. We have written about many of them, including a post [1] that followed this one and describes using extended discontinuous reception (eDRX) [2] with DTLS Connection IDs and analyzing power consumption.
[0]: https://blog.golioth.io/power-optimization-recommendations/ [1]: https://blog.golioth.io/turn-off-subsystems-remotely-to-redu... [2]: https://www.everythingrf.com/community/what-is-edrx
The example diagrams also incorrectly show port numbers exceeding 65535. The port fields in TCP and UDP headers are 16 bits [2].
[0]: https://www.rfc-editor.org/rfc/rfc5382 [1]: https://www.rfc-editor.org/rfc/rfc4787 [2]: https://textbook.cs161.org/network/transport.html
(D)TLS session resumption (I'm not sure if their "Connection IDs" are that or something similar) seems like the most foolproof solution to this scenario, assuming that the remote host can support it.
You'd be surprised... PCP (Port Control Protocol) implemented by large vendors such as Cisco and Apple are able to punch through a firewall for up to 24 hours in a single session.
https://github.com/Self-Hosting-Group/wiki/wiki/Port-Mapping...
I'm of the opinion that IPV6 changed some small things just enough to get people to have to learn new stuff -- and also forgot that NAT is not a firewall, somewhere along the way.
Instead of making IPv6 just "IPv4 with 128bit instead of 32bit address space" the designers changed a bunch of things. NDP is new and different than ARP. ICMPv6 is extended and different. MLD instead of IGMP for multicast group membership. DHCP is extended and has a bunch of other alternatives, like SLAAC. IPsec is mandatory now.
All the new IPv6 protocols seem more secure, more powerful, and more efficient. So I guess that is "why". But I bet in an alternative history where backwards compatibility and the upgrade process were prioritized IPv6 would be a lot more widespread.
That being said, most modern SoCs are competitively priced... and will boot Linux just fine for under $5/part. =3
The resource overhead is minimal for modern mcus. Dropping dhcp and arp can save a lot of resources too. Also I have mcus with more ram than my first pc.
Disabling IPv4 saves 25KiB of flash and less than a KiB of RAM. If you're down to the last kilobytes, disabling IPv6 makes more sense than disabling IPv4. Both options are choices of last resort, though.
However, I am a fan of their price point for hobby projects. Best regards =3
Contrary to popular belief, IPv6 does have NAT, it's just stupid and unnecessary in most use cases. There are even different kinds of NAT; there's the "swap the public prefix with a private one" NAT (useful for shit ISPs that rotate your IPv6 prefix to make you pay extra for a static one, so your addresses don't keep changing), or the IPv4-like "masquerade all traffic when forwarding" one that'll work in your weird enterprise network.
With the latter option, you can set up an IPv6 with DHCPv6 on the upstream and whatever combination of SLAAC or DHCP you want on the VM side, and set up NAT using guides like this (possibly outdated iptables-based) one: https://forums.raspberrypi.com/viewtopic.php?t=298878
The threat model on this big enterprise WAN isn't "an authorized user creates a VM on an existing machine", but rather "some rando plugs something in with an Ethernet cable and goes to town on the intranet services". It has a whole web portal where you can log in and register your devices. So it's not like anything's really being bypassed here.
Anyway, I still prefer NAT for VMs over most passthrough schemes, which I've found unreliable even on IPv4 networks. What the outside network doesn't know about can't hurt it.
Isn't that what the MAC filtering is for? I don't see the point of stopping a single registered MAC on a single port from having multiple IPs.
(If I had to guess, just letting any machine have a million new IPs for the firewall to track has its own issues, so you'd end up with policies upon policies.)
In any case, sometimes middleboxes just don't behave precisely how we want them to, and that's why I'm skeptical of the typical IPv6 position of "a flat /64 network (or something emulating one) is all you'll ever need".
And sure a million IPs could cause problems, but that's not a good reason to set the limit to 1.
> I'm skeptical of the typical IPv6 position of "a flat /64 network (or something emulating one) is all you'll ever need".
That's not the position. You can have as many networks as you want. Connect them with routers like you would on IPv4.
IPv6, set up the normal way, does not allocate IPs. Devices can and will have several at once and their VMs can easily get some too.
And even in the face of a supposed malicious network admin like yours, you can still adopt IPv6, and use the IPv6 private network ranges for the VMs, and then do NAT like it's 1983, same as you would have otherwise done with IPv4, except that all other uses cases get easier, while this one use case remains as hard as it was before.
[1] https://lore.kernel.org/netdev/1258465585.3682.7.camel@johan...
Possibly this is just at the DHCP level, though, and what the router sees just comes from my host MAC address? A quick packet trace suggests that.
UDP still needs state tracking, unfortunately.
If you really want IoT wifi devices, put them on a separate wifi, and only let them talk to a local device that you can keep up to date. Assume they're vulnerable to local attacks over wifi and act accordingly, e.g. don't give the IoT wifi access to your other devices beyond to that controller, and definitely not to the wider internet.
If they're closed source, assume they're already compromised from the factory
I've always wondered how it is economically feasible to run these central services without a monthly subscription. If you stop selling devices you'll go under fairly quickly.
https://www.malwarebytes.com/blog/news/2024/04/ring-agrees-t...
and somehow they are still in business, and popular.
If you do care about security, keeping your home-automation within your own control is probably the only sane path. Homeassistant and similar open source things like openhab are pretty good if a bit fiddly, and a wireguard vpn like tailscale a fairly practical way to access it when away from home.
It is also very expensive, so only a minority of informed users would select such a route. Not to mention that it is not trivial to setup.
To my knowledge the only company that has decent security practises is Ubiquiti.
What the article describes is actually fairly standard in the IoT industry. I'm surprised there have been so few lawsuits.
It's not a perfect solution for every scenario, and requires configuration, but there it is.
This is how people on residential lines run web servers, mail servers, ... they map TCP ports like 443, 80, 22 and 25 on their router to go to specific internal hosts.
Doh!
Common suggestions I've heard:
"Use a VPN"
I tried to buy a computer from Apple directly. They detected the VPN and wouldn't let me purchase it. I turned off the VPN and the purchase worked. I then got a call from Apple asking me if I really did intend to buy the computer.
"Use Tailscale"
On my setup, tailscale can't really navigate the NAT setup correctly. I have a 200Mb downlink and I will often only get a 10th of that through tailscale. Also the routing table no longer works as routing is handled by a myriad of netfilter rules -- which may or may not conflict with docker on the same box.
"Use Ubiquiti (or other networking gear) to get rid of the last NAT"
Gah. I didn't need to perform a huge cash outlay 10 years ago and rewiring the house to do the same thing I'm trying to do today.
[0]: https://blog.golioth.io/golioth-announces-connection-id-supp...
I don't understand why that stalled.
Also considering the state of iot security its probably not a great idea to have everything accessible anyway. But that's a slightly different problem to solve.
You try to continue a tcp session that’s timed out on my firewall and the packets will be dropped.
This applies a fair amount t to me when I suspend my laptop, my ssh session will drop as both the server and the firewalls drop the session while it sits there peacefully. When it comes back the tcp packets get sent into the void.
Meanwhile my WireGuard connection which runs through two separate ipv4 nats works just fine, as it doesn’t rely on sessions or a server timing out a socket.
Nat is irrelevant to the problem.
Just have relatively secure devices using ipv6 - not shipping with standard default admin passwords would nearly be enough if there aren't obvious vulnerabilities.
Is "less important" a typo?
Note that we had a solution for address exhaustion by 1991, but it was just "good" and not "perfect" and worst of all it used the hated OSI protocol stack (TUBA - TCP & UDP on top of OSI CLNS - also known as IPv9). It even had at least two usable implementations at the time it was proposed (for SunOS and Cisco IOS)
That was inevitable and can't reasonably be blamed on NAT. As but a few examples. ISPs arbitrarily break things unless you pay them extra. Stateful firewalls are a good thing. Even on my local LAN I can't reliably SSH into my laptop because it's on WiFi and does funky power saving stuff with the chipset I guess. My phone is far worse than my laptop in that regard.
Setting up a reliable and widely reachable server requires deliberate effort regardless of the existence of NAT.
The others kinds of problems are from there over time.
Does your router not already do that by default?
It sounds like you want an ISP-provided stateful firewall though, upstream of your (metered, slow) connection, which I'd agree would be a great feature to have!
Are you thinking about CGNAT which is done by the ISP? That results in double NAT which causes problems.
From outside there's 1 IP address. With IPv6, every device would get it's own address outside. Why do I want that? That sounds less private to me. Am I mis-understanding something? Lots of traffic on one IP address sounds more obfuscated than all separate.
You could disable the rotating addresses, or disable MAC-based ones by using DHCP, but there's usually no point.
As for why you would want something like that: a whole bunch of software and hardware breaks because of NAT. Consumer NAT has some monkey patching inside of it rewriting some protocols to make them work again (which also allowed random websites to open arbitrary ports to arbitrary addresses in some Linux routers a while back, because NAT overrules firewall settings to work) but there are still limitations.
For instance, if you're having issues with your Nintendo Switch, Nintendo will tell you to forward every single port to your Switch (https://en-americas-support.nintendo.com/app/answers/detail/..., hope that IP address doesn't get reassigned to an unpatched device later). Multiple Xbox consoles behind the same NAT requires tricking them into super-restricted-NAT mode to work, or enabling UPnP which allows devices to open ports in your firewall without any authentication.
NAT just kind of sucks. IPv6 wasn't ready for deployment when NAT gained popularity, but all of the reasonable problems have been solved over a decade ago.
*=default rotation happens daily, but your OS may allow you to pick a shorter duration. I've found out the hard way that setting this to five minutes will fill up Linux' route table real fast after a few days.
Then again, the "typical" multi-user household is likely logged in to most things via SSO with Google or Facebook and probably has approximately zero fingerprinting mitigations in use so perhaps it isn't worth worrying about?
If you aren't the typical household then given 2^64 addresses and a Linux box serving as a router you've got quite a few options available. Including various creative reinventions of NAT that don't break basic functionality.
Why? With privacy extensions (which are normally enabled for user devices), then all someone can do is look at the prefix. This is identical to looking at the IPv4 address in a NAT setup, and it hasn't been that much of a privacy disaster.
As I see it, nothing is lost on that front.
It is not identical unless the OS uses a new IP for every new outbound connection. I believe that would qualify as a (very) nonstandard configuration.
> it hasn't been that much of a privacy disaster.
Indeed, it was tongue in cheek which is why I went on to point out SSO. The reality is most people aren't willing to sacrifice convenience to retain even a shred of privacy.
If you are one of the few who care then you can implement one of the many possible non-standard solutions.
If my PC uses the same IP as my partner's to talk to Google, it hardly matters for our privacy if they mix up the attribution of traffic between the two of us.
Granted any service that I consistently interact with is likely to be able to figure out at least some of that information if they put in some effort. But I don't want to be freely providing a complete picture for zero effort.
Creepy data aggregator stories pop up on the HN front page regularly so hopefully I don't need to explain why I feel this way.
Using the same IP for all of my devices, for me, generally falls into the same bucket of anti-fingerprinting techniques that are used by the Tor Browser like letterboxed resolution that I don't find practical for general use. If I want to actually prevent fingerprinting by IP, resolution, etc. then I'll actually use the Tor Browser.
Having recently enabled IPv6 for my home network, the "why" was that a) IPv6 to IPv6 connections are nominally more efficient than those that have to traverse NAT and b) it enables connectivity to/from IPv6-only internet devices.
The privacy upsides of a single IPv4 IP for a household are, to me, more marginal than the above benefits.
This is the key problem. I think it would be best solved by NAT devices having some way of probing their timeout policy, and then notifying the endpoints when they expire a mapping.
One way to do that might be for a client to deliberately send a packet to the server with an insufficient TTL.
The reply (via ICMP) could then contain fields specifying the timeout in minutes of the mapping, together with some generation number specifying if the mapping table has been cleared due to reboot or overflowed since last queried.
There might even be a way to request a specific mapping become long lived - perhaps for months or years.
The benefit of all this is that client devices can do push notifications from servers or P2P notifications, all with no polling - allowing for example coin cell devices to last for months or years (with appropriate WiFi protocol improvements too)
If you don't like NAT you could go IPv6.
But really, why do you need to talk to your device? If it's just reporting in NAT is irrelevant. If you want to do device management just write something into your protocol to check for updates/commands and deal with it on a periodic basis. You can even do that on startup, so you can tell the customer to power cycle the device. It's unlikely that any IoT device needs instant updates, so long periodic updates are probably fine.
You can avoid polling for messages, but you have to send packets outbound regularly in order to maintain a NAT mapping & connection, so that the external side can send messages inward.
The latency is overcome this way, so latency is a solvable problem, but this need to constantly wake up a radio every <30s in order to keep a NAT session alive is a significant power draw.
In theory you might be able to avoid this with NAT-PMP / UPnP however their deployments are inconsistent and their server side implementations are extremely buggy.