Comment by repolfx | Hacker News Reader

Comment by repolfx | Hacker News Reader

Parent Full thread

They are IP based.

There are several problems that make replicating FCM hard, which is one reason why Google tries to push you to using it (wordplay definitely intended).

It all boils down to IPv4. IPv4 address space is exhausted, so carriers have had to deploy carrier NAT in front of cell towers, often multiple levels of carrier NAT. Well NAT requires translation tables to be held in RAM and carrier-grade boxes are very expensive in general, so to keep the machines alive they need to aggressively garbage collect dead translations. Otherwise they'd run out of RAM.

This is the origin of the 'keep alive' problem - NATs want to close your connection to free up their own resources, and you want the connection to stay open so you can receive push messages. So phones have to wake up every so often and send keepalive packets or do a connection rebuild.

Google and Apple have an interesting solution to this problem .... it's a mix of learning and data analytics to figure out what NAT timeouts used by each carrier are, and cutting deals with carriers directly to adjust the timeouts for their IP ranges specifically. Therefore you cannot compete directly with Google or Apple on energy efficiency. This is something a lot of hackers don't realise. Getting to the level of efficiency FCM has is very hard and takes a lot of work, and basically requires you to be a giant company. This is why it's an OS level service.

They also use very tight protocols, batch things together and of course provide oodles of server side disk space for buffering messages to disk until the devices return, lots of other MQ type things that are hard to do at scale.

IPv6 solves this problem by allowing carriers to lose the NAT boxes. No more machines that need stuff in RAM for every TCP connection, which means connections are no longer a scarce resource that must be culled from time to time. If you only have to maintain connections to devices that support IPv6 you could theoretically maintain very long lived connections if the kernel, radio and server cooperate. Of course there are still timeouts: the TCP connection requires some state server-side, so the servers will kill off connections from time to time because devices may roam across different IP addresses. But it should be a lot more stable and not require cutting deals anymore.

> and cutting deals with carriers directly to adjust the timeouts for their IP ranges specifically

In theory, seems like they could just sidestep the NAT entirely. Co-locate a front-end server inside the carrier's private network.

Then the phone and front-end server talk to each other with private addresses. Once it hits the front-end server, everything can be multiplexed over a single TCP connection back to the Google data center. (You'd need a way to discover the server, but that's doable. For example, the regular server can refer you based on cell carrier, source IP, etc.)

Google already has reasons to want servers co-located, such as serving static web content and making sure new TCP connections warm up fast. Apple too, somewhat. So the hardware might be there already and this might be just a configuration change, possibly including the addition of a private address.

Point being, the economies of scale here might be even greater. They might have a nearly free way to do it, and without the need to convince the carrier to give up precious NAT resources.

Thank you for clear explanation.

Haven't most big land and mobile carriers deployed IPv6 on their core network yet for other benefits? If so, one could create alternative cloud notification service based on IPv6.

It's interesting to look how Exchange ActiveSync handles this. They do a hanging request to the server, and the client keeps track of when the connection closes, and negotiates with the server to push the keepalive ping time up and up, until the device hits the carrier TCP timeout. The client then knows how long it can stay asleep before having to wake up to "PING" to keep the connection open.

I doubt the accuracy of what you are saying. Treating certain IP ranges differently would not be net neutral, which is exactly why we have net neutrality: otherwise it's impossible for new entrants in the market to compete. Also, most ISPs don't run CGNAT (except historically notoriously many in Italy, and by now also a few in the USA). And "oodles" sounds like a lot but running a push server doesn't sound like it would require that much disk space. At Google's level, sure, but for your average app... it can easily be run without Google.

The real problem (afaik) is that people change address whenever they change network (so it doesn't all boil down to v4, either), so you need regular keepalives. If every app starts doing keepalives individually, your phone will be using its radio every 30 seconds 24/7 (assuming you have four apps that need push, and those devs want a maximum delay of 2 minutes, and their timers are roughly evenly distributed). By having a single server to keepalive, the issue is avoided.

> Also, most ISPs don't run CGNAT (except historically notoriously many in Italy, and by now also a few in the USA).

My understanding is that essentially every mobile network in the world uses CGNAT. I happen to have AT&T, T-Mobile, and Verizon SIMs handy at the moment, and all 3 of them are behind NAT.

Great answer, thanks for taking the time

Reply on news.ycombinator.com