Hacker News Reader

Comment by subarctic | Hacker News Reader

Pretty interesting article. TLDR: they're now using anycast for egress, not just ingress.

Each data center has a single IP for each country code (so that they can make outgoing requests that are geolocated in any country). In order to achieve that, they have a /24 or larger range for each country, and announce it from all their data centers, and then they route the traffic over their backbone to the appropriate data center for that IP.

Then in the data center, they share the single IP across all their servers by giving each server a range of TCP/UDP port space (instead of doing stateful NAT).

View on HN

It’s not a single IP address per data center. Otherwise they’d only be able to make 64k simultaneous egress connections, nor would their scheme of different ip addresses per “geo” and product work.

Edit: based on this part of blog post:

“With a port slice of say 2,048 ports, we can share one IP among 31 servers. However, there is always a possibility of running out of ports. To address this, we've worked hard to be able to reuse the egress ports efficiently.”

I guess for large data centers they could use multiple IPs and the overall scheme stays essentially the same, but the article seems to strongly imply that it is one single egress IP per DC. The 64k simultaneous connections limitation is only per origin server IP.

Do you mean 64k simultaneous egress connections per origin server IP? That's still a reasonable number of connections.

I might not be understanding, but it needs to know how to route the packets back to the server and it does that with ports.

I don't know if their stack allows for it, but the number of concurrent connections is limited by unique 5-tuples (which is what the host uses to identify the sockets), not just source ports. There is presumably quite a lot of entropy available in destination IP & port to enable far more than 2048 connections per server. But it's hard to be deterministic about how much, so they might be choosing to ignore this and use the lower bound.

> TLDR: they're now using anycast for egress, not just ingress.

I don't see this behavior in my setup. I have a server in AMS and connect from IST, but `netstat` reveals a unicast IP address that is routed back to IST.

Reply on news.ycombinator.com