Hacker News Reader

Comment by remram | Hacker News Reader

TLDR:

> To avoid geofencing issues, we need to choose specific egress addresses tagged with an appropriate country, depending on WARP user location. (...) Instead of having one or two egress IP addresses for each server, now we require dozens, and IPv4 addresses aren't cheap.

> Instead of assigning one /32 IPv4 address for each server, we devised a method of assigning a /32 IP per data center, and then sharing it among physical servers (...) splitting an egress IP across servers by a port range.

View on HN

Ha, I guess this is one way of summarizing it :) Author here. I wanted to share more subtleties of the design, but maybe I failed.

Indeed, the starting point is sharing IP's across servers with port-ranges.

But there is more:

* awesome performance allowed by anycast.

* ability to route /32 instead of /24 per datacenter.

Generally, with this tech we can have much better IP usage density, without sacrificing reliability or performance. You can call it "global anycast-based stateless NAT" but that often implies some magic router configuration, which we don't have.

Here's one example of problems we run into - the lack of connectx() syscall on Linux - makes it hard to actually select port range to originate connections from:

https://blog.cloudflare.com/how-to-stop-running-out-of-ephem...

I was surprised IPv6 was only briefly mentioned! Is that something you're looking at next, or are you already running an IPv6 egress network?

Of course not every destination is an IPv6 host, so IPv4 remains necessary, but at least IPv6 can avoid the need for port slicing, since you can encode the same bucketing information in the IP address itself.

I've seen this idea used as a cool trick [0] to implement a SOCKS proxy that randomizes outbound IPv6 address to be within a publicly routed prefix for the host (commonly a /64).

I guess as long as you need to support IPv4, then port slicing is a requirement and IPv6 won't confer much benefit. (Maybe it could help alleviate port exhaustion if IPv6 addresses can use dynamic ports from any slice?)

Either way, thanks for the blog post, I enjoyed it!

[0] https://github.com/blacklanternsecurity/TREVORproxy

I was also interested to know how this was handled for IPv6, but it was only briefly mentioned.

Probably they didn't need to do much work with IPv6, since half of the post is solving IPv4 exhaustion problems.

Cloudflare wants to make money. The IPv6 features can come second as v6 usage increases.

All of Cloudflare services ship with IPv6 day 1. IPv6 not an issue as we have enough IPv6 for each machine to have own IPs.

Is the geofencing country level only? So if, using warp, I use trip advisor and go and see nearby restaurants it will have no idea of what city I’m in? Guessing that’s not so but wondering how it works

This blog post has some info: https://blog.cloudflare.com/geoexit-improving-warp-user-expe...

Warp uses its own set of egress IPs and their geolocation is close to your real location.

From your article it seemed that your use of anycast was more accident than feature, due to the limit of BGP prefix sizes. If you could route those IPs to their correct destination, you would, you only go to the closest data center and route again because you have no choice.

Maybe this ends up reducing cost on customers though, because the international transit happens in your backbone network rather than on the internet (customer-side).

In english they now do carrier grade nat.

well, vanilla NAT really.

right? "We don't like distributed NAT" couple paragraphs later - "hey check out this awesome dist NAT implementation".

Reply on news.ycombinator.com