Comment by xg15 | Hacker News Reader

Comment by xg15 | Hacker News Reader

> However, while anycast works well in the ingress direction, it can't operate on egress. Establishing an outgoing connection from an anycast IP won't work. Consider the response packet. It's likely to be routed back to a wrong place - a data center geographically closest to the sender, not necessarily the source data center!

Slightly OT question, but why wouldn't this be a problem with ingress, too?

E.g. suppose I want to send a request to https://1.2.3.4. What I don't know is that 1.2.3.4 is an anycast address.

So my client sends a SYN packet to 1.2.3.4:443 to open the connection. The packet is routed to data center #1. The data center duly replies with a SYN/ACK packet, which my client answers with an ACK packet.

However, due to some bad luck, the ACK packet is routed to data center #2 which is also a destination for the anycast address.

Of course, data center #2 doesn't know anything about my connection, so it just drops the ACK or replies with a RST. In the best case, I can eventually resend my ACK and reach the right data center (with multi-second delay), in the worst case, the connection setup will fail.

Why does this not happen on ingress, but is a problem for egress?

Even if the handshake uses SYN cookies and got through on data center #2, what would keep subsequent packets that I send on that connection from being routed to random data centers that don't know anything about the connection?

Yep, it can happen that your packet gets routed to a different DC from a prior packet. But the routers in between the client and the anycast destination will do the same thing if the environment is the same. So to get routed to a new location, you would usually need either:

* A new (usually closer) DC comes online. That will probably be your destination from now on.

* The prior DC (or a critical link on the path to it) goes down.

The crucial thing is that the client will typically be routed to the closest destination to it. In the egress case the current DC may not be the closest DC to the server it is trying to reach so the return traffic would go to the wrong place. This system of identifying a server with unique IP/port(s) means that CF's network can forward the return traffic to the correct place.

Dang this stuff is super interesting. As someone who knows very little about networking and DNS, both your comment and the parent comment are incredible insightful and illuminating! Thanks for sharing.

It works because the route to 1.2.3.4 is relatively stable. The routes would only change and end up at data center #2 if data center #1 stopped announcing the routes. In that case the connection would just re-negotiate to data center #2.

Ah, ok, that makes sense. So for a given point of origin, anycast generally routes to the same server?

Correct. From a single place, you're likely to BGP-reach one Cloudflare location, and it doesn't change often.

This is a problem in theory. In practice (and through experience) we see very little routing instability in the way you describe.

You mean, it's just luck?

calculated luck

right? also seems like load should or at least could be changing all the time. geo or hops proximity is really the only things that decide a route? not load also?

But although I would be surprised if load were not also part of the route picker, I would also be surprised if the routers didn't have some association or state tracking to actively ensure related packets get the same route.

But I guess this is saying exactly that, that it's relying on luck and happenstance.

It may be doing the job well enough that not enough people complain, but I wouldn't be proud of it myself.

Anycast is implemented by BGP and doesn't take load into account in any way. You will reach the closest location announcing that address (well, prefix).

TFA claims that Anycast is an advantage when dealing with DDoS because it helps spread the load? A regional DDoS (where it consistently hits a small set of DCs) is not a common scenario, I guess?

Basically yes. Large-scale DDoS attacks rely on compromising random servers and devices, either directly with malware or indirectly with reflection attacks. Those hosts aren't all going to be located in the same place.

An attacker could choose to only compromise devices located near a particular data center, but that would really reduce the amount of traffic they could generate, and also other data centers would stay online and serve requests from users in other places.

Why shouldn’t they be proud of a massive system like Cloudflare that works extremely well? As a commentor below described, it’s not luck or happenstance, it’s a natural consequence of how BGP works. Seems pretty elegant to me.

Your intuition is more or less all wrong here, sorry.

Most routers with multiple viable paths pass was too much traffic to do state tracking of individual flows. Most typically, the default metric is BGP path length, for a given prefix, send packets through the route that has the most specific prefix, if there's a tie, use the route that transits the fewest networks to get there, if there's still a tie, use the route that has been up the longest (which maybe counts as state tracking). Routing like this doesn't take into account any sort of load metric, although people managing the routers might do traffic engineering to try to avoid overloaded routes (but it's difficult to see what's overloaded a few hops beyond your own router).

For the most part, an anycast operation is going to work best if all sites can handle all the forseable load, because it's easy to move all the traffic, but it's not easy to only move some. Everything you can do to try to move some traffic is likely to either not be effective or move too much.

In addition to what others have said about route stability, there is possibly a more relevant point to make: for 'ingress' traffic, the anycast IP is the destination, and for 'egress' traffic it is the source. This distinction is important because the system doesn't have any reasonable way to anticipate where the reply traffic to the anycast IP will actually route to. A packet is launched to the Internet from an anycast IP, and the replies could come to anywhere that IP is announced - most likely not the same place the connection originated. Contrast to when the (unicast) client makes the first move, and the anycast node that gets the initial packets is highly likely to get any follow up packets as well, so in the vast majority of cases, this works just fine.

As others have mentioned, this is not often a problem because routing is normally fairly stable (at least compared to the lifetime of a typical connection). For longer lived connections (e.g. video uploads), it’s more of a problem.

Also, there are a fair number of ASes that attempt to load balance traffic between multiple peering points, without hashing (or only using the src/dst address and not the port). This will also cause the problem you described.

In practice it’s possible to handle this by keeping track of where the connections for an IP address typically ingress and sending packets there instead of handling them locally. Again, since it’s a few ASes that cause problems for typical connections, is also possible to figure out which IP prefixes experience the most instability and only turn on this overlay for them.

Yes, as others have mentioned, route flapping is a problem. But, in practice, not as big a problem as DNS-based routing.

- See: https://news.ycombinator.com/item?id=10636547

- And: https://news.ycombinator.com/item?id=17904663

Besides, SCTP / QUIC aware load balancers (or proxies) are detached from IPs and should continue to hum along just fine regardless of which server IP the packet ends up at.

Reply on news.ycombinator.com