TCP Over IP Anycast – Pipe Dream or Reality?

We had the same experience at Quantcast. Better results with anycast within continents, and DNS to distribute a per-continent anycast addresses.

Some know-it-all busybodies like to proclaim that anycast can't work for TCP because the route can change during the session, but the reality is that:

- route changes are infrequent

- individual sessions are typically much shorter than route change intervals

- we never experienced proposed problems with sustained route flapping

- route changes are readily detected and handled as (rare) dead connections. The nicest bit is that transferred live connections can get rejected and closed immediately by the new server allowing for rapid recovery (unlike DNS changes which leaves things hanging for a long time).

We had extremely fast datacenter failover using anycast (100% of traffic moved as soon as 5 seconds). Contrasted to the minutes-to-hours nightmare delays that characterize DNS failover. (The DNS time-to-live is an advisory at best, and outright ignored by many clients)

It's too bad more hosting providers don't allow route announcements.

What is LinkedIn sending that generates so much traffic that they need this? They're not Netflix.

(Never mind, I just looked at LinkedIn, which I don't use much. With their new layout, I had to scroll through three screens of ads to get to anything that affected me.)

Having done TCP Anycast for more than 8 years in production now, with extensive testing, and third-party monitoring, there has never been an issue with it.

When a route flap happens that causes an AS path selection change, the user almost universally routes back to the exact same node they would have routed to prior to the flap.

The theoretical "IT person in texas doing per packet load balancing on a Sprint T1 to Atlanta and an ATT T1 to Denver" doesn't actually exist.

Sometimes you don't have as much control over a client (many non-HTTP TCP applications). A middle ground is to use anycast for authoritative DNS and return a set of records for unicast IPs local to that POP. You get the benefits of anycast and the stickiness of unicast. It's a short jump to overriding decisions to correct certain clients, or to shed traffic from servers or POPs.

Would be interested to read how CloudFlare solve flapping problem with their Anycast network: https://blog.cloudflare.com/a-brief-anycast-primer/

Most important - how much has this work improved linkedin's site performance? 30%? 40%? 50%?

We are working on this as well at http://datapath.io. You can use multiple iaas providers using global elastic IP addresses that you can announce from multiple locations at the same time. If you want to participate in our beta please write to beta@datapath.io

Best, Sebastian

"Anycast has historically not proven to be useful for stateful protocols in the internet because of the inherent instability of the internet."

Inherent instability meaning packet-switched networking, a.k.a. the most basic mechanism of how the internet works.

This is exactly how EdgeCast CDN (now Verizon) has worked for many years, down to the ip per region approach. So this is known for a long time to be a good way to do global content delivery.

I'm curious why linkedin didn't just use a commercially available CDN service for this? Could save them a lot of time and maintenance, while probably getting a better result because CDNs have this down to the details.

it has been a reality for many folks for years. real-world anycast can be tricky, but it is certainly deployable and usable.

Plan9 can turn any protocol into TCP via 9p.

Connect to a remote plan9 machine by whatever means using 9p, import /net from the remote machine into your local namespace, et viola you now have all the network connections of that remote machine available, including the tcp stack.