Service provider story about tracking down TCP RSTs

I have heard fastly competitors tut-tut their use of anycast as potentially unreliable in the face of issues like this.

Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone), but that's the problem Fastly are supposedly paid to deal with

Their scheme is fine and dandy with a protocol like DNS where UDP retries are transparent and TCP is a tiny fraction of weird traffic, but for business applications handling credit cards, surely the occasional RST is already too many

Or another way to look at it, basically they're saying their IP addresses are special snowflakes and actually the full address includes the route, and source networks are wrong for assuming things work the way they're supposed to everywhere else on the Internet

Arista's default of having the TTL as part of the hashing function to calculate packet path means that any TCP connection that does not have a consistent TTL when it reaches this (source) ISP's routers can potentially be bucketed onto multiple different egress paths (transit or peering connections) onto different upstream SP's.

Because BGP works on the concept of AS Numbers rather than IP's (and because Fastly uses Anycast, which announces the same set of IP's from multiple POP's), it's possible that packets that leave the ISP network over different transit connections end up at different Fastly POP's, as each transit provider or peer will have a different route to Fastly's AS.

Fastly does not share TCP session information between POP's as I imagine at that scale it's massively prohibitive to do so.

The TCP RST happens because there is no established TCP session at the secondary POP which receives the TLS handshake packet, and because TTL is part of the hash function it will always end up following a different path (and therefore hitting a different datacenter, if routing stays consistent) if the TTL is reduced by 1.

It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another, as this usually happens rarely, in response to changes in the path between two AS numbers.

For things like websites, browsers will usually just retry the connection, and as the path has changed and stays consistent, it starts to work and all you notice is a slight delay.

It doesn't work in this situation because the second packet with the reduced TTL is consistently routed to an 'incorrect' datacenter and breaks the TCP session every time, effectively dropping service to 0.