greenpau/caddy-security is fantastic and "just works" for OIDC sso.
mholt, thanks for recently adding the ability to bind to multiple specific IP addresses by default, this help me conserve precious public IPv4 addresses.
*http3/quic gets around this be setting a connection id in the UDP packets so mobile phones jumping towers can still be tracked if they get a new IP.
It's called BGP protocol, as OP mentioned. This is basically how cloudflare works, you have the same IP for dozens of datacenters around the world. If routing changes and suddenly you are routed to different POP then your TCP session is broken but in most cases this is rare. You could "fix" this by routing TCP packets to POP B when detecting on POP A that session origin was POP B but this would incur additional latency for the rest of the session etc. But this has nothing to do with Caddy, you could use any other software same way using anycast IP.
I think I wasn't clear enough. Yes, BGP will route the packet to the nearest POP for that IP if it was announced in different locations, that's how "anycast IP" works, thanks to how BGP works. OP asked:
> or something in front of Caddy
So I've described what is it and how it works in simple terms.
> If routing shifts in the middle of a TCP connection, there's no magic that makes a new machine that didn't SYN+ACK the original connection pick up where the old machine left off.
If you are talking only about BGP then of course you are right but I was not talking about solving this using only BGP. You can tunnel the packet to your other POP which was origin, as I mentioned in my answer.
So if you have a session established after 3 way handshake in POP B
client <-> POP B
and suddenly you are routed via POP A then just tunel all packets for that session:
client <-> POP A <-> POP B
or you could use MPTCP to solve this if supported etc.
Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up. But if you care about it --- as the previous commenter does --- you want a real answer.
Yeah, I wasn't clear enough in my first comment, my last comment was more detailed so I guess it should match "a real answer" now.
> Obviously, in reality, people mostly rely on geography to deal with this: you're just unlikely to get randomly rerouted from a Singapore rack to a Sydney rack, and so the issue doesn't come up.
Yeah, most of the time it doesn't come up, anecdotally it's also more specific to some regions, in Europe for example I've seen this to occur more often (between POPs in EU) than in other regions.
1: EdgeRouter4 connected to ISP. This operates as a typical router, nothing special other than some static routes to item 2.
2: Pair of VyOS routers running in VMs. I think of them as top of rack L3 switches in a datacenter. Their primary purpose is for ECMP [1]. They also use VRRP so the can be upgraded without downtime.
3: Three node caddy cluster I mentioned. Running plain old Debian. anycast-healthchecker configures the anycast IP on each VM's loopback interface. Bird advertises the address to the two VyOS routers via BGP.
4: Caddy binds to each address.
All of the hosts in this network are configured with a default route to the VRRP address on the VyOS VM's. That's pretty much it.
I've recently started using Calico in k8s on some separate VM's, and they work the exact same way, advertising Services of type LoadBalancer to the VyOS routers which does ECMP across the nodes. I learned quite a bit configuring anycast-healthchecker with Caddy, then comparing it to how Calico works.
[1]: https://codecave.cc/multipath-routing-in-linux-part-2.html