Re: NAT and source port changes
I am experimenting building a similar tech but on AWS. To use anycast, I front my Wireguard servers in multiple regions with GlobalLoadAccelerator and set ClientAffinty to two-tuple (source-ip, destination-ip) instead of the default 5-tuple (source-ip and port, destination-ip and port, protocol).
Re: Network switch (WiFi to Cellular and Mobile IP)
This one stings. I haven't impl it yet, if ever I get to the scale to warrant such a design: I was thinking abt sticky routing the traffic using only the destination port (of the wireguard server). At the time of new connection establishment (VPN turned on / off), ask the app server for a port to use and use that. On the server end, have one beefy wireguard server serving a range of ports per region behind the anycast load balancer, so that the balancer has no choice but to send the incoming to that single server that is serving incoming destination port. Use the usual IP route commands to send the traffic along to approp exit server depending on the actual destination IP (now that wireguard has decrypted the packet).
Re: Clients:
I'm predominantly focusing on Android, and the I've found things work differently across OEMs. It is just too much work. I have gone with the workaround that Blokada so wonderfully uses: employing a watchdog, heartbeat, keep-alive service, and aggressive wake-ups for some of the common problems across OEMs.
[0] Google's paper on NetworkLoadBalancer is simply amazing: https://ai.google/research/pubs/pub44824/
[1] https://ai.googleblog.com/2015/08/pulling-back-curtain-on-go...