The missing part for me was that fly.io supports WireGuard access to your infrastructure [https://fly.io/docs/networking/private-networking/]. Did not know this!