Podman 4.0 brought a new network stack for rootless called Netavark. Works nicely for me.