I don't use Podman desktop but I haven't had any issues at all with basic Podman. It just seems like a better overall design. It would be great if Docker copied the rootless / daemonless approach if possible.
Still, having a gitlab runner with a docker in docker setup that can access the host docker socket is not something you do lighty, so I am eyeballing buildah etc.
I guess what really grinds my gears is that I'd use podman a looooot more if it could nicely coexist with Docker. I'd use Docker for the big services and podman for all my own stuff. But that is just not possible (at least not afaik), so it is also difficult to switch step by step (yes VMs, bla bla, but it's all complicating).