If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies).
There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs...
There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.