I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?
I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.