The only downside is they're not really an answer to docker-compose on the local development side and the podman team doesn't seem super interested in tackling that segment. User containers are nice for long running local test infra (i.e. a background database) but are too clunky for a normal compile-> docker compose up -> test -> docker compose down loop. The best answer is either .kube Quadlets (kubernetes plays) or using docker compose [0] against the podman socket.
Either way, I've enjoyed using quadlets enough that I've spent the last few months writing a gitops tool for managing them in my spare time. They just feel like the right way of managing containerized servers.
[0] NOT podman-compose, which the article points out as being not very good and under-developed. Podman implements most of the compose spec so you can use docker compose for most situations. I suspect many people who tried Podman when RH first started pushing it ran into Podman 3 being kinda of bleh and podman-compose being awful and bounced off it.