I don't like using Docker on a VPS because of the way it bypasses the firewall rules. It's such a pain that I avoid it altogether for anything public.
Even rootless docker usually pierce through that. Might be some setting in a podman, thought, never really worked with that.