> ...and my current podman use on my workstation is to run rootful podman containers inside a VM...
This makes me feel that perhaps we're slowly moving in the direction of a full circle, to how Docker Toolbox worked inside of VirtualBox, at least on Windows: https://docs.docker.com/toolbox/
Here's an example of someone's experience with setting it up: https://medium.com/@peorth/using-docker-with-virtualbox-and-...
The industry seems to be slowly advancing, with Podman, Docker with HyperV and now with WSL2 integration, all just to solve the hard permission and runtime problem, whereas the actual OCI standard and the tooling that Docker provides for getting things up and running (environment variables, resource limits, exposing ports etc., with a few warts here and there) was sufficient from the beginning for the most part.
> I think though, you can't really replace docker with podman (without docker API) in a general sense.
With this, i agree, but perhaps i've ended up with a different set of conclusions. Docker and the tools like it are good for deploying applications in a mostly reproducible way and essentially solving the dependency management issue in a sub-optimal yet passable way.
Because of all of the tools solving this problem and it oftentimes being a chief concern for it being chosen, all else becomes secondary, short of pressing issues like exploits and such. To that end, Docker inside of a VM, with separate VMs for separate apps and separate clusters for separate environments seems like the least painful option - decent security, somewhat limited attack surface, not relying on just one technology to be secure (even Podman has its exploits), but with the wide support that Docker has.
For example, you could have your project, which has a few front end containers, a few various services for the back end and a few databases, also in containers. You could essentially have 1 VM per environment with all of those running (development, testing, accept testing) in the less important deployments, but have as many VMs and servers as you need for the important ones (staging, production). Then, have separate container clusters (with something like Docker Swarm) for your development environments and the production ones and you should be good.
Sometimes the path of least resistance isn't all that bad. At least until Podman is stable enough to be a daily driver, be it in 5 years, 10 years or never.