Hacker News Reader

Comment by zeroxfe | Hacker News Reader

Anyone have experience with this? I love the idea of it being daemonless.

If you run it as non-root it is significantly slower than docker as root. Docker can us the overlay2 kernel driver, whereas podkan would use fuse-overlayfs in userspace. This has a high CPU overhead (e.g. don't try to run AFL inside podman), and a 1024 FD limit for the entire container (so a 'make -j40' usually dies).

There are ways around it: raise the ulimit for your user and run new enough podman to raise limit for fuse-overlayfs, use the 'vfs' driver (it has other perf issues). I heard (but haven't tested yet) that the 'btrfs' driver avoids all these problems and works from userspace. Obviously requires an FS formatted as btrfs...

There are also compatibility issues with Docker. E.g. one container was running sshfs inside Docker just fine, but fails with a permission error on /dev/fuse with podman.

Are these fuse limitations or fuse-overlayfs limitations ?

I think FUSE limitations: fuse is served by a single userspace process, which is limited the same way as any other userspace process by ulimit. It is not a fundamental limitation of podman, just of podman's default rootless behaviour.

You can run podman as root, but it doesn't default to it, for generally sensible security reasons.

Also, docker runs as root, so it won't have permissions problems. You can change the permissions of /dev/fuse if you want to allow podman containers to access it or update the group of the user launching podman.

If I understand correctly support for native rootless mounts is currently under development: https://github.com/containers/storage/pull/816 The functionality requires Linux kernel 5.11 (soon to be released)

That is correct. We're targeting that work for RHEL 8.5 in November-ish, so you'll likely see that drop in Fedora and other Linux distros sooner.

I tried it, and went back to Docker with BuildKit.

* The newer version of Docker+BulidKit supports bind-mounts/caching/secrets/memory filesystem etc. we used it to significantly speed up builds. You could probably find a way in buildah to achieve the same things, but it's not standard.

* Parallel builds - Docker+BulidKit builds Dockerfile in parallel, in podman things run serially. The combination of caching and parallel builds and Docker being faster even for single-threaded builds, Docker builds ended up an order of magnitude faster than the Podman ones.

* Buggy caching - There were a lot of caching bugs (randomly rebuilding when nothing changed, and reusing cached layers when files have changed). These issues are supposedly fixed, but I've lost trust in the Dockerfile builds.

* Various bugs when used as a drop-in replacement for Docker.

* Recurring issues on Ubuntu. It seemed all the developers were on Fedora/RHEL, and there were recurring issues with the Ubuntu builds. Things might be better now.

* non-root containers require editing /etc/subuid, which you can't do unless you have root access.

More information about the new BuildKit features in Docker:

https://pythonspeed.com/articles/docker-buildkit/

FWIW, I'm using Podman and I'm happy with it given that it's fast enough for my purposes and I haven't encountered any caching bugs.

Buildkit's caches are a killer feature. They go a long way towards making Docker builds usable for iterative development.

We also migrated back to Docker from Podman because of Buildkit and the general bugginess of Podman.

One huge advantage over docker is that if you mount a directory into the container, and the container writes files there, on the host system they always have the owner of the user that started the container.

That makes it much more convenient for build environments, without having to hard-code user IDs both in the container and on the host.

If you're on a Linux system you can actually make this work better with sssd. So sssd's architecture is actually client-server over a unix socket.

So all you need to do is create a very simple base container layer that just installs sssd-client, and wires up /etc/nsswitch.conf to use it (your package manager will almost surely do this automatically). Then just bind mount the sssd socket into your container and boom, all your host users are in the container.

If you already log in with sssd you're done. But if you only use local users then you'll need to configure the proxy provider so that sssd reads from your passwd on your host. In this case the host system doesn't actually have to use sssd for anything.

Sounds interesting, do you have a link to an example where this is done/demonstrated?

(Also I'm not sure how that's better (and not just different), except maybe it allows more than one host user in the container, but I haven't had a use case for that).

This is glorious and just the thing I was looking for. I am trying to move towards an even more container based dev environment, basically shell into long running containers. Maybe even window manager in docker .

Totally eliminates dependency hell, e.g. ROS heavy workflows where it wants to control every part of your environment.

From limited experience: podman and the associated toolset (buildah, skopeo) show a lot of promise. That said, they are evolving and may currently require administrative care and attention when used in production.

If your environment would benefit from smaller, isolated containerization tools then my two cents would be that it's worth keeping an eye on these as they mature, and perhaps perform early evaluation (bearing in mind that you may not be able to migrate completely, yet).

The good:

- Separation of concerns; individual binaries and tools that do not all have to be deployed in production

- A straightforward migration path to build containers from existing Dockerfiles (via "buildah bud")

- Progressing-and-planned support for a range of important container technology, including rootless and multi-architecture builds

I have a Makefile for a Rust project which binds the local repository to a Docker volume, builds it in the container using muslrust, and then does a chown to change the target directory back from root ownership to my own user.

All I had to do was 's/docker/podman/g' and remove the chown hack and it works fine: https://github.com/sevagh/pq/commit/6acf6d05a094ac2959567a9a...

It understands Dockerfiles and can pull images from Dockerhub.

I had pretty negative experiences with it. I'm not sure how much of my experience is _actually_ the fault of Podman and how much of it was around using it as a drop-in replacement for Docker. It baffled me though and I'm a long time Docker and Linux user.

I used Arch Linux and followed their docs[1], mainly because I wanted it to also run without root. Big mistake. Attempting to run a simple postgres database w/ and w/o root permissions for Podman resulted in my system getting into an inconsistent state. I was unable to relaunch the container due to the state of my system.

I mucked around with runc/crun and I ended up literally nuking both the root and non-root directories that stored them on my computer. I reset my computer back to Podman for root only, hoping that I had just chosen a niche path. It still would leave my system broken, requiring drastic measures to recover. No thanks.

After much debugging, finally switching back to Docker, I realized my mistake: I had forgotten a required environment variable. Silly mistake.

Docker surfaced the problem immediately. Podman did not.

Docker recovered gracefully from the error. Podman left my system in an inconsistent state, unable to relaunch the container and unable to remove it

Again, I'm not sure how much of my experience was just trying to force Podman into a square hole? I'm sure there are other people that make it work just fine for their use cases

Edit: I should note that I used it as a docker-compose replacement, which is probably another off-the-beaten path usage that made this more dramatic than it should have been

1. https://wiki.archlinux.org/index.php/Podman

I don't know when you tried this but I recall it having sucked in the summer when I tried it and I gave up. New job in the fall, decided to give podman a shot "if it doesn't work on the first try I'm going to use docker", then podman-compose 3rd party instructions (I can't remember which) just works out of the box. I'm running a fleet that includes postgres and redis.

I've been using my company's docker compose script transparently with podman-compose for three months now and I have basically forgotten that I've switched. I actually didn't use docker before this, so I can't comment on what the difference is.

I feel like the podman experience is binary, either it works perfectly or it sucks balls. My suggestion is to give it a try, if it fails, then maybe file a bug report, fail fast and fall back on docker.

I've used podman-compose for a few web projects. Rootless means you can't bind to privileged ports etc. but after a bit of fiddling I can now spin up my choice of DB/server/mock pretty quickly - just like I could with docker.

The lack of 'cannot connect to dockerd' mysteries makes for a much-improved developer experience if you ask me.

Out of curiosity, why do you prefer daemonless setups?

If I have 30 containers running, why should a single daemon being restarted cause all 30 to shutdown as well?

Similarly, the docker security model is that there isn't a security model. If you can talk to the docker socket you have what ever privileges the daemon is running as.

So Docker supports live restore https://docs.docker.com/config/containers/live-restore/ which addresses the first point.

Second point, yep if you run Docker as root and someone can access the socket file they get root.

If that's a concern, you can run Docker rootless.

And as we're talking file permissions on a local host to allow that access, the same applies to podman containers does it not? If there are permission issues allowing another user to use access the container filesystems, you have the same problem.

Rootless Docker is basically a joke, I've tried to run production workloads on it for about a year before I gave up. Numerous docker images will refuse to work, a bunch of things will subtly fill your error logs with warnings and it doesn't mesh well with running docker swarm at all.

Rootless docker only left experimental status with 20.10 which came out in December 2020, so maybe they would have addressed some of those issues...

As to swarm, I was comparing Docker rootless to podman, which is more a developer use case than prod. container clusters.

When you use a system-level daemon[0], the daemon has to have privileges to start a container as anyone (that is... root). In a daemon-less environment, you only need the privileges of the user who is starting the container.

[0] I suppose you could have a user-level daemon that runs for each user that needs to run containers, but that's even more overhead.

Docker does allow for daemonless execution, but as you say one daemon per user will add a bit of overhead.

There's some tradeoff I guess though, between rootful setup and per user, as images duplication per user could add up.

It's a pain having to setup root access or a user for Docker. At a financial institution I worked at we had to waste about half a day to get this setup (and that's once we worked out who we had to speak to).

FWIW you can run Docker rootless (as an ordinary user) now.

At my firm you're forbidden from getting root access, docker is not allowed, podman is.

I’m a GPU driver dev and you guys are talking ~alien~, right now. What does “daemonless” mean? Why is that good?

In Docker each container process is a child of the Docker daemon process. If you need to apply security patches to the Docker daemon it kills all your running containers.

Technically the parent process for a contained process with Docker is containerd-shim.

Also, Docker does support live restore if you want to keep containers running over daemon restarts https://docs.docker.com/config/containers/live-restore/

Reply on news.ycombinator.com