Podman in Linux | Hacker News Reader

Podman in Linux

diego-pacheco.blogspot.com

146 pointsdiegopacheco89 comments

Somehow a lot of these blog posts seem to get the fine technical details of the marketing speak wrong.

> Docker recently changed the license

From my understanding this is related to Docker Desktop not Docker. I'm pretty sure Podman can't replace Docker Desktop. This is also what the linked website says.

Or did I miss something?

And in the linked Blog post it says

> Docker was dropped from Kubernetes.

which from my understanding is also incorrect, because it's the docker-shim which is deprecated (not dropped) and Docker inc could probably make Docker Engine CRI compliant.

It seems to me that almost no one understands the difference between the various layers of "Docker" (that is to say: Desktop, CLI, daemon, containerd, runc) and which layers tools like Podman aim to replace.

To be fair, podman has it's own "Docker Desktop" part, podman machine. But of course on Linux this doesn't make any sense.

From what I've seen in all these Podman articles, I've come to the same conclusion as you.

This is how Podman fits in:

Podman (and its various components) can replace everything but parts of Docker Desktop, namely the GUI that Docker Desktop, has. For everything else, it has it's own Docker compatible CLI, an optional daemon (by default it's daemonless), there is no containerd component as it communicates directly to any OCI compatible runtime. They've also implemented their own OCI compatible runtime, crun, which is supposed to be faster than runc, and more lightweight.

I suppose for the average Joe/Jane this probably doesn't matter. They want to run a few commands the first time around to set up, and then do "start/stop/remove my container" and that's all they want to do.

Docker Desktop isn’t really about the GUI so much as the management of, and host integration with, a Linux VM for running on non-Linux OSes. Notably, there’s no Docker Desktop for Linux, only macOS and Windows.

Yep. Podman takes cares of those things using the `podman machine` sub-command. From what I understand, currently only macOS might be working, but Windows support will be coming if it's not already there.

the only issue really left is that volume mounts don't work with podman machine as of yet. The issue has been live on github for awhile, but its finally getting traction. Likely because of the recent Docker for Desktop changes.

EDIT: github link: https://github.com/containers/podman/issues/8016

AFAICT podman machine relies on VirtualBox. This would be an enormous performance and resource regression on MacOS — Docker Desktop uses the much more performant and resource-conscious native hypervisor frameworks instead.

If true, also untenable since last I checked VBox doesn't plan to support ARM hosts (including the new Mac chipset)

This doesn't seem accurate.

According to a post on the podman website: "Current implementation relies on qemu which currently has some platform dependencies." [1]

I also just checked on my machine and it is indeed running on Qemu.

IIRC Qemu also has support for Apple's Hypervisor.framework.

[1] https://podman.io/community/meeting/notes/2021-04-06/#podman...

You are correct - thanks!

> Notably, there’s no Docker Desktop for Linux, only macOS and Windows.

Looks like they are adding it https://twitter.com/glours/status/1438526841577357315

I quote: "A dream coming true for a lot of #linux users, @dieuthicao announced that we’ll start working on a #Linux version of @docker Desktop"

What does Docker Desktop even really offer on Linux? k8s, I guess? There are other open-source (Docker Desktop is proprietary ofc) ways to do that.

As a Linux user, far from a dream at all, really.

My understanding is that Podman is working on a "GUI" since it is one of the most requested items. Taken from the interview with the founders of Podman over on the Google Kubernetes podcast: https://kubernetespodcast.com/episode/164-podman/

ETA: Oh hey here it is: https://github.com/heyvito/podman-macos

Good to know about the GUI!

> ETA: Oh hey here it is: https://github.com/heyvito/podman-macos

This GUI isn't an official Podman application. It's a third party. Not that it's bad or anything, it's just not official.

Yeah true. They talk about a first party app on the podcast episode tho, I'm just not smart enough to find it.

> For everything else...

Except for:

  - Docker Compose (because podman-compose has a large amount of open issues https://github.com/containers/podman-compose/issues)
  - Docker Swarm (since Docker provides a lightweight orchestrator out of the box), you'd need to use something like K3s or another Kubernetes distro
  - anything that uses the Docker socket (/var/run/docker.sock), so you can forget about running anything like Portainer or any of the other tools out there

No longer true. Podman nowadays ships with an optional daemon that serves a Docker-compatible interface, so you can use docker-compose directly if that's what you want.

See for example: https://www.redhat.com/sysadmin/podman-docker-compose

That does seem better! Of course, there are a few lingering issues with support, but overall the trend is good: https://github.com/containers/podman/issues?q=is%3Aissue+is%...

Of course, there's no Swarm support, as evidenced by that very article:

> Caveats

> One known caveat is that Podman has not and will not implement the Swarm function. Therefore, if your Docker Compose instance uses Swarm, it will not work with Podman.

Feels like people will either be pigeonholed into Kubernetes for all of their deployments, or will have to migrate over to something like Hashicorp Nomad: https://www.nomadproject.io/

Curiously, it also supports Podman as a task driver: https://www.nomadproject.io/docs/drivers/podman

Yes, but Docker won't work on Windows without having some machine-compatible integration with Hyper-V. Docker Machine was deprecated in favor of Docker Desktop:

https://docs.docker.com/machine/

For now however, you can use minikube which provides drivers for Hyper-V and hyperkit:

https://minikube.sigs.k8s.io/docs/drivers/

works whit wsl2, who is include in non pro versions of window, and i think the installer works out of the box (if you have the ubuntu/wsl2 installed)

I was never able to get docker to properly work with WSL2 without installing Docker Desktop and enabling the integration. WSL2 just seemed too buggy for my likes and I am a happy Arch Linux user.

But on Windows/Mac there really is not a way to run Docker w/o Docker Desktop installed. And that could be a thing confusing people.

Linux, yeah, I don't think there is a Desktop version for that. It's all CLI

Podman can replace docker desktop. It doesn't have the same GUI but it does have the capability to run containers via familiar CLI commands on a variety of OSes, connecting to (and in some cases provisioning through podman machine) a VM when required

I also think that it's more than safe to say that K8s is dropping Docker when they've deprecated it as a container runtime

You can also run regular docker with vagrant instead of Docker Desktop. This seems to be the way you can replace Docker Desktop with Podman: https://www.redhat.com/sysadmin/replace-docker-podman-macos

There was Kitematic for Docker, I think it was independent, but it's owned by Docker now and they shut it down. https://github.com/docker/kitematic

It evolved to become Docker Desktop

No it didn't, Docker Desktop was already released before Docker bought Kitematic. I have receipts: https://www.zdnet.com/article/docker-reels-in-kitematic-as-i... (2015) https://en.m.wikipedia.org/wiki/Docker_(software) (Initial release: 2012)

If you mean Docker Desktop added some features from Kitematic, that's a misleading way of saying it.

Guess I don't know what I'm talking about

Hmm, I can't find an official Docker installer for Mac that's older than this (2015-08-11), and it has Kitematic. Wikipedia date is for Docker in general. I stand corrected. https://github.com/docker-archive/toolbox/commit/a7e7a070ba9...

The kitematic peeps are wonderful, smart, and genuine people.

> I also think that it's more than safe to say that K8s is dropping Docker when they've deprecated it as a container runtime

This is wrong. Docker itself is no longer a container runtime: it has spun out that capability into containerd. Kubernetes can now call Docker’s container runtime (again: containerd) directly instead of going through a redundant docker-shim.

In other words: Kubernetes has dropped Docker’s old container runtime in favor of… Docker’s new container runtime.

Docker Desktop also includes a full Kubernetes distribution. It is actually the easiest way to get a local K8s cluster on Windows or Mac.

Docker desktop also includes Docker-compose (a way to chain multiple docker container together)

And of course the UI.

Podman is just replacing the basic docker CLI. I would say that podman replaces about 20% of what Docker desktop offers.

It depends on what you mean by “replace” Docker Desktop, but Podman can run a VM on macOS and has replaced Docker Desktop for me. There’s no fancy (useless) UI nor Windows support yet though.

Windows works via a "Bring your own VM" approach, but since WSL exists I'd expect most Windows users would just run it there

When you install Docker desktop and already have wsl2 installed you get a pop up saying “you just want us to use wsl2?” And away you go. At least that’s what happened when I did it 6 months ago. Works great!

If you don't need to run `docker` command under Windows, install docker directly from distro repo is enough. No need to run another background service. Another benefit is it is faster to build docker images when the files are large, because it doesn't need to move files between WSL VMs.

I thought there was Windows support for Podman the same way that Docker Desktop provides Windows support -- primarily through WSL2?

podman machine uses qemu with Hypervisor.framework on macOS

Oh, nice! Thanks for the correction.

You are correct on both counts. Most people are not intimately familiar with the details of the Docker and Kubernetes stacks. Unscrupulous vendors have been taking advantage of this confusion to spread FUD and shine a more flattering light on their own products.

> I'm pretty sure Podman can't replace Docker Desktop.

Multiple siblings here have already mentioned that it absolutely can (and does for many).

My question is: in what way does it not? What is Docker Desktop doing for you that podman lacks? Just UI?

Volume Mount integration is the big one for me. UI is neither here nor there, but managing the VM is what Docker Desktop means for most users. Podman Machine is getting there, but not quite there yet.

Docker Desktop also includes a full Kubernetes distribution. It is actually the easiest way to get a local K8s cluster on Windows or Mac.

Docker desktop also includes Docker-compose (a way to chain multiple docker container together)

And of course the UI.

Podman is just replacing the basic docker CLI. I would say that podman replaces about 20% of what Docker desktop offers.

I think you are reading their marketing speak too charitably by assuming it contains "fine technical details". They purpousefully renamed everything to "Docker" to get people think that the desktop virtual machine offering is the same thing and this caused a lot of confusion.

podman user here, because of the ability to run in rootless mode.

Using it on RPi4 with Fedora IoT running linuxserver io containers.

Appreciate the systemd integration as well making the containers services that gracefully go down and come up when the pi gets rebooted without me needing to do anything.

It’s great that podman serves a good role for you, and I’m not going to argue that. My points:

1. Docker containers absolutely can be run without root. Yes, it’s not the default policy, but containers can have a user ID. If you are referencing the daemon-less root-less nature of podman, that’s a clear advantage of podman vs Docker. 2. Docker containers also have a restart policy which I use to also have them startup on machine reboot. By graceful, you must mean sending SIGTERM to the containers which Docker does as well.

Perhaps podman does these things better, but I want to point out that Docker does have many features for better or for worse.

Thanks

> root-less nature of podman

I see this repeated a lot, but it's not the default, its has to be explicitly configured: https://github.com/containers/podman/blob/v3.3.1/docs/tutori...

And in addition to the known upsides, there are some lesser known downsides:

1. There are feature limitations with it: https://github.com/containers/podman/blob/v3.3.1/rootless.md

2. There are security implications, quoting Arch Wiki:

> Warning: Rootless Podman relies on the unprivileged user namespace usage (CONFIG_USER_NS_UNPRIVILEGED) which has some serious security implications, see Security#Sandboxing applications for details.

Also worth noting that Docker itself has a rootless mode as well by now: https://docs.docker.com/engine/security/rootless/

I'm happy that there are Docker alternatives, but I have the feeling that podman has been hyped a lot recently and many articles and comments give the impression that it's more secure by default and without any downsides.

Why should I trust the Arch wiki? People like Christian Brauner think the value of not running as UID 0 outweighs the increased attack surface from the user namespace.

https://people.kernel.org/brauner/runtimes-and-the-curse-of-...

Thanks for writing this. He knows what he is talking about. One of the LXC maintainers.

AFAIK, the default packaging on Fedora enables rootless podman without additional configuration.

Probably worth pointing out that docker has had a "userns-remap" option for quite a while, which causes all containers to run in a separate user namespace where UID 0 inside remaps to something else outside, so theoretically a user with access to the docker daemon isn't able to view the outside filesystem as root[0].

I have gone back and forth with podman. At some point it seemed to sometimes get into a funny state where I would simply delete everything[1] to fix it. On all systems where I run docker, I make sure to have "userns-remap": "default" in /etc/docker/daemon.json. Haven't looked into the rootless mode yet, but I was aware a few years ago that they were working on it.

[0] Without remapping root inside a different namespace, anyone with access to the docker daemon can access the outer root filesystem as root using a command such as `docker run --rm -it -v /:/oops alpine`

[1] Amusingly, the simplest way to do this was by using root to run `rm -rf` on my own ~/.local/share/containers/ directory, since the containers used UIDs other than my own (ones that are part of my subuid range)

When people talk about “rootless” in this context, they’re not talking about “launching containers whose process runs as non-root”. The innovation of podman is that a non-root user can “safely” be given permission to launch containers whose maximum permissions are “the perms that user has”.

Docker doesn’t have this functionality: the daemon runs as root, and anybody who is granted access to launch containers by invoking Docker commands can inherently access root-level privileges. The most mundane way to do this is to launch a container with the host’s namespaces instead of generating new ones.

It's not the default setup and not trivial to do, but Docker has had the ability to run the daemon as a non-root user for a few years. The standard .deb and .rpm packages even include scripts to automate the transition for you on Debian and RedHat Linux variants. See https://docs.docker.com/engine/security/rootless/

The only thing podman gives you that docker itself can't is running without a daemon at all.

I was enamored with this feature too, but a comment here on HN[1] made me reconsider its use. Apparently rootless requires unprivileged user namespaces, which provides a different security context than most apps are expected to run in, and might be less thoroughly tested than you would think.

I still like the systemd integration and that it doesn't require a daemon too, and I still favor it over Docker.

1: https://news.ycombinator.com/item?id=28393949

Edit: Clarified that it's the unprivileged user namespaces feature specifically, not namespaces in general. Thanks for the feedback solarkraft.

Thanks for linking the original comment. Every container uses namespaces, this is (TL;DR attempt) about user namespaces, which is a not particularly well audited kernel feature, meaning you may open up potentially insecure kernel code to unprivileged users.

The way I understand it, with containers running under a root user, is that to break out of a container you‘d have to find a vulnerability in standard (rootful) namespaces, which is much less likely (since it’s the same thing everything including Docker uses).

But the vulnerability without usernamespaces is much more severe, because the process then has UID 0 on the host.

Frankly if you are that concerned about security (e.g. you have multitenant workloads or are dealing with sensitive data), you should be using KVM or gvisor.

If you're using user namespaces and allowing a previous root only API to be used by a user, it's possible that some escape in that which provides root access (which wouldn't be as huge a deal if it was only ever able to be run as root) allows root level escalation outside the container.

That intermixes with security concerns about what's possible if running as root directly in different ways, and be more or less problematic than a root container depending on the use case, and also more or less likely based on how well those APIs are exercised for the specific use case.

It's not that these should be avoided, it's just that people should be aware that it's not necessarily a pure security increase at the expense of a bit of extra CPU due to kernel checks. There's a bit to consider. Maybe later everyone will consider this tested enough that's it's mostly a pure win. Maybe it's already at that point but people haven't internalized it. I don't know enough to know what stage we're at, but I thought it was worth mentioning, as it took me by surprise when I learned of it.

"the vulnerability" means one specific vulnerability in docker or somesuch "privileged container" i presume. there are also some sleeping in the kernel code that userns opens up, and even outright intentionally allow, that were previously not on the radar.

ultimately we must consider userns vs privileged-ns a fork in the road. one direction sweeps privilege concerns under the rug, and opens up new attack surface today leaving the door open for more non-obvious problems tomorrow. the other relies on highly competent engineers that know the nuances of the system they are working with, and have strong will to stomp out needless complexity from design to implementation.

I am generally conservative about these things, but recently the consensus appears to be that unprivileged user namespaces are stable enough for GA. If you work in a targeted environment (e.g., you are a reporter covering civil liberties), maybe you should wait a bit longer, but for average people the advantages of rootless containers probably exceed the advantages of keeping the unprivileged_userns_clone code out of your attack surface.

I would encourage most security-conscious users to enable it and migrate to recent podman over using Docker, assuming a sufficiently recent kernel. The latest batch of major Linux OS releases have all enabled kernel.unprivileged_userns_clone, so Red Hat, Canonical et al seem to agree.

For those interested, though, you can read the anatomy of a userns clone() vulnerability here:

https://lwn.net/Articles/543273/

I love podman. When you write a file to a volume from a container, the user that spawned the container is the owner. This is exactly what you expect, but with docker, everything is written as root. After many years, this has not been fixed:

https://github.com/moby/moby/issues/2259

This has made it impossible to use docker in scripts where root access is not available. Thank you podman!

Thank you! This has been my main headache with docker. I'll switch to podman at home.

Podman is not (YET) a drop-in replacement for Docker CE (on Linux), at least for me (even after 3.0.0 release when it adds seamless Docker Compose support, the way it does it contradicts daemon-less...).

Deploying stacks (on Linux) Docker Compose style are typical use cases for using Docker, `podman-compose` works sometimes, however, over half of such initial attempts fail and require tinkering.

Docker CE (at least for Linux) is still the best option when the use cases are simply clone the repo & deploy/run the stack using `docker-compose up -d`, it simply does the job at the best cost.

podman / buildah / skopeo trio is definitely worth learning, podman can be very useful to migrate containerized workloads from local / standalone dev machines to production k8s clusters, by leveraging `generate kube`.

No, it doesn’t contradict daemon-less (wrt resources used by the daemon). This “daemon” only runs activated by a systemd socket for a few seconds. Whereas Docker CE always runs at least containerd (dockerd is also socket activated). You could argue that conmon is the daemon of podman and every container has its own daemon. It’s a bit wasteful wrt memory but that how it’s designed.

> It's the natural open-source Docker replacement. Docker recently changed the license and plans.

Come on. While maybe not an outright lie, this is certainly bending the truth and leaving out crucial details–Docker CLI is still open source and free, just not Docker Desktop.

While you can use podman on Linux (and should if it fits your needs!), you're not being forced to use it because Docker became paid or anything like that!

The challenges I’ve had switching out Docker Engine, CLI, and Compose with Podman has been the absence of a vetted Docker Compose replacement.

What I’ve done so far in replacing Docker Desktop has been to use just the CLI, Engine, and Compose on WSL2.

I never liked the GUI or desktop behavior. I just install containerd, point dockerd at it, and download everything manually. This is pretty much my install, sans systemd units:

    curl -LO https://download.docker.com/linux/static/stable/x86_64/docker-20.10.8.tgz
    tar -xvzf docker-20.10.8.tgz
    mv docker/dockerd /usr/bin/dockerd
    mv docker/docker-init /usr/bin/docker-init
    mv docker/docker-proxy /usr/bin/docker-proxy
    mv docker/docker /usr/bin/docker
    rm -rf docker 
    rm docker-20.10.8.tgz

As of 3.0 you can apparently run docker-compose directly against podman in daemon mode (thought I haven't tried it).

I've been running this way since the beginning of the year. Works great, just need to set DOCKER_HOST when launching docker-compose.

When I found out Podman supports Kubernetes pod YML files with `podman play kube`, I "ported" my Compose files to those, and that works pretty well (in the newest versions of Podman).

I tried to give Podman a go on Ubuntu 20.04 and it _worked_ but I could never get it going with a docker-compose file.

There is apparently support in the works, but I can't find or get that support on Ubuntu at this stage. Shame, I was hoping it might support IPv6 better than docker currently does.

I can’t speak for Ubuntu, but all of my containers are setup in docker compose for podman things start find there is a couple tweaks like container name I had to make, but over all pretty straightforward. I’d recommend another go if you are looking for an alternative.

As I like rootless containers. I think the implementation feels awkward for local directory permission mounting volumes.

While I understand the phrasing, it's still funny to read "open-source Docker replacement".

When I first used docker on macos a while back, I was surprised with all the telemetry/phoning home it did. It started doing it at the beginning of the installer, before docker was even installed.

I spoke to Red Hat's Dan Walsh, container team lead, and Brent Baude, the lead architect of Podman, on my podcast last week. It's a good primer to the what and why of Podman, Buildah, and all the other "not Docker" tools Red Hat makes.

https://kubernetespodcast.com/episode/164-podman/

The code snippet in the gist is posted twice in the blog post... and I don't know why it's not just posted in code blocks...

Im just wondering if the full loop will ever be complete and deployment patterns start shifting to single file compiled executables.

    cp ./app.AppImage /srv/app/app.AppImage
    for deployment_host in $( <deployment_hosts jq '.[].hostname');
        ssh deployment_user@deployment_host /srv/app/app.AppImage &
    done

okay boss, what's the next problem?

app.AppImage used up all the RAM on the machine, and the kernel OOMkilled sshd. Now it's time to roll back.

(But I guess it wasn't the fault of that deployment -- you just staged the binary, you didn't start it running.)

The app.AppImage is staged to the network share and started on the remote host. The AppImage might eat all the RAM and the AppImage will get OOMkilled but sshd is unlikely to be killed. When the AppImage gets OOMkilled, the ssh session will end with an error and the ssh client will report that the session terminated unexpectedly.

I've done it (and later cringed). But it definitely works.

For programming languages that offer a good experience with static linking, sure, but even those may depend on external config files, environment variables, specific ports and sockets, all of which can potentially conflict with other programs on the same physical host. Go binaries are almost always just one file and yet are among the most popular to deploy as containers along with making up virtually all of the CNCF ecosystem.

Can you use podman with ZFS on Linux volumes?

There is a dedicated storage driver for zfs: https://www.jwillikers.com/podman-with-btrfs-and-zfs

That said, the default overlay storage driver already supports reflinks, which will get you most of the benefits.

Last I looked, ZFS didn't support reflinks. https://github.com/openzfs/zfs/issues/405

There's only one Question really: can it use a Dockerfile?

Yes, and the more standard "Containerfile" works as well (same grammar but non-branded naming).

Overall, it's a drop-in replacement with additional features and a more linux friendly underlying architecture. You can even alias podman to docker and shouldn't run into any issues.

This is good. The inability to take build artifact inputs into FreeBSD bhyve/jails, is usually why people say they can't go there (it's certainly whats limiting me) -So having an alternate explicitly handle the input I HAVE moving to the product I WANT is really good. Thanks for clarifying.

what a shitty blog :)

Reply on news.ycombinator.com