Migrating from Docker to Podman
marcusnoble.co.uk
marcusnoble.co.uk
Goal: no more daemons to run 3rd party containers, systemd is good enough by now: resource limits, isolation, chroot, dynamic users, logging, and more.
Low-level tooling is done, I am now building ecosystem around it: easy installation, convert to deb/rpm, systemd units, etc.
I love it. It does everything I need it to, it's fast, decently documented, supports everything Docker does (just not 1:1 in terms of DX) and is present wherever I need it.
I'm considering them for an upcoming project.
As for just using systemd and not nspawn at all, that seems useful for a lot of cases, but not multiplexing of networked services since systemd can't assign UNIX domain hostnames and IP addresses to individual services. At least, I think it can't, can it?
JoinsNamespaceOf can be used to set up network namespaces (albeit in a clunky way).
It is possible, but multiplexing networked services requires significantly more effort with systemd than with others. I do not have a strong use case yet to invest in tooling for it, but it may come.
Also, as good as systemd is (and I've defended it here a few times as better than what was before), it still feels like an organically grown accumulation of edge cases that were patched. I try to keep it's use to the subset of features that are easy to grow for not just myself but those of my team, and systemd is known (in my circles at least) as being somewhat obtuse. For me, that's a good argument for a dedicated application to handle it.
+1000
Can I still get the same networking functionality with undocker/systemd that docker provides? For example, container hostname resolution, private networks, container->localhost port forwarding?
Have a look at these in systemd.exec[1]: PrivateUsers, DynamicUser, ProtectProc, RootDirectory.
There are more, but these are the main ones.
[1]: https://www.freedesktop.org/software/systemd/man/systemd.exe...
https://github.com/b0o/arch-lwc
Not intended for any sort of production. I personally use it to run Firefox inside a container and for testing.
Edit: looking at the docs is that system-nspawn that's actually doing the heavy lifting. For Linux namespaces. I feel like this has great potential!
As far as systemd vs dockerd goes, dockerd provides a bit more isolation by default, but `systemd-analyze security` can guide you so far beyond the Docker defaults. And it is always compatible with system daemons, giving consistent configuration between, say, postgresql from package manager and prometheus from docker container+undocker.
Running them is a different story. I've never tried systemd user services, so can't tell. Curious myself. :)
Isn't that what Firecracker[1] does ?
(Well, ok, not really, but a bit closer to the base system than Docker/Podman)
It's worked very well for me after a few initial hiccups a year or so ago.
Now that Podman-compose[0] is in the works, it'll really be comparable in the UX space soon, and outperforms Docker in several ways when it comes to security.
The key difference with Podman compared to Docker is that is does not run a deamon as root, like Docker does, thus all containers are created with the privilege level of the user who created it.
This can be a learning curve for those used to Docker as privileges (e.g. for filesystems, files) and capabilities (e.g. for devices, low level networking) need to be handled more explicitly as opposed to Docker's approach of "simon (root) says".
Additionally, Podman is very light weight due to the lack of a daemon since there is no service or supporting software which needs to run beyond the capabilities baked into Linux.
[0] https://github.com/containers/podman-compose
EDIT TO ADD: I run Linux both on desktop and server so I have no data for usage in Windows/Mac. Docker Desktop, as I understand it, is a Linux VM.
It's worth noting for others that (it appears from a quick read, I haven't actually used this yet), the compromise for gaining the "docker-compose" superpower is that you will have to run a podman service (Daemon). This comes counter to some (not all) of the benefits I mentioned above, but is a necessary compromise if one wants the power of compose style orchestration; that is, that there must be some deamon to manage it.
This is not authoritative, I may be mistaken, but this is my educated guess based on a quick read and my knowledge of Docker et al.
It should be said that I was mixing that with non-root podman (although that should be a supported usage).
I went back to podman-compose.
Dunno
Podman now supports the docker API which means you can use docker's own docker-compose with podman.
I'm not sure where I got the impression it was official.
The fundamental architecture of docker makes rootless awkward but the company needs to compete with podman now which is architected to fix many of Dockers deficiencies while maintaining it's many strengths.
Docker has been a great tool but running as root has always bothered me. I'm glad they're evolving but it feels a little too late and the migration to rootless, as far as I'm concerned, is not simple. Currently I'm investigating migrating my homelab and development efforts to podman.
"Podman relies on the unprivileged user namespace usage (CONFIG_USER_NS_UNPRIVILEGED) which has some serious security implications..."
A lot of hype around Podman on HN this week (two FP conversations regarding, but nothing "new" with Podman). Seems to be an intentional push to get people talking about it for not much of an apparent reason.
You can still use podman rootful and use it like Docker. Then there should be not security issues.
They recommend using Ubuntu. And they are not joking. Just click on the other distros to see the amount of hoops you have to jump through. You can't even get overlay2, which offers the best filesystem performance. Not to mention the benchmarks I've seen have slirp4netns at about ~3% of the performance of root veth. I would consider both of those incredibly meaningful limitations. You have to use sysctl/setcap to get ping working, to bind to ports <1024, muck with systemd to get user processes starting at boot, etc. etc.
It's a tough sell when you can just install root Docker with a single package command and never have to worry about a bunch of caveats that might just break or change on the next release.
Rootful-<tool> is always going to be smoother to get working. Docker can be a lot smoother than it is today, though.
It just didn't feel like a turn key solution. I have no idea how this would work in CI/CD systems though docker doesn't always need to be as secure there. Docker is a great tool but, like a lot of tech, in the mad rush to market, security was an afterthought and nowhere has it felt more clearly to me than in rootless mode.
That said, certainly a no-daemon approach takes an extra step out of the mix here.
This is NOT a drop-in replacement on Mac, very far from it.
No volumes mount from the host, no auto port forward, complications when building images, bugs where the socket isn’t cleared etc.
They will get there eventually but this is way over hyped for a sensible replacement for docker on Mac.
The nice thing is that even if it's not useful for you on your development system, your Docker work will still run on it the exact same elsewhere where it does work (hopefully).
There is a somewhat recent article from Red Hat on how to get it working on a Mac[1], but it seems somewhat convoluted in that a lot of stuff (virtualization) that Docker for the Mac apparently does for you is not automated yet.
Edit: Oh, I guess this announcement for 3.3.0 is more/better non-unix support? Is that the version you tried? Maybe they made some of this easier.
1: https://www.redhat.com/sysadmin/replace-docker-podman-macos
I do, but not for my workstation, where I do media production and a lot of other things that require a GUI and I use software that, unfortunately, doesn't run as well on Linux (if at all).
I have plenty of Linux servers, but if I wanted to do all my development work 'over the wire' I wouldn't use Docker Desktop either.
The bugs cited in the article are already fixed in the code, so I don't hold those against it.
I did have an issue that was my fault, encountered one genuine bug, and was disappointed to find that there's some work to be done before third-party tools can use it as a Docker replacement:
https://github.com/containers/podman/issues/11397
I agree that Podman is not yet a complete replacement for Docker on macOS, but the experiment was successful enough for me that I intend to try Podman instead of Docker on Linux servers.
Shameless plug: We built Simplenetes[1] around Podman (a simpler alternative to Kubernetes, but written in 100% shell script :)
How do you handle load balancing of inbound traffic.. do you use a pod running Traefik or similar? How do the pods communicate when they are deploying, unavailable or busy and so on?
I guess I could get this from your site but there's a LOT of information on the first pages there and possibly not what I am looking for.
We use haproxy pods for inbound traffic, they perform TLS termination and simply pass the traffic over to a local proxy pod.
The haproxy pod (and all other pods) communicate with each other over this local proxy service which is running on each host.
We have a very simplified and robust overall architecture, were each pod allocates a specific virtual port and the proxy will try each host for that pod (and remember status), meaning we don't need to keep track of global routing tables and update each host's ip tables (shivering) when pods come and go. We don't use iptables.
If some instance of a pod is unavailable the proxy will seamlessly try another instance of the pod.
Initially we tried to config haproxy to do all this proxying for us, but it was asking too much.
Good to know is that Simplenetes is still in beta.
I wonder what tools there are, currently (I noticed it's in beta), to get an overview of the state of the cluster, maybe what is talking to what, how much bandwidth they use etc (I don't know what one would need to know)
Also, most of it is not written in Bash, it's written in Posix standard, which is even more spartanic, but is then compatible with Dash and Ash (BusyBox) also, which is good because Bash is not always available.
To make Simplenetes we used another tool we also created which is meant for writing shell script apps and to perform agent-less automation, it is called Space.sh [1]
About tools for getting an overview of the cluster, there is only the command line tooling as for now, which does parts of the job, but tools for analyzing traffic and such is not created yet.
I wonder if there's a way to get notified when it's closer to stable, .. maybe following you on Twitter? I barely ever check Twitter though
[0]: https://github.com/lima-vm/lima [1]: https://github.com/containerd/nerdctl
I don't get the whole "Let's spend weeks rewriting our build infrastructure to save $240 a year" strategy going on here.
My company has over 3500 engineers. That's almost a million dollars in new spend and you gain basically nothing. It's a product the entire org has already been using for a long time, now you just have to pay a million dollars a year to use it.
I think that's because 2 things are at play here:
1. People are busy and often rarely read past the headline announcing the change, so they are unlikely to know of the caveat.
2. People don't like being taxed at the infrastructure level, so the default price for plumbing tools like Docker is $0. What's the assurance that the $240/yr will not rise to $2400/yr a year later, especially if they fail to hit their revenue numbers? Once the glove comes off, it's so much safer to cap your costs today by seeking out an alternative, rather than be confronted with uncapped costs (or any kind of vendor uncertainty) in the future.
We aren’t even big enough for this to affect us, it is just one more reason for us to look at how we can remove docker from our desktop infrastructure and replace it with something that makes us more efficient.
I had the same frustrations until the other day I saw a tip to enable 'Use the new Virtualization framework' under Preferences -> Experimental Features. Since then Docker's background CPU usage has dropped to 1% and I no longer bother stopping it when I'm not using it.
For people in .gov, that can be waiting a fiscal year to get it in the budget and slogging through paperwork to show that they’re in compliance:
So if your whole machine’s environment is already specified, why do you need to add Docker as another layer of abstraction? Is it simply to deal with needing to run multiple apps on the same underlying machine (defined by Nix) with possibly conflicting dependencies (those would be managed by Docker)?
Docker on the other hand doesn’t solve this problem itself - docker images can be built declaratively (eg. with nix itself), but it is more about managing the running of services with possibly different environments. A typical dockerfile will not be identical at all between two separate creations (most of the time it just installs programs from a repository without any versioning other than perhaps the distro’s major version)
The fact that we as in software developers use it for dev environment is just the unfortunate way it is.
You're not using Docker in development because of the sync issues on Mac between the host and the containers, but ideally you still want as much as possible the development and production environment parity (same versions of dependencies, etc). You can build your Docker images with Nix to ensure the dependencies versions you're developing with will be the same in prod.
I haven't used any of them, but maybe B.r. works with podman?
I like that with docker swarm one can without too much headache go from dev environments to a cluster. At university this seems much better than maintaining k8s. Podman seemed to be more interesting in rootless environments like HPC to me. I really what benefit switching to podman would actually bring to us.
It's worth noting if you have an M1 Mac, podman machine doesn't work on it yet. Docker Desktop does.
Of course, the easiest way to use containerization technology if that's what you want to do is just use Linux workstations and don't worry about needing a kernel virtualization layer between your workstation and your container engine at all.
Edit: Noting since sibling comments are discussing Windows and WSL that I'm talking specifically about replacing Docker Desktop on Mac because the article linked here is talking about using podman machine to replace Docker Desktop on Mac.
I don’t think it does. minikube probably comes closet. (But you could run podman instead of Docker inside minikube...)
https://docs.podman.io/en/latest/markdown/podman-machine.1.h...
Generally, if you publish an Open Source project, you cannot complain if people copy parts of it, even it's the command line arguments / API / whatever.