LXC and LXD: a different container story

When LXD was running, I found that it feels like something between a VM and OCI container. In the whole pets vs cattle analogy, LXD containers definitely feel more like pets.

On the host, concepts are similar to OCI containers (networking, macvlan, port mapping, etc). I like its approach to building up containers with several commands instead of one long command. You can add and remove ports, volumes, and even GPUs to running containers. It doesn't have compose files, I just used bash scripts to create reproducible containers.

Inside the LXD container is where things get different. You're containerizing a whole OS, not an application. So you end up administering your container like a VM. You jump inside it, update its packages, install whatever application and dependencies you need. It doesn't work like a glorified package manager (which is how most homelabbers use Docker). As a long time VM user I actually prefer installing things myself, but I can see this turning away many people in the selfhosting community.

I liked LXD and would've kept using it if it weren't so intimately linked to Snap. Last month it failed to start my containers after a system update...again. So I moved that box over to Rocky with Podman containers running as SystemD units. I do kinda miss having fast, throwaway OSes on that server for experiments (sometimes its nice to have pets).

I also use it to host a Jitsi instance. Jitsi is rather picky about the underlying distribution.

All this without the overhead of an actual virtual machine of course.

The only pain I have with LXC is its behavior when errors occur. It’s not easy to tell what’s wrong, the error messages are worthless. Sometimes the debug log can help, sometimes not so much.

This sort of mediation makes sense and works great for ephemeral containers in Kubernetes. For full virtual servers, LXD is the worst of both worlds.

LXD is a treat to work with, and I feel its container model is perfect for home/small business servers that need to run lots of third-party software, much of which would work poorly in a "proper" stateless container.

I still use lxd on an Ubuntu 18.04 host that still has the deb instal but looking forward I’m trying to get things migrated to systemd-nspawn.

Still Dockerfiles and all the magic that goes with them was just so much easier to pick up and convince others on the team to try.

The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built on ubuntu hosts and (for some stupid reasons) centos LXC containers running on top of those.

LXC is a good system and all, but what we were doing with it was a total nightmare, we were basically trying to reinvent kubernetes using traditional configuration management and LXC with two flavors of linux and a million dependencies.

I think that feature combined with overlayfs2 is quite useful, despite my many criticisms of Docker.

It is sort of a middle ground between Nix like granularity which requires rewriting upstream, and big LXC blobs created with shell scripts.

Although I also think we need some kind of middle ground between docker and nix :)

One trend with docker that I personally don't like is that a lot of projects prefer docker-compose over regular Dockerfiles (though some of them support both), and this leads to a lot of bloat with how many containers it takes to run one application and duplication where each app will need its own database, redis, webserver, etc. containers.

That's not a problem when you are at the scale to need that kind of orchestration and have the resources to run that many containers, but personally I would rather have one database host that all my containers can talk to and one reverse proxy/webserver to make them accessible to make better use of resources and get better density on one host.

One downside with lxc is that there's been some fragmentation with the image creation side of it, I had been used to using lxc-templates for years, which are just shell scripts for bootstrapping and configuring your container and pretty common between distros. I found a bug with creating containers for the latest version of Alpine that I fixed, and only then did I find out that lxc-templates are deprecated and essentially unmaintained, and that now distrobuilder is the preferred way to build lxc containers, at least per Canonical. That seems to be another Canonical-ism, since the distrobuilder docs say that snap is the only way to install it unless you build from source, so that's a huge no from me, and I also didn't feel like learning the yaml config for it.

I was actually considering moving my docker daemon into an unprivileged lxc container like the article mentions, but haven't gotten around to it.

Have a few apps running in plain LXC containers (like AdguardHome) but maintenance is non-free (unlike a docker-compose stack with Watchtower keeping everything nice and fresh).

Linux folks love saying: use the right tool for the right job when it comes to telling people why Kubernetes is a strong enough reason for any company to switch to linux, however when it comes to things such as lightweight containers (or ZFS-on-root for the matter) for some reason the same reasoning isn't applied.

LXC and LXD look like a patchwork compared to FreeBSD jails. If you are on the desktop and want to run graphical apps, maybe it can make some kind of sense if you are already using Linux for everything else. But if you want to have a lightweight container setup on a headless server, FreeBSD is by far the best option.

In FreeBSD land, no one boasts of running 5 virtualized environments on a single machine because this is just a common thing for us and part of every workfow. You want to work on some github repository? You create a new jail in 1 minute using a tool such as cbsd.

A server with 32gb of RAM can easily run 1000s of jails, even a small VPS with 1-2gb of RAM can run a dozen jails or more without problems. It is intriguing that so many people dismiss the power of that.

And it is intriguing that people easily accept: "there is no equivalent to dockerfiles but it's fine because things can be scripted easily with basic shell commands" when it comes to LXC/LXD but people dismiss this very argument when FreeBSD folks try to explain why there is no need for Kubernetes on FreeBSD.

Anyway, if you run a startup not operating a resource intensive products (like ML requiring spinning up massive amounts of cores and RAM in an irregular basis) FreeBSD and jails can really save massive amounts of time and allow you to focus on the right thing. just run everything from a single Root VPS (usually hosting companies already handle things such as RAID replication and hardware redundancy transparently) and before you reach scalability problems, you'll already be making millions of dollars in revenue.

Even after, running thousands of jails on a single hosy essentially means doing what AWS and other cloud providers are doing. This doesn't prevent you from builing a small high-availability layer on top to ensure that critical apps run from at least two different physical hosts. This isn't very difficult to build with things like Consul, HA proxy, built-in system utilities and a small custom web interface.

Even my mail server runs in LXC, and it is a direct descendant of a Red Hat 5 server from ~20 years ago!

I've often seen LXD vs Docker described as LXD being better suited to running distros and distro-like environments, with Docker being better suited to single-application environments.

So that's what I do. For many of dev and testing environments, there's an LXD container, which I use over SSH/rsync daily. Compiles, long-running tests, bandwidth heavy network actions and so on are done in those containers, on a few fast servers in data centres. It's much faster than my laptop.

One motivation for running Linux in LXD containers instead of on bare metal was to allows me to decouple changes to host OS version and networking from the containers in which I do most my work. Previously I used bare metal, i.e. ran Linux on servers, but found it annoying that I couldn't update the host OS and especially the kernel or disto major version, without shutting down everything in Screen sessions and long-lived networking tests that are not so easy to shut down and restart (without a rewrite anyway).

Being containers, they can use host filesystems and devices almost directly. So they are great for things like I/O performance tests, with confidence that it's basically testing host performance and behaviour.

Unfortunately, after getting deep into both those things, I found LXD wasn't quite as run-a-distro friendly as it first appeared, and it wasn't as reliable at replicating host I/O performance either.

Doing file I/O with "shift=true" host-filesystem mounts in LXD (the most sensible mode) turns out to be have very much slower O_DIRECT performance than it should, so that screwed up my database storage tests unexpctedly until I realised. Now I run low-level storage performance tests outside LXD, because I don't trust it.

As for distro-like environments, I eventually found stateful snapshot+restore or migation of dev and test environments is permanently broken. No LXD container I've used over many years has ever successfully been able to live-snapshot/migrate, without hitting an error which prevents it. This isn't some obscure bug, either. It's never worked, and my browsing of forums and issue trackers leads to the view that it's not actually expected to work for almost any distro-like environment, despite being one of the headline features.

As a result, one of the main factors motivating using LXD instead of bare metal for me turned out not to work. It's always possible to shut down a container, but that loses so much state, long-lived Screen sessions, running processes and so on that it's about as disruptive as updating bare metal. I.e. no real advantage.

I could switch back to VMs, which are excellent for snapshotting and migration, as I used to use (with libvirt+kvm) but they have a different problem for much of my work: Host filesystem/blockdev sharing is relatively slow. Not only could I not use them to measure performance against various kernels on real hardware (it would be measuring the VM as much as anything), I also run many data-intensive jobs, and the closer I can get to those host storage devices, the better.

The last reason it's felt buggy and rough is when removing a host-container filesystem mount from a container, it has often deleted the mount point on the host as well, disrupting other processes using it. Operations on the container are not supposed to change the host itself. You get used to working around this, but it's annoying.

Ah well, nothing's perfect. It's still a very useful tool, with some rough edges. I'd look into fixing the I/O performance and host-unmounting bug if I thought the snapshot/migration feature would be made to work someday, but because that looks unlikely, plus issues that came with Snap, I'm not as motivated and will live with hackish workarounds.

For things like my mail, web, SpamAssassin and other services, LXD has been great, and the issues I encountered aren't really a problem. This is the sort of thing which Docker is pretty good for as well. However, due to history I've tended to keep servers going for a long time (with my mail server winning the crown as it's an image that's been gradually modified and upgraded for about 20 years, across many different ways of running Linux in container-like environments, starting with chroot). LXC/LXD is better than Docker for this use case, though either can be made to work.

Does anyone have a use case where they couldn't use Docker? I'm sure they exist but the list must be tiny. Listed in the article is:

  - run systemd in a container
    - why?? does your system not have systemd or equivalent?
    - bad design; how will you monitor what's running and what's not, or upgrade an individual service in this container? restart everything?
    - apparently systemd can run in docker?? https://medium.com/swlh/docker-and-systemd-381dfd7e4628
  
  - run lxc/docker in a container
    - docker-in-docker