Hacker News Reader

Comment by jordemort | Hacker News Reader

Hi, author of the article, pleased to see it here! I'd really like to hear more from folks about how they're using LXC and/or LXD, and what they think their greatest strengths are compared to Docker or Kubernetes.

View on HN

We use LXC on physically distributed Proxmox nodes. It let's us easily launch new VMs and migrate around when we need to. Nothing we have is very complicated but it works well. We use SaltStack to tear up special debug containers with a bunch of integrated tools. To update Saltstack just downloads the new container and creates a new container in Proxmox.

I guess you could technically replace what we built with pure Docker but the Proxmox UI is a godsend for our on the ground support engineers who aren't the most technically savvy types.

We started going down the Kubernetes route initially but it quickly turned into an absolute nightmare for us and we gave up quickly and we're all pretty strong with Kubernetes. I guess though when all you need is a few stateful services running the "dynamic scaling" of kubernetes is kind of pointless.

Even though I am sure I am perfectly capable of figuring out the command line incantations to make these containers work, I still really appreciate the convenience of the Proxmox environment where it’s something I don’t have to think about.

I use it in a vanilla KVM host to have lightweight VMs for Remote Desktop environments (https://taoofmac.com/space/blog/2022/04/02/2130, https://taoofmac.com/space/blog/2022/04/12/2330). That way I can have full GNOME and XFCE desktops with different sets of developer tools installed, sharing the same projects tree on a bind mount, and to which I can remote via Xrdp (with audio, device sharing, the works). Like the two KVM Windows VMs running alongside, each LXC can have its own IP, outbound VPN connection and even Docker running as well, which is great.

A single i7 with 32GB of RAM can have at least 3 people working on it without hassle, like an old time-sharing system. Myself, I stuck a Raspberry Pi to a monitor (https://taoofmac.com/space/blog/2022/08/14/2030) and use it to access both GNOME and Windows desktops remotely.

All of this works solely with Ubuntu system packages, so I have no third party software to install (yes, ok, snap is a pain, but the host only runs KVM and LXD, nothing else, and all distros have xorgxrdp these days).

It’s like a lightweight Proxmox.

In the past, I used LXC to host several of my services on a single beefy Linode box, and I would go that way again today - I already have a similar setup inside Parallels on my M1 Mac, with one LXC per project.

I use Kubernetes within an LXD container with Btrfs backing storage. This isn't anything special. But it has two advantages. The first is that you could try out multi-node K8s clusters on a single system. The second is that the containers can be deleted and rebuilt easily without affecting the host system. Both are very useful when you're learning multi-node K8s. I plan to expand my homelab to true multi-node setup. However, I will probably retain K8s inside LXD. It is useful to run certain applications that demand full control of the system - like PiHole or MailInABox. There are probably better ways of hosting them. However, LXD gives me a lot of flexibility to experiment and make mistakes.

> The first is that you could try out multi-node K8s clusters on a single system.

Why can't you do this without BTRFS/LXD and just with Docker/OCI?

Yes, that's certainly possible - especially with tools like kind. However, the reason I prefer LXD over Docker/OCI is that the former behaves more like a full OS. LXD containers are system containers which are like VMs except for the lack of guest kernel. So it allows me to experiment with some K8s deployment configurations that's needed on a proper cluster.

> LXD containers are system containers which are like VMs except for the lack of guest kernel.

How is this different than a Docker/OCI container/cgroup thingie?

Do you have instructions (e.g., tutorial or documentation) on what needs to be done to have this setup? There are many configurations that would need to make it work properly. Thanks!

I'm using LXC just to have a painless way to have a second SSH server (for SFTP) running on the computer that has different usernames and passwords.

Literally all that is installed is Alpine Linux and the SSH server. There's a symbolic link to get to the shared files.

Similar use case for me. But instead of SSH/SFTP I was running Caddy with the file_server directive!

I've since switched to podman because my installation on Alpine Edge kept breaking on updates and I don't want to use Snap. Someone mentioned OpenSuse had it in their package manager and its been stable. Maybe I'll check that out in the future.

I use lxd for running core services at home, such as samba/sftp/webdav or home automation. I find it easier to attach a vpn to the container, which combined with a nat traversing sdn/mesh gives me easy, private, encrypted access to data or services. Also easy to pass a usb device (say zwave controller) to a container. I run it over a btrfs array and have a workflow to snapshot containers and send to a backup host. Using unique namespaces for each unprivileged container, I am reasonably confident that a security oops in one container should be isolated.

I also use lxd on vps hosts, often nesting docker to give more granular control over networking.

LXC is how Chromebooks let you install Debian or Ubuntu or Arch in a container, with graphical applications running in the container integrated into the ChromeOS UI.

I basically use LXC for light-weight VMs (a name server here, a simple web server there, pi-hole on my home router). Where I care about security separation (my containers are privileged) I use real VMs via KVM.

Though I'm an old fashioned pets-not-cattle type. Docker and friends have the disadvantage of being invented (or becoming practical/popular) in/after my mid 30s!

i introduced proxmox and saltstack to a customer who was running all sorts of applications together on a few linux servers. splitting up the services over multiple containers made maintenance easier because upgrading software in one container would not affect apps running in other containers.

on my own servers i use plain LXC without proxmox, but created and managed through saltstack for similar reasons. and also to allow me to use different linux distributions for different use cases, and to seperate various development environments

The best futuristic feature by far I've heard of is live-migration of workloads between hosts (read: your compute and memory gets transferred to another VM and resumes without having to restart). This would be amazing for, e.g. auto-scaling workloads when they reach their memory limit. Or to run them on a Spot instance and then seamlessly migrate to another, when the current one is on the verge of preemption.

It's too bad no one appears to be working on that in OCI/Kubernetes world.

Checkpoint/restore is supported by runc and there were demos of live migrating Docker containers several years ago. A better question is why Kubernetes isn't working on integrating it (though I suspect there's a KEP for it somewhere).

Could you share a link please? I've only seen similar demos of a Doom game migration on LXD (I believe).

EDIT: very interesting, thanks TIL! Here's at least someone from RH playing with it in the context of K8s: https://youtu.be/DDJxqV98b4U?t=1932

EDIT2: and apparently, at least its minimal implementation has been merged: https://github.com/kubernetes/kubernetes/pull/104907

Adrian is one of the CRIU folks and has been working on integrating CRIU and containers for a while.

I have a proxmox system that I host lxcs on to serve as development environments for various web apps or network services that have different Linux based tech stacks. I also run some services inside lxcs on my Linux based router. Local network name services, an NFS volume, etc. The strength is the simplicity. I found it all easy to set up and very low maintenance.

I've been considering building a system to spin up containers to run repeatable statistical analyses. LXC was the direction I decided to go.

Reply on news.ycombinator.com