Hacker News Reader

Comment by buybackoff | Hacker News Reader

LXC via Proxmox is great for stateful deployments on baremetal servers. It's very easy to backup entire containers with the state (SQLite, Postgres dir) to e.g. NAS (and with TrueNAS then to S3/B2). Best used with ZFS raid, with quotas and lazy space allocation backups are small or capped.

Nothing stops one from running Docker inside LXC. For development I usually just make a dedicated priviledged LXC container with nesting enabled to avoid some known issues and painful config. LXC containers could be on a private network and a reverse proxy on the host could map to the only required ports, without thinking what ports Docker or oneself could have accidentally made public.

View on HN

We do something similar with btrfs as the filesystem. There have been some issues with btrfs itself, but the LXC side of this has worked pretty good. Any significant storage (such as project directories) is done with a bind mount into the container, so that it is easy to separately snapshot the data or have multiple LXC containers on the same host access the same stuff. That was more important when we were going to run separate LXC containers for NFS and Samba fileservers, but we ended combining those services into the same container.

Good comment. It was a revelation to me when I used Proxmox and played with LXCs. Getting an IP per container is really nice.

It's an annoying that you can only make snapshots on a stopped container. With VMs it works in a running VM.

Also can't do live migrations or backups and moving storage around is a headache.

We've pretty much stopped using LXC containers in Proxmox because of all the little issues.

Dumb question, but what have you replaced it with? I’ve been working on setting up Proxmox/LXC on a box at home as we speak — but if it’s not worth the headache I’ll stick to VMs.

I'm not who you asked but I use them for different purposes. There are some quirks here and there. Never anything that won't run, but it could potentially add extra troubleshooting.

Things that got me recently, just off the top of my head:

- htop sees memory usage incorrectly

- Docker tries to use overlay2 on ZFS which fails (I think, I needed to create and mount an ext4 volume for reasons)

- Hashicorp Vault needed disable_mlock because I believe LXC blocks the syscall.

On the other hand, I like my Samba file server in a container though, because it is much easier to share storage from the host into LXC than a VM.

LXC and VMs both have pros and cons.

Weird, I just tested on my proxmox instance and I was able to create a snapshot of a running container (PVE 7.1-10)

Yeah, I haven't got these issues either.

That highly depends on the underlying storage. If it is something that supports snapshots (ZFS, Ceph, LVM thin) then it should work fine, also backups will be possible without any downtime as they will be read from a temporary snapshot.

Even with ZFS you still have to wait for RAM to dump, haven't you? And it will freeze at least for the dump write time. Do they have CoW for container memory?

But even if they had, the RAM snapshot needs to be written, but without freezing the container. I would appreciate an option when I could ignore everything that was not fsyned, e.g. Postgres use case. In that case the normal ZFS snapshot should be enough.

RAM and other state can be part of a snapshot for VMs, in which case the VM will continue right where it was.

The state of a container is not part of the snapshot (just checked), as it is really hard to capture the state of the container (CPU, network, all kinds of file and socket handles) and restore it because all an LXC container is, is local processes in their separate cgroups. This is also the reason why a live migration is not really possible right now, as all that would need to be cut out from the current host machine and restore in the target machine.

This is much easier for VMs as Qemu offers a nice abstraction layer.

> LXC via Proxmox is great for stateful deployments on baremetal

Reminds me of (now defunct?) flockport.com

They had some interesting demos up on YouTube, showcasing what looked like a sandstorm.io esque setup.

You've basically described my homelab set up here.

Proxmox, a few LXC, each with their own containerisation running.

Reply on news.ycombinator.com