Hacker News Reader

Comment by judofyr | Hacker News Reader

What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.

View on HN

I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).

I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).

I've basically gone full circle at this point:

- Docker

- LXD with Bash scripts

- LXD with a ton of Python

- NixOS

- Proxmox with Ansible

- Proxmox with Bash scripts (albeit much simpler and flatter than last time)

Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.

One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.

Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.

Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.

As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed:

Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script?

Also I assume you have a script to set up the (Proxmox) host machine?

I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.

Edit: I've made the repo public: https://github.com/kstenerud/proxmox-containers

Here's one of my host setup files (run this immediately after installing Proxmox from the iso):

    DATA_SIZE=600GiB

    # Switch to free mode
    rm /etc/apt/sources.list.d/ceph.list
    rm /etc/apt/sources.list.d/pve-enterprise.list
    echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list

    # Setup locale
    sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
    locale-gen
    update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8

    apt update
    apt dist-upgrade -y
    apt install -y software-properties-common
    add-apt-repository -y non-free
    apt install -y vainfo intel-media-va-driver-non-free lm-sensors
    apt install -y ansible pip python3-proxmoxer htop strace git

    # Setup user data area
    lvcreate -V ${DATA_SIZE} -T pve/data -n home_data
    mke2fs -t ext4 -O ^has_journal /dev/pve/home_data
    mkdir /home/data
    echo "/dev/pve/home_data    /home/data   ext4    rw,discard,relatime   0    2" >>/etc/fstab
    mkdir -p /mnt/containers/media
    echo "//media/media  /mnt/containers/media cifs  uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab
    mkdir -p /mnt/containers/files
    echo "//files/files  /mnt/containers/files cifs  uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab
    systemctl daemon-reload
    mount -a

The host is kept VERY simple.

Here's my basic Ubuntu container template:

    # ============
    # Local config
    # ============

    TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst"

    INSTANCE_CT=200
    INSTANCE_NAME=template-ubuntu
    INSTANCE_MEMORY=2048

    # ======
    # Script
    # ======

    pveam download local ${TEMPLATE_IMAGE} || true

    pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \
        --hostname ${INSTANCE_NAME} \
        --memory   ${INSTANCE_MEMORY} \
        --rootfs   local-lvm:1 \
        --net0     name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \
        --ostype   ubuntu \
        --start    1 \
        --timezone host \
        --features nesting=1 \
        --unprivileged 1

    # Use a fast mirror
    pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list

    # Set up locale
    pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
    pct exec $INSTANCE_CT -- locale-gen
    pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8

    # Bring everything up to date
    pct exec $INSTANCE_CT -- apt clean
    pct exec $INSTANCE_CT -- apt update
    pct exec $INSTANCE_CT -- apt dist-upgrade -y
    pct exec $INSTANCE_CT -- apt install -y software-properties-common curl

    # Turn this into a template
    pct stop $INSTANCE_CT
    pct template $INSTANCE_CT

Here's my Plex template (with GPU passthrough):

    # ============
    # Local config
    # ============

    TEMPLATE_CT=200
    INSTANCE_CT=201
    INSTANCE_NAME=template-plex
    INSTANCE_MEMORY=2048

    # ======
    # Script
    # ======

    passthrough_gpu() {
        local instance_id="$1"
        echo 'lxc.cgroup2.devices.allow: c 226:0 rwm
    lxc.cgroup2.devices.allow: c 226:128 rwm
    lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
    lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128"
    ' >> /etc/pve/lxc/${instance_id}.conf
    }

    pct clone $TEMPLATE_CT $INSTANCE_CT --full 1
    pct resize $INSTANCE_CT rootfs 2G
    pct set $INSTANCE_CT \
        --hostname ${INSTANCE_NAME} \
        --memory   ${INSTANCE_MEMORY} \
        --net0     name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0

    passthrough_gpu $INSTANCE_CT

    pct start $INSTANCE_CT

    pct exec $INSTANCE_CT -- apt update
    pct exec $INSTANCE_CT -- apt dist-upgrade -y

    # Install Plex
    apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D
    apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main"
    pct exec $INSTANCE_CT -- apt update
    pct exec $INSTANCE_CT -- apt install -y plexmediaserver

    # Turn this into a template
    pct stop $INSTANCE_CT
    pct template $INSTANCE_CT

Here's the Plex instance script (starts a Plex instance going from the Plex template):

    # ============
    # Local config
    # ============

    TEMPLATE_CT=201
    INSTANCE_CT=20001
    INSTANCE_NAME=plex
    INSTANCE_ADDRESS=99
    INSTANCE_MEMORY=2048

    # ===========
    # Host config
    # ===========

    HOST_DATA=/home/data
    HOST_BASE_UID=100000
    HOST_BASE_GID=100000

    # ======
    # Script
    # ======

    pct clone $TEMPLATE_CT $INSTANCE_CT --full 1
    pct set $INSTANCE_CT \
        --onboot   1 \
        --hostname ${INSTANCE_NAME} \
        --memory   ${INSTANCE_MEMORY} \
        --net0     name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0

    # Mount the media dir
    pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media

    # Get the UID and GID of the plex user
    pct start $INSTANCE_CT
    PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex)
    PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex)
    PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID))
    PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID))
    pct stop $INSTANCE_CT

    # Mount the Plex config dir
    mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver
    chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}
    chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver
    pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver

    pct start $INSTANCE_CT
    pct exec $INSTANCE_CT -- apt update
    pct exec $INSTANCE_CT -- apt dist-upgrade -y

    IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs)
    set +x
    echo "==========================================================="
    echo "FOR FIRST TIME SETUP (no existing config):"
    echo
    echo "Go to https://www.plex.tv/claim/"
    echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'"
    echo "Then, go to ${IP_ADDR}:32400"
    echo "==========================================================="

Maybe chuck that on GitHub/gist (or some other online repo) so people can potentially suggest stuff in an easy way? :)

Hmm OK. I already had it on github so I'll just make it public:

https://github.com/kstenerud/proxmox-containers

Thank you. I've been meaning to try out proxmox.

Proxmox really needs to get their cloud-init story together. If they supported cloud-init for LXC a lot of automation and setup issues would just go away…

Funny, I can imagine bash is a real improvement on ansible. I once started with bash, switched to bash in ansible and then to "good style" ansible and then to a nixops cluster.

There wasn't really a good point between bash and nix.

I'll echo this. I can't imaging going back.

And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.

Reply on news.ycombinator.com