Hacker News Reader

I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).

I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).

I've basically gone full circle at this point:

- Docker

- LXD with Bash scripts

- LXD with a ton of Python

- NixOS

- Proxmox with Ansible

- Proxmox with Bash scripts (albeit much simpler and flatter than last time)

Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.

One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.

Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.

Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.

View on HN

DATA_SIZE=600GiB # Switch to free mode rm /etc/apt/sources.list.d/ceph.list rm /etc/apt/sources.list.d/pve-enterprise.list echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list # Setup locale sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen locale-gen update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 apt update apt dist-upgrade -y apt install -y software-properties-common add-apt-repository -y non-free apt install -y vainfo intel-media-va-driver-non-free lm-sensors apt install -y ansible pip python3-proxmoxer htop strace git # Setup user data area lvcreate -V ${DATA_SIZE} -T pve/data -n home_data mke2fs -t ext4 -O ^has_journal /dev/pve/home_data mkdir /home/data echo "/dev/pve/home_data /home/data ext4 rw,discard,relatime 0 2" >>/etc/fstab mkdir -p /mnt/containers/media echo "//media/media /mnt/containers/media cifs uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab mkdir -p /mnt/containers/files echo "//files/files /mnt/containers/files cifs uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab systemctl daemon-reload mount -a

# ============ # Local config # ============ TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst" INSTANCE_CT=200 INSTANCE_NAME=template-ubuntu INSTANCE_MEMORY=2048 # ====== # Script # ====== pveam download local ${TEMPLATE_IMAGE} || true pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --rootfs local-lvm:1 \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \ --ostype ubuntu \ --start 1 \ --timezone host \ --features nesting=1 \ --unprivileged 1 # Use a fast mirror pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list # Set up locale pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen pct exec $INSTANCE_CT -- locale-gen pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8 # Bring everything up to date pct exec $INSTANCE_CT -- apt clean pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y pct exec $INSTANCE_CT -- apt install -y software-properties-common curl # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT

# ============ # Local config # ============ TEMPLATE_CT=200 INSTANCE_CT=201 INSTANCE_NAME=template-plex INSTANCE_MEMORY=2048 # ====== # Script # ====== passthrough_gpu() { local instance_id="$1" echo 'lxc.cgroup2.devices.allow: c 226:0 rwm lxc.cgroup2.devices.allow: c 226:128 rwm lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128" ' >> /etc/pve/lxc/${instance_id}.conf } pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct resize $INSTANCE_CT rootfs 2G pct set $INSTANCE_CT \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 passthrough_gpu $INSTANCE_CT pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y # Install Plex apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main" pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt install -y plexmediaserver # Turn this into a template pct stop $INSTANCE_CT pct template $INSTANCE_CT

# ============ # Local config # ============ TEMPLATE_CT=201 INSTANCE_CT=20001 INSTANCE_NAME=plex INSTANCE_ADDRESS=99 INSTANCE_MEMORY=2048 # =========== # Host config # =========== HOST_DATA=/home/data HOST_BASE_UID=100000 HOST_BASE_GID=100000 # ====== # Script # ====== pct clone $TEMPLATE_CT $INSTANCE_CT --full 1 pct set $INSTANCE_CT \ --onboot 1 \ --hostname ${INSTANCE_NAME} \ --memory ${INSTANCE_MEMORY} \ --net0 name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0 # Mount the media dir pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media # Get the UID and GID of the plex user pct start $INSTANCE_CT PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex) PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex) PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID)) PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID)) pct stop $INSTANCE_CT # Mount the Plex config dir mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME} chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver pct start $INSTANCE_CT pct exec $INSTANCE_CT -- apt update pct exec $INSTANCE_CT -- apt dist-upgrade -y IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs) set +x echo "===========================================================" echo "FOR FIRST TIME SETUP (no existing config):" echo echo "Go to https://www.plex.tv/claim/" echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'" echo "Then, go to ${IP_ADDR}:32400" echo "==========================================================="