Adventures in upgrading Proxmox | Hacker News Reader

Adventures in upgrading Proxmox

blog.vasi.li

60 pointsspeckx57 comments

> As an aside... Because one node didn't start, and my Proxmox cluster has only two nodes, it can't reach quorum, meaning I can't really make any changes to my other node, and I can't start any containers that are stopped. I've recently added another Zigbee dongle, that supports Thread, and it happens to share same VID:PID combo as the old dongle, so due to how these were mapped into guest OS, all my light switches stopped working. I had to fix the issue fast.

Lesson in here somewhere. Something about about a toaster representing the local intelligence maxima?

Lesson 1: clusters should have an odd number of nodes.

I really, really think there are better lessons there. Maybe more like "Lesson 0. Don't put distributed clusters in control of your light switches"

Why not?? It's fun!

Yes, but then I'm going to have to manually go around my house and turn off all the lights when I leave the house and when I come back I have to turn them on manually instead of them just turning on when I open the door to a room. Also my AC/heating automatically turns off when I leave the house and turns on when I come back, my lights automatically dim/change to a warmer temperature in the evening as it gets closer to bed time, my desktop goes to sleep when I leave the house, my TV automatically turns on when I power on the living room media PC, etc. etc.

Two node / even node clusters can work fine.

For even n>2 you define a tie breaker node in advance and only the partition connected to that node can make a quorum at 50%. For n=2 going from no quorum to quorum requires both nodes but losing a node doesn't lose quorum, and when you lose a node you stop, shoot the other node, and continue. For split brain the fastest draw wins the shootout.

> For split brain the fastest draw wins the shootout.

I bet there is still space for a race condition there.

In fairness to proxmox, that's the recommended way.

Most homelabbers ignore recommendations because if anything breaks nothing of corporate value is lost and no one's gonna lose their job.

Originally I was planning on building the NAS with just the Minisforum MS-01, but truenas and USB enclosures do not play well together.

So I went for the AOOSTAR NAS mini-pc as a "proper" solution. Ended up with two machines, so why not join them into the cluster!

Probably can chuck proxmox on a RasPi somewhere, just for quorum purposes :)

proxmox even makes it easy by letting you run something like a raspberry pi as an additional quorum member if you dont have enough hardware for a 3rd node

The lesson is use dumb light switches and have a shotgun ready if the printer starts to act up.

Also regularly print out sheets of electronic recycling facts to remind the printer of its place.

I see you made the mistake of buying an hp inkjet in the last twenty years as well...

At least I was laughing at the Cloudflare oopsie, since all my light switches (et al) are all local. Unlike those people with a fancy smart bed that went into a W shape because it couldn't talk to AWS.

Yup, if you're going to have smart lights, get ones that still have a physical switch!

Just adding my anecdata. I started using ESXi around 2016 for homelab purposes. This paid off nicely as my $CORP also uses VMWare products, so it was easy and natural to get into VMs and fix them up as needed.

Pretty unhappy about the Broadcom acquisition as VMWare previously seemed to care about the homelab market and recognized the value created by letting hobby admins learn their tools. That's going away now and Broadcom hopes the locked in market will provide enough income to suit their short term goals.

Anyway, I switched to Proxmox 9 in my lab this year and couldn't be happier. It's so much more powerful, flexible, and intuitive after one spends a few days in the UI. I ported all my VMs over and things are rock solid and better than ever under Proxmox.

My local systems team at $CORP knows of Proxmox but acknowledges nothing will be changing soon with our environment. But time will tell I suppose.

Btw the issue that the author encountered are not really with proxmox itself but with an out-of-tree kernel driver they installed.

Any debian system (proxmox is based on debian) would have broken in a similar (if not the exact same) way.

Not to mention, Proxmox does not support running Docker in an LXC officially (of course many users still do it). It is not a supported configuration as of now

best to use a vm

Rightfully so, running docker in an lxc container is quite a dumb idea. Have you tried podman instead?

> Running docker inside LXC is weird.

Knowing when to use a vm and when to use a container is sometimes an opaque problem.

This is one of those cases where a VM is a much better choice.

This seems like a niche issue, running Docker in LXC for years with dozens of images without a problem.

Guessing you are only running a single node though, not a cluster with HA and live migration and all that.

Am I crazy or is converting a dockerfile into LXC something that should be possible?

In the new Proxmox VE 9.1 release this should be possible, from the changelog:

> OCI images can now be uploaded manually or downloaded from image registries, and then be used as templates for LXC containers.

It should in an ideal world but docker is a very leaky abstraction imho and you will run into a number of problems.

It has improved as of newer kernel and docker versions but they were problems (overlayfs/zfs incompatibilities/ uid mapping problems in docker images/ capabilities requested by docker not available in LXC, rootless docker problems,...)

I can't speak for the author, but they said they have a Coral TPU passed into the LXC & container, which I also have on my Proxmox setup for Frigate NVR.

Depending on your hardware platform, there could be valid reasons why you wouldn't want to run Frigate NVR in a VM. Frigate NVR it works best when it can leverage the GPU for video transcoding and TPU for object detection. If you pass the GPU to the VM, then the Proxmox host no longer has video output (without a secondary GPU).

Unless you have an Intel Arc iGPU, Intel Arc B50/B60, or fancy server GPU, you won't have SR-IOV on your system, and that means you have to pass the entire GPU into the VM. This is a non-starter for systems where there is no extra PCIe slot for a graphics card, such as the many power-efficient Intel N100 systems that do a good job running Frigate.

The reason why you'd put Docker into LXC is that's the best supported way to get docker engine working on Proxmox without a VM. You'd want to do it on Proxmox because it brings other benefits like a familiar interface, clustering, Proxmox Backup Server, and a great community. You'd want to run Frigate NVR within Docker because it is the best supported way to run it.

At least, this was the case in Proxmox 8. I haven't checked what advancements in Proxmox 9 may have changed this.

I have Frigate and a Coral USB running happily in a VM on an N97. GPU pass through is slightly annoying (need to use a custom ROM from here: https://github.com/LongQT-sea/intel-igpu-passthru). I think SRIOV works but haven’t tried. And Coral only works in USB3 mode if you pass the whole PCIe controller.

I've been debating if I should move my frigate off an aging Unraid server to spare mini PC with Proxmox. The mini has a N97 with 16gb ram. How cameras do you have in your frigate instance on that N97? Just wondering if a N97 is capable of handling 4+ cameras. I do have a Coral TPU for inference & detection.

I have around 6 cameras, mostly 1080p, and about 8 GB RAM and 3 cores on the VM (plus Coral USB and Intel VAAPI). CPU usage is about 30 - 70% depending on how much activity there is. I also have other VMs on the machine running container services and misc stuff.

There are some camera stability issues which are probably WiFi related (2.4 GHz is overloaded) and Frigate also has its own issues (e.g. with detecting static objects as moving) but generally I’m happy with it. If I optimize my setup some more I could probably get it to a < 50% utilization.

Perfect thanks. I'll give the N97 a go and put it to good use as a dedicated frigate NVR box. It certainly has a much lower power draw than my Unraid server.

I'm running thingino cameras off wifi and the stability is kinda meh... Want to try a wired setup with a PoE USB Ethernet adapter...

My Thingino camera seems to have some RTSP issues, which is a shame because I’d like to use it on more devices.

2.4 GHz sucks though. Wish my mesh allowed me to use multiple 2.4 GHz channels concurrently or per node.

I might just succumb and do the prudynt restart by cron every once in a while as this does seemingly fix it.

> Unless you have an Intel Arc iGPU, Intel Arc B50/B60, or fancy server GPU, you won't have SR-IOV on your system, and that means you have to pass the entire GPU into the VM.

This is changing, specifically on QEMU with virtio-gpu, virgl, and Venus.

Virgl exposes a virtualized GPU in the guest that serializes OpenGL commands and sends them to the host for rendering. Venus is similar, but exposes Vulkan in the guest. Both of these work without dedicating the host GPU to the guest, it gives mediated access to the GPU without any specific hardware.

There's also another path known as vDRM/host native context that proxies the direct rendering manager (DRM) uAPI from the guest to the host over virtio-gpu, which allows the guest to use the native mesa driver for lower overhead compared to virgl/Venus. This does, however, require a small amount of code to support per driver in virglrenderer. There are patches that have been on the QEMU mailing list to add this since earlier this year, while crosvm already supports it.

To add to this, while I haven’t used it yet myself (busy with too many other projects), this gist has the clearest and most up to date instructions on setting up QEMU with virglrenderer that I’ve found so far, with discussion on current issues: https://gist.github.com/peppergrayxyz/fdc9042760273d137dddd3...

This is great news and I'm looking forward to these exciting changes in QEMU.

At first I had the unholy abomination that is Frigate LXC container, but since it's not trivially updatable and breaks other subtle things, I ended up going with Docker. Was debating getting it into a VM, but for most part, docker on LXC only gave me solvable problems.

The way I understand it is that Docker with LXC allows for compute / resource sharing, where as dedicated VMs will will require passing through the entire discrete GPU. So, the VMs require a total passthrough of those Zigbees, container wouldn't?

I'm not exactly sure how the outcome would have changed here though.

Proxmox FAQ calls running Docker on LXC a tech preview and “kind of” recommends VMs. At the very bottom of the page.

https://pve.proxmox.com/wiki/FAQ

> While it can be convenient to run “Application Containers” directly as Proxmox Containers, doing so is currently a tech preview. For use cases requiring container orchestration or live migration, it is still recommended to run them inside a Proxmox QEMU virtual machine.

It's not always better. Docker on lxc has a lot of advantages. I would rather use plain lxc on production systems, but I've been homelabbing on lxc+docker for years.

It's blazing fast and I cut down around 60% of my RAM consumption. It's easy to manage, boots instantly, allows for more elastic separation while still using docker and/or k8s. I love that it allows me to keep using Proxmox Backup Server.

I'm postponing homelab upgrade for a few years thanks to that.

Related ongoing thread:

Proxmox virtual environment 9.1 available - https://news.ycombinator.com/item?id=45980005 - Nov 2025 (56 comments)

Man Proxmox... I love it, I use it, but I swear there has to be a more straightforward way to implement this technology.

I ran into the same issue over the weekend. The end-goal for my Proxmox setup is basically the same deployment you have. It's good to see the issue was addressed quickly by the community.

I recently gave up on Proxmox for my home lab needs after a failed upgrade from 8 to 9. I also never liked the feeling of not having an easy to use API.

Ive put off that upgrade as I just dont have the time to fix it if goes sideways. What did you end up moving to?

Depending on what your needs are, have a look at Incus-OS. While recently released and with lot of stuff “still in progress”, it’s something to keep an eye on. Even comes with ZFS:

https://linuxcontainers.org/incus-os/

https://github.com/lxc/incus-os

Question: did you run the pve8to9 script? Read their extensive documentation [0]on how to upgrade? Fix the stuff aforementioned script comes up with?

My cluster went from 6 to 7, 7 to 8 and recently 8 to 9, along with Ceph - all without a single problem.

Given it’s more or less Debian underneath, not too surprising I’d say? Granted, there’s always a chance for something to go sideways, however, it’s unlikely you’re the first person to encounter this problem and if you check their forums, you should find a solution.

[0] https://pve.proxmox.com/wiki/Upgrade_from_8_to_9

Look ma, I'm on the TV. The merch is in the back, sub to the YouTube channel...