Migrating from Proxmox to NixOS and Incus
nijho.lt
nijho.lt
Roughly ten years ago, my homelab consisted of a dozen virtual machines running on SmartOS. I was not familiar with Illumos, and this was before it had a widely available web UI, but it was simple enough to use that these challenges didn't matter much. SmartOS was designed to boot reliably from USB flash storage, allowed me to use all my SATA ports for VM storage, and was my first "immutable" operating system. The primary focus on ZFS storage was another great quality of SmartOS.
Two moves and several years later, it was time to rebuild the lab, and I decided to go with Proxmox because it had decent ZFS support. Experience with Proxmox has been very good too. The GUI, many more virtualization features (in addition to the key ones I care about), and better hardware support through the Linux kernel have kept me on Proxmox for a long time.
Customizing my Proxmox installation always gave me anxiety. How could I defend my hypervisor from configuration drift? I wished there could be an immutable version of Proxmox.
Later on, I learned about govulcheck, which offers a novel dynamic/static analysis hybrid approach to vulnerability management. Nothing else out there does this (without teaming up with some huge company). I began to think that I should favor software solutions based on golang.
Ultimately, Incus (and IncusOS) fit this need very well. My IncusOS hosts excellent and I'm glad I can run Incus itself on most Linux distros - including NixOS!
I'll keep a small Proxmox host around for experimenting with new kernel features (Intel GVT-g / SR-IOV graphics) and old operating systems like Windows XP or anything else that needs special QEMU options.
The VM feature of Incus is based on QEMU/KVM so actually there's no need to keep Proxmox around, unless you really want to keep a host or cluster for experimentation with the Proxmox environment. With some configuration you can get SR-IOV and older operating systems working aswell.
There's a entire section about allocating GPUs to containers or VMs here: https://linuxcontainers.org/incus/docs/main/reference/device...
You can do the same with USB devices, NICs, infiniband adapters and whatever (as can be seen below and above the gpu part in the documentation)
For SR-IOV with VFs on a virtual machine the CLI command should look something like:
incus config device add <instance name> <device name> gpu gputype=sriov pci=<pci address>
https://linuxcontainers.org/incus/docs/main/reference/device...
But the possibility to just reroute a entire GPU to a virtual machine or container might be even more interesting:
incus config device add <instance name> <device name> gpu gputype=physical pci=<pci address>
https://linuxcontainers.org/incus/docs/main/reference/device...
Note that there's a possibility you'll need to play with the parameters a bit. All are mentioned in the docs.
CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.
[1]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs...
I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo.
I yanked vmfactory out and into a standalone repo if anyone is interested: https://github.com/whalesalad/vmfactory
I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.
No agentic stuff on our stack, as our security posture can't afford that currently.
As for the load-balancing, I think the later versions have supported targeting proxmox clusters vs a single node, and the newest Proxmox can do resource auto-balancing. That might get you what you need
They have auto-balancing now? Damn. I wrote a balancer using the Google OR-SAT solver because there was no VMWare DRS equiv.
I hardle ever access Proxmox GUI, everything is fully managed via Ansible playbooks, from deploying the LXC containers to updating them, from taking snapshot to syncing stuff to my TrueNAS is done via batching.
>I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS
The most common mistake people seem to make when it comes to NAS, TrueNAS is a NAS not a Proxmox server.
I see people running dozens if not hundreds of services on UnRaid, one fart and the whole infrastructure, if I can call it that, goes to sh1t.
Also the mentions and requirements relating AI in the article sound like they are from another world. Did things really come to this? Even if they had, you one can still snapshot proxmox vms as well as host (zfs).
Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.
Look for Terraform providers and you'll pretty much only find things to declare VMs and a few other resources around running them, but not a lot to define infrastructure, networking, firewalls, etc.
I searched the documentation but it wasn't really clear what its live migration and ZFS migration story is, but when I asked Claude to research it, it tells me that it supports live migration via ZFS snapshot replication, which is exactly what I'm looking for. I implemented a ganeti storage driver that does the same thing and am just getting ready to start testing it, but if Incus supports it I might look at moving that direction.
Anyone use Incus live migration with ZFS?
Like... this... this is not great documentation (I know I know, contribute myself): https://wiki.nixos.org/wiki/Incus
Within a few lines:
> To provide non-root access to the Incus server, you will want to add your user to the incus-admin group. Don't forget to reboot.
I mean I get that they probably mean /etc/group, but going on from there plenty of examples of "just change the config to use x" or similar.
I don't even do that, I go into a shell and run qm commands for more complicated things. And for anything I ask an agent to do, it goes straight to qm and other CLI tools as well.
Weird.
Also, do they get PBS using ZFS snapshots? Do they get HA, live migration, shared storage, easy CephOS, easy snapshots, quick cloning? Do you really want to migrate a VM from one node to another using the command line when you're in some serious situation?
Sure, for a homelab this might be OK, but the UI does make things easy for a reason.and it's not a gimmick.
I agree on a lot of the points, though, I just set up a second cluster and it took over 3 work days because of how much repetitive work is needed to do so. To be able to just take a file with instructions, adjust it a bit and deploy would be so much easier.
All valid comments about the fact that Proxmox is not limited to clicking around in the web UI.
It has qm, pct, config files, a REST API, Terraform providers, and Ansible workflows. My point is not that Proxmox cannot be automated.
Even with that automation, state drift can still creep in when debugging means running one quick command, especially if an agent is allowed to execute imperative fixes that never make it back into the automation framework. It is that, for my setup, I wanted the reproducible configuration itself to be the source of truth.
The thing I care about is not buttons versus commands, but whether I can rebuild the host from version-controlled text files and know that every important change is captured there.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
Don't install everything at the system level (NixOS). Home Manager is better suited for things like Claude. And even if you did choose to install Claude from your NixOS configuration, you can draw from a different Nixpkgs commit.
Usually, using the latest stable channel for NixOS and nixpkgs-unstable for the rest is good enough.
I converted my homelab with an Nvidia graphic
Oh man, you buried the lede there.
I switched over to NixOS around a month ago from Ubuntu and it's been just a dream. I expected there to be some friction with installing things that aren't already built for NixOS, but honestly it's been easier with LLM+NixOS than it was with Ubuntu.
edit: Thinko
I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.
ssh $username@$routeraddr '/import file=setup.rsc'One of the developers building Sylve gave a talk last year [2].
[1]: https://sylve.io/
[1]: https://docs.vultr.com/how-to-install-webmin-on-freebsd-14-0
I wouldn't be surprised, glancing at webmin's site and docs, if it can do most/all of what more focused management tools like Sylve or proxmox can, but it seems to have so many other dials and knobs that IDK if I'd reach for webmin unless I were already familiar with it.
So Sylve is sort of the opposite of webmin in certain ways: highly focused, very new, built on a specific OS. It just looks interesting to me because FreeBSD has been great to work with already at my network gateway (running opnsense for firewall and routing), and I like the idea it replacing TrueNAS on a box I already own and use both for storage and running various self-hosted services (e.g. Jellyfin, Vaultwarden, Home Assistant, and the like) in containers and/or VMs as appropriate.
The "my entire infrastructure is defined in text files" alone tells OP never hear of Ansible.
> While you can automate it with Terraform or Ansible, it..
Share the whole thing :)
Again, if you are managing everything as IaC, why touching the UI to make changes??
OP POV I just shared above tells everything that is wrong and Proxmox is not it.
You lost me there!!
Firstly, NixOS is hype, like everything being moved to Rust and failing miserably.
Secondly, "AI ... can .... safely modify my infrastructure", OP is either being a troll or haven't seem how the whole IT world is upside down because of those very same statements.
Thirdly, "my entire infrastructure is defined in text files", you clearly never heard of Ansible.
All my Proxmox LXC containers from DNS servers, to NGINX firewall aliases feeding OPNSense firewall rules, from Forgejo hosting my repos to PostgreSQL database, from Semaphore running my Ansible playbooks on schedule to *Arr collection, everything is fully infrastructure as code, there is no GUI.
I do not log into Proxmox to deploy my stuff, I enjoy CLI and Ansible makes everything like a walk in the park. I use Proxmox CLI tool "pct" for everything, even snapshots are CLI via "vzdump" and its config file.
My take from that post and comments resume in "hype" "not understanding processes" "seeing problems where there isn't one"
I only run Linux here, even my 3D printer runs Debian Netinst Linux. I am missing something here.
NixOS is more than 20 years old, and virtually all of the things that make it compelling were already present a decade ago. If it's not for you, you'll know after you give it a try. But for those for whom it clicks, it's desirable because it just provides a more enjoyable computing experience. It's not more complicated than that.
> Secondly, "AI ... can .... safely modify my infrastructure", OP is either being a troll or haven't seem how the whole IT world is upside down because of those very same statements.
I do lots of Terraform work and some Nix work with LLM agents at my job. Is it worth it to rewrite a huge amount of whatever infrastructure-as-code your LLM agent generates? Hell yes; they generate way too much code and they make lots of mistakes. Are LLM agents still useful for experimentation via infrastructure-as-code? Also yes.
> Thirdly, "my entire infrastructure is defined in text files", you clearly never heard of Ansible.
I've used Ansible and Puppet at previous jobs. They don't manage state in a comparable way to NixOS. It just ain't the vibe. Domen Kozar wrote a decent blog post about the technical differences a decade ago: https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansibl...
But the real reason is that those technical differences add up to a more pleasant experience for NixOS.
In my earlier days as a NixOS user, I used to get really excited with its design and how cool it is, and the neat technical properties that fall out of that (atomic upgrades! rollbacks! (and no filesystem snapshotting needed!)). I still think those things are awesome. But at the risk of feeding into your impression that "NixOS is hype", I've learned since then that the better pitch is about the subjectivity of using it: it feels good to use because experimentation is extremely cheap, reversible, transparent, and... fun. If you know, you know.
20 years???
NixOS mention only started this year, I have been working in IT for the past 20 years, from e-commerce to banking and airline companies, I have never ever heard of NixOS.
Not until "social media influencers" started flooding YouTube with it this year.
We must be living in a completely different world then.
I'm not even a real Nix old-timer, but I've been using NixOS on the job for roles in IT operations and software development for more than 10 years now.
And for a few years before that, I used Nix on a personal basis as a college student.
I'm aware of the rapid growth in user interest, of course; I was there for it!
That's exactly why you want your infrastructure defined in version controlled files with easy revert when something gets screwed up.
> Thirdly, "my entire infrastructure is defined in text files", you clearly never heard of Ansible.
Ansible is a half-assed version of what nix or even Puppet gets you. Having a version controlled record of which shell commands you ran doesn't help that much when you're still running random uncontrolled shell commands and hoping they do what you want them to.
>There is a deeper philosophical difference too. Systems like Proxmox or TrueNAS are designed as appliances. You aren’t supposed to run arbitrary commands on the host; installing packages or tweaking config files is discouraged because you might break the middleware or lose changes on upgrade. You are effectively locked out of your own hardware’s full potential. With NixOS, the host is fully mine. I can mess with it—installing Kodi, tweaking network drivers, running local LLMs—without fear. Because the state is declarative, it is 100% obvious and reproducible. I can break the host configuration and recover to a working state in seconds, even if the machine is running essential services."
Total nonsense. The project advises caution about modifying the host because you might conflict with their package versions or lose changes on upgrade, but that's not being "locked out," that's just a maintenance consideration. The entire point of Proxmox is that you spin up VMs and containers where you do whatever you please with full isolation which is arguably more flexible than running everything on bare metal NixOS, not less.
The whole article is a guy who replaced a working hypervisor with a more complicated stack that does the same thing, had AI write it for him, and wrote a blog post about how superior it is.