I've been thinking about upgrading my current "homelab" (just an old PC), and I've been waiting for a small form factor with an integrated power supply and a bunch of NVMe slots. Does that exist yet?
It was so good actually that I've made it my main PC. I already had 3 monitors and this SFF just worked fine with them. No games of course.
I also have Precision Tower with Xeon/64GB ECC RAM and 1 native NVMe slot. I use PCI-to-NVMe adapters there (2 slots), it works well with ZFS mirror.
Have you considered using a NAS as your homelab? I run a few containers (no VMs yet, although I can) and just do that on my QNAP.
The TBS-h574TX (https://www.qnap.com/en-us/product/tbs-h574tx) has five E1.S/M.2 slots. The HS-453DX-8G has two SATA 3.5-inch drive bays and two M.2 2280 SATA SSD slots (https://www.qnap.com/en/product/hs-453dx), and the TBS-h574TX. Or you can just throw SATA SSDs into any NAS.
My problem is that none of them have an internal power supply. They all require a power adapter that's decidedly less easy to source than an IEC cord. I have had 3 separate experiences where I give someone instructions to grab a box and the power adapter... and they show up with the box and without the power adapter (or with a power adapter in a box of similar power adapters and no volt/amp/size rating on the box to disambiguate). In short, the internal power supply is something I actually care about.
The best bet might be to wait until they've all finalized on USB-C.
That said, I was running the Homelab on effectively a single Unraid hyperconverged NAS/compute layer before then, and that worked wonders for quite a long time. I wouldn't consider Unraid my first choice now, given the advancements in TrueNAS SCALE in the time since I chose it as the solution, but the concept of NAS + Container Orchestrator as hyperconverged homelab today is quite attractive.
Yeah, that's what I was looking at Proxmox (8.x) for recently as well, but it never made it through basic qualification testing. VM migration's would randomly hang forever about 25% of the time. Definitely not a case of bad network interconnect either. :(
I'm using Ryzen 5000 series cpus on all the test gear though, and recently saw a mention that it may have been a known problem that's fixed. But not super keen on wasting more time.
What sort of cpus are your hosts using, and are you using your VMs with Ceph as their storage? :)
That said, I do have a NAS that I can provide both NFS and iSCSI storage to, so I imagine if I run NFS as the backing storage, I shouldn't have to worry about clustered file systems. Each compute does have local storage, but the plan is to provide that via a CSI for workloads that don't need to be highly available or where fault tolerance isn't a concern.
Can proceed with more testing now. :)
* https://eshop.macsales.com/shop/ssd/owc-accelsior-4m2 (4x NVMe drives)
* https://eshop.macsales.com/shop/ssd/owc-accelsior-8m2 (8x NVMe drives)
I'd personally get them without drives included, as the OWC drives seem pretty expensive.