Is anyone using Proxmox on their homelabs? Would you recommend blowing away Windows and installing Proxmox and then install Windows with PCIE passthrough?
Anecdotally, it's a very effective setup when combined with a solid KVM. I like keeping my main Debian desktop and the hypervisor separate because it keeps me from borking my whole lab with an accidental rm -rf.
It is possible to pass all of a systems GPU's to VM's, using exclusively the web interface/shell for administration, but it can cause some headaches when there are issues unrelated to the system itself. For example, if I lose access to the hypervisor over the network, getting the system back online can be a bit of a PITA as you can no longer just plug it into a screen to update any static network configuration. My current solution to this is enabling DHCP on Proxmox and managing the IP with static mappings at the router level.
There are a few other caveats to passing all of the GPU's that I could detail further, but as a low impact setup (like running emulators on a TV) its should work fairly well. I have also found that Proxmox plays well with mini PC's. Besides the desktop, I run it on an Intel NUC as well as a Topton mini PC with a bunch of high-speed NICS as a router. I cluster them without enabling the high availability features in order to unify the control plane for the three systems into one interface. It all comes together into a pretty slick system
I'd recommend a separate device if you need any access to a GPU. But I do recommend Proxmox as a homelab. I still have it running on a separate 2012 Mac Mini.
My server was originally a single debian installation set up to host local services for things like git. That grew into hosting a site, vpn, then some multiplayer game servers. When I reached a point where too many things were installed on single machine, I looked at vm options. I've used VMWare/VSphere professionally, but settled on Proxmox for these main reasons: easy to set up and update, easy to build/copy vms, simple way to split physical resources, monitoring of each vm, and simple backup and restores. All without any weird licensing.
That server houses 4 vms right now. That might be a bit much for your mini pc but you could do a couple. The multiplayer servers are the main hog so I isolate resources for that. The windows machine is only for development which isn't your exact use case. I can say however that I've never had issue when I need it. Only thing I can't speak for is the need for graphics passthrough.
That said, I wouldn't mix the two use cases either initially nor over the long-term. House/network infrastructure should be on a more stable host than the retro-game console connected to your TV (IMO).
In your case, I'd recommend buying another PC (even an ancient Haswell would be fine to start) and getting experience with vanilla proxmox usage there before jumping straight into trying to run infra and MAME/retro gaming on PCIe passthrough on the same singleton box.
ProxMox is on my list to try out. So far I’m very happy with Unraid. It makes it easy to set up network shares, find and deploy containerized services, and handles VMs if you need them. I try to avoid the VM and focus on containers because it’s more flexible resource wise.
MIG does need hardware suppoet, and from what I understand really expensive licensing. I don't know if AMD has a parallel technology.
That is pretty cool that it is vendor agnostic. I've found a few docs from a few years ago talking about stuff like it on Linux, but development of it seems to have stopped or just not progressed at all.
But she is they would use it in Azure, I imagine there they are using enterprise GPUs. But in Andheri scale datacenters it definitely sounds like an advantage!
But i'd be happy to be proven wrong.
At the time I had no idea how popular it was to run this setup, I thought I was being all weird and experimental. Was surprised how smoothly everything ran (and still runs, a year and change later!)
Bonus, I was able to just move the PC to another disk when the SSD it was on was getting a bit full. Moved PC's storage onto a spinny HDD to make room to shuffle some other stuff around, then moved it to another SSD. Didn't even need to reboot the PC VM, haha.
Proxmox backup server running on my NAS handles deduplicated backups for it and other VMs too which is great.
[1] https://jellyfin.org/docs/general/administration/hardware-ac...
I have a ubuntu server install running on an old laptop to do very basic background jobs, backups, automation, run some containers etc. – am I missing something by not using a hypervisor? What are the benefits?
- Unifi Controller installs like a half dozen dependencies to run (Mongo, Redis, etc last time I used it), much easier to isolate all that in a VM
- Home Assistant's preferable and most blessed install method is Home Assistant OS, which is an entire distribution. I've run HA in Docker myself before but the experience is like 10x better if you just let it control the OS itself
- I have Plex,Sonarr,Radarr, etc running for media - there is software called Saltbox which integrates all of these things for you so that you don't need to configure 10 different things. Makes it a breeze, but requires a specific version of Ubuntu or you're in unsupported territory (kinda defeating the purpose)
Lots of stuff you can be totally fine just using Docker or installing directly onto the host. But having the bare metal system running Proxmox from that start gives you a ton of flexibility to handle other scenarios.
Worst case you just setup a single VM & run your stuff on it if you have no need for other types of installs. Nothing lost, but you gain flexibility in the future as well as easy backups via snapshotting, etc
For major upgrades, I may go a step further and do a manual snapshot before upgrading and then decide whether or not to commit (usually) or rollback (easy, when needed).
The (emotional) security provided by this is nice, as is the time-savings (after initial time expense to learn and setup the base proxmox infrastructure).
Firewall is a great example of what not to run on a vm, at least for me! I consider gateways as appliances though, and I haven't run my own router on Linux since I first got Cable internet in 2002. I remember how awesome it was compared to the weak routers available at the time!
Now I just buy a UDM Pro and forget about it!