Proxmox VE 6.1 – Open-Source virtualization platform | Hacker News Reader

Proxmox VE 6.1 – Open-Source virtualization platform

proxmox.com

152 pointstlamponi74 comments

Can't say enough about how I appreciate Proxmox. I had tried Xenserver but it didn't play well with my setup. Proxmox is robust from GUI and CLI. I run two small businesses from my home and the setup allows me to efficiently host pFsense, a mail server, multiple web and DB servers.

But even better for me, the ease and ability to pass through a GPU and run Windows in a VM while using modifiable resources makes it so flexible.

I've been a customer for over two years and they provide support with a super reasonable yearly subscription of just $99. The forum is active and the updates are recent and up to speed.

What do you mean by "using modifiable resources"?

Yes, unclear sorry. I meant modifying the availability of resources, and simply praising the general value of VMs.

But also in the cases when the VMs are never running at the same time. Sometimes the "shop" closes and gaming starts...

The Xenserver fork is really great. https://xcp-ng.org/ very active community and timely updates without all the licensing restrictions.

> Proxmox is robust from GUI and CLI.

I'm looking through the CLI docs now and I can't even figure out how to launch a VM. At face value, it doesn't look like a good CLI (well actually they list a bunch of CLI tools for VM management). Am I just looking in the wrong place?

I also want the ability to run setup commands (think AWS user-data) at first boot. The docs indicate user-data support though cloud-init, but I see forum discussions that indicate that you can only set up basic things like IP addresses, SSH keys, etc. Is there a programmatic way to inject bash commands at first boot?

You might want to consider ganeti. It works great from the command-line. There is a GUI for it as a separate project, but I've never used it. I've been running ganeti in dev/stg for 6-7 years and production for around 4, and it's been extremely solid.

As far as running bash commands at first boot, in ganeti I have customized their "create" script to include a bash script that installs some packages, clones our Ansible setup, and then runs Ansible to set the machine up. We respin our entire staging cluster every other day (even machines one day, odd the next).

http://www.ganeti.org/

For documentation: https://pve.proxmox.com/pve-docs/

The "qm" page shows:

  qm start <vmid> [OPTIONS]
  Start virtual machine.

Please read the docs properly mate.

pct - controls containers

qm - controls vms

do you know if you can pass through raw disks as well? I.e. i see that it supports ZFS could i attach that pool within a vm and not directly to proxmox?

I have done PCI passthroughs and USB passthroughs. One of my PCI passthroughs is a MegaRaid card. Not sure how you are physically connecting the pool, but if the pool is running on dedicated hardware, there's a good chance you could passthrough the hardware...very good if PCI or USB based.

Proxmox's hosts controller does not require being on ZFS itself...you can have multiple drives and filesystems. If it were available to Proxmox and not passed through, it could be allocated only to a particular VM. In case that was your only end goal..

Last year I had applied for a job where we would have to manage Windows servers. Stuff with Active Directory and storage... I'm Linux guy but have extended experience in WinServer 2012. I just set up a server, boot ProxMox, spun up as many as Win Server 2016 instances I wanted to experiment with all the aspects of the new Win Server and guess what? I was second in the job interview. Thank you ProxMox. You is the best.

I can't say enough good things about PVE! I started managing VMware in 2003 and ESXi is a wonderful product. I was running ESXi at home on the free license which only gives you eight vCPUs (if I remember correctly). After hitting that limit, I slowly decided to move to PVE. I was able to migrate my machines quite easily and have never looked back. For so many use cases, PVE is just as good as ESXi. If you're looking to virtualize and have relatively straightforward requirements, it's worth a look. Free is great and it's rock solid. I made that switch three years ago and haven't had a single issue.

One thing I've never understood with Proxmox is why networking can't be configured host-side. It seems you are expected to go into each new VM and manually set network settings. Why not have a DHCP server on the host that hands out prescribed addresses by VM id?

If Proxmox could do this in addition to maintaining port forwarding to these VMs I would be happy.

I've set up Proxmox a couple times. I do exactly what you described, using dnsmasq. There are a few blog posts out there, this is one [1] I used for inspiration.

I too wish it was a built-in Proxmox feature, or at least officially documented...

[1]: https://blog.jenningsga.com/private-network-with-proxmox/

you can configure the network in the guests in cloud-init tab.

I installed Proxmox at a previous job, doing snapshots in seconds on a database container was nice for backups and the UI makes you feel informed and in control

I would recommend it for most businesses honestly

Proxmox is awesome. :)

Back several years ago I worked on a project to build a system for making demos of network solutions. We ran Proxmox inside of an ESX virtual machine and then used it to virtualise various pieces of the demo architecture. The ESX image could simply be coped to an external drive for the consultant to use.

Ultimately the company went for a central-lab demo environment instead but it was pretty cool anyway.

The demo itself was stored as a bunch of Ansible playbooks and assorted scrips so it was Git-friendly, to be applied to a generic ESX template.

We built cool things like a canvas-based Visio-like UI that could be used to define VMs / containers / networks and provision at the touch of a button.

Similarly the solution itself would be visualised using a diagram that supported dynamic components like links showing traffic, counters, graphs etc that would show traffic running between the VMs inside the demo.

We also had a central server that would store pre-created demo templates using Ceph that could checked-out for use via a web UI.

Fun times :)

> We built cool things like a canvas-based Visio-like UI that could be used to define VMs / containers / networks and provision at the touch of a button. Similarly the solution itself would be visualised using a diagram that supported dynamic components like links showing traffic, counters, graphs etc that would show traffic running between the VMs inside the demo.

Nifty! What frontend/backend dev stack did you use? Does anything like that exist in OSS, for any virt platform?

No, I built it for this project.

The canvas diagram was plain JS and for logos we used SVGs from company official PowerPoint icon pack to make it look consistent.

The backend was Node.js that ran Ansible playbooks that provisioned things via Proxmox API and then configured the VMs as needed.

Love Proxmox. It has its quirks but overall a great platform for home usage (labs etc) and for small businesses.

Probably one of the more notable current applications that still use a lot of Perl. The admin interface is mostly Perl, as are many of the command line utilities.

Zoneminder uses a lot of Perl still IIRC.

Yep. Also, Booking.com is probably the most notable current Perl user.

Craigslist.

I've been running proxmox since v3.0 with lots of success. I admit that back then you could break stuff easily if you did not know what you're doing. Over the time I've been running it continuously with every version in my homelab and it just got more awesome. I tried ESXI and Microsoft Server.. But in the end I always went back to Proxmox and the very stable debian that runs it. Now with the latest version ZFS encryption is also native and I have pretty much everything I could ask for. Can only recommend!

Proxmox is good with the large exception of high speed interconnects.

If I have Mellanox IB cards in my servers, proxmox fails to handle ipoib without a lot of legwork. Compare that to something like oVirt; that supports it out of the box.

There is very little incentive for me to recommend a proxmox subscription to any of my clients because having >= 40 gbit interconnects is far better than using lags on single gbit. High traffic internal applications, (and migration!) benefit so much from those interconnects.

I have run dual 100gb mellanox in it with no problems without the IPoIB. It seems a pretty specific problem that is probably related to the mellanox cards and not to proxmox, as similar bugs show up with mellanox in oVirt.

The company behind it in Vienna tried to recruit developers for 3200eur bruto/month, I found this super low compared to current standards.

I had a brief stint with proxmox a few weeks ago and didn't like it much. I didn't like the nag popup when you don't have a subscription (and I had installed the nosubscription package!). I migrated to it to see if it would be easier to manage than OpenStack, but I honestly found OpenStack easier to deal with, probably due to familiarity.

Yeah that nag is my only nitpick with Proxmox. It's a bit too much in the face everytime I login. You can easily disable it with a code snippet, but I think the devs should just advertise the non-subscription model differently.

I've used Proxmox on my dedicated host and it was working really well, however one problem I've encountered is that it doesn't really play well with guest OpenBSD (guest randomly was hanging, with no apparent pattern). But Linux and Windows worked perfectly.

I agree, FreeBSD (FreeNAS) doesn’t play well with it and iSCSI did not work well according to a lot of forum postings and blogs. I wanted a robust NAS and doing it through a VM did not provide the peace of mind I was looking.

The only upside is now I can build a dedicated visualization machine or grab some old enterprise hardware from Craigslist. My wife is not so amused.

I gave proxmox a solid try for virtualizing a mattermost instance. But it was just way too slow compared to bare metal. Chats took seconds to load instead of ms. I'm sure there's some explanation for it but I couldn't be assed to figure it out and reverted.

You were probably missing the virtualization flag if it is that slow.

Hmm possible. I will give that a try.

Proxmox is pretty cool, but it doesn't make sense unless you're running many machines and utilizing ceph. For a home user, Debian (or some other stable distro) is still going to be the best distro for doing hypervisor stuff.

I run it on a single node to run LXC guests for all sorts of non-production workloads. That it takes care of all the networking, storage, OS templates, backups, KVM access and has a comprehensive web UI is all very useful to me.

What else is there to take its place? oVirt can't do LXC.

virt-manager can do lxc. I rdesktop to the server and set up things with virt-manager.

You can also install pve on a debian system if you want the proxmox gui. If you want to do full disk encryption, that's the recommended approach, because proxmox's installer doesn't support it.

I'd disagree entirely. If you want an extremely easy to use gui that gives you access to KVM and LXC virtualisations, along with easy access web remote consoles, snapshots and backups, then Proxmox has been a fantastic go-to for over 6 years.

My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.

> Proxmox is pretty cool, but it doesn't make sense unless you're running many machines and utilizing ceph. For a home user, Debian (or some other stable distro) is still going to be the best distro for doing hypervisor stuff.

> My only objection is that it's a 'full-fat' linux distro that needs to run off a real disk (unlike ESXi), however on every other level it's far superior.

Why not Alpine Linux? I use that in diskless mode https://wiki.alpinelinux.org/wiki/Installation#diskless_mode

> If you want an extremely easy to use gui that gives you access to KVM and LXC virtualisations, along with easy access web remote consoles, snapshots and backups

My understanding is that alpine is a very light weight linux ditro and isn't a productised, easy to use hypervisor with full control via a gui.

It's a bit like me saying "I like music editing using xxx because it has a tonne of value added, makes it easy, but it requires an amount of hardware resource" and someone replying "pah, I just use a reel-to-reel, some sharp scissors and a roll of tape!" - unless I'm missing something, it's not really comparable.

edit: as tfigment pointed out along side my comment, I'm slightly annoyed that proxmox is a full-fat linux install for the hypervisor. The comparison I tried to make was with vmware ESXi which boots from an SD card or USB stick and is almost entirely held in memory after boot. ESXi has a whole number of other issues attached to it, but the fact I can throw a £10 usb stick in a server and use that as a boot drive, rather than "wasting" two of my SATA/SAS bays to a pair of real disks with the hypervisor installed on is nice.

They mean proxmox requires full debian. Pretty sure its not a choice. In VMs you can run anything qemu runs. Also no Docker. The devs are opinionated, not that that is necessarily an issue.

Docker has been on the wishlist for a while but it seems like the developers refuse to integrate it. I just installed docker-ce manually and added portainer as a systemd service. That way I can manage containers with a 2nd webui.

Oh, neat.

I've been wanting to do something like that, but I was afraid that the combination of LXC + Docker would blow up due to networking or cgroups or something.

Are you running Docker and LXC containers side by side?

Any documentation on how you did it?

What are your thoughts on this setup vs just docker+portainer on regular Ubuntu LTS, or even a dedicated Docker distro like Rancher?

I had a previous home server built around docker, and found it fiddly to get right— most of the services I wanted to deploy didn't already have containers, and there was a lot of outer config stuff to play with, getting the containers to restart properly on reboot, getting DNS to resolve them, getting the right data directories set up as binds.

But that was also in 2016, so it's likely things are a lot better now, and conversely, it's not necessarily clear that a VM-based approach would be a lot better in some of these areas.

Yeah, proxmox is debian. I think people don't realize you can install their fancy web UI on debian.

Proxmox is opinionated about how storage is allocated (I think lvm-thin is a bad idea), doesn't support docker, and has some quirks about getting the apt repos set up correctly if you're not a paying subscriber. It's fine if you want to use things in their opinionated way, but know that they're not doing any magic to make KVM work. KVM and LXC are the exact same as you'd get from debian out of the box.

Mostly agree but proxmox still has quite a few quirks. Its cloud-init support is difficult to use unless you just need hostname and single user and single NIC which does cover a lot of cases but not mine. Its web api is non-standard and cannot upload full cloud-init config files but requires files already on disk. Also no Docker, just LxC. I found ESXi free tier better for test lab for somethings like backup and snapshots but no creating vms from scratch via api was an issue which is why I used Proxmox in first place.

But you don't get that great UI. I'm the world's biggest Linux cheerleader; one thing you never hear is that almost every 'modern' Linux has a hypervisor built-in (KVM/QEMU) to it.

PVE is basically Debian with a wonderful web-UI wrapped around KVM/QEMU. You don't really need it, but it sure makes life easy!

Unless you need clustering and shared storage. Without Proxmox you are gonna need to setup corosync/ceph yourself.

I disagree completely.

It makes sense to use Proxmox at home or in a business on a lot of planes. Ceph usage is not required, and it comes with excellent ZFS support. There are a lot of other great features which Proxmox facilitates.

You can see it as Debian, with a web gui for managing virtualized resources. Ceph is not required, but it is something nice to have if you have many hypervisors and no central storage.

But it is meant for using the machine mainly as an hypervisor, not as a desktop with the ability to ocasionally run a few virtual machines (in that case kvm+virt_manager or virtualbox are more desktop friendly).

I'm currently using Yunohost, but I'm finding the system a bit brittle for some use-cases, and some choices rather arbitrary (which was also the point, as I initially chose it to spend less time managing my server).

I would like something a bit more fleximble, and was seriously thinking about jumping to Guix or Nix. Does any of you have any input on this? Proxmox is also a candidate, as I could run my old Yunohost side-by-side with Nix, Guix, and others, I guess. I'm just afraid of overcomplexifying my nginx config.

Allso, if I run Guix/Nix on Proxmox, I'll lose the declarative config advantage... Is the opposite possible?

I am sure usecases are slightly different for these two. At least for me personally, "plug and play" offer was the key - so I could easily move stuff around and not think about it. And it has browser access so non-tech people can use email/Nextcloud/etc.

But perhaps Yunohost can be used to model what you have in mind, and once solidified, use it as a base to build VM with precise setup.

Ansible is perhaps the closest you can get to declarative config... See "sovereign" on github - I migrated from it to Yunohost, as things kept breaking and I was looking hands off setup (hence Yunohost), but given that Sovereign is ansible-based - perhaps you can leverage it.

Proxmox running on a nuc i7 with 64GB of ram and a fast nvme drive. The system is very fast and takes very little space. Just wish I had an extra nic.

The Hades Canyon NUC actually has 2 RJ45 ports: https://simplynuc.co.uk/hades-canyon/ But it does cost you some serious money. I have a small i3 NUC and would rather plugin a USB3.0 to ethernet adapter.

I'm using a miniPCIe ethernet card and added a hole to the case. Works well, but wouldn't trust it in prod unless some serious testing was done.

There's a dual (Realtek) NIC ASRock Ryzen Embedded for $410, don't know how well it supports Linux, https://www.newegg.com/p/N82E16856158065

You can easily add a USB gigabit nic, or even 10 GbE via TB3 https://medium.com/@moondev/my-adventure-adding-10gbe-networ...

Used proxmox in its earliest iterations within a startup and found it fantastic.

Recently I tried it again when attempting to install Untangle on a system with EFI. Untangle didn’t support EFI so I needed to virtualize it.

I found it too difficult to get software network adapters to work with different physical network cards.

Switched over to VMware ESXi and had vswitch configured within minutes.

YMMV and I’d still recommend Proxmox for simple virtualization projects.

Just set up Proxmox on my home lab last week. I’m slowly ramping up the learning curve. Sounds like I made the right choice.

Would be cool to see an addon for NAS services in Proxmox... I'm setting up a home server when I get the chance and just want it to run a VM, a handful of docker containers and provide NAS storage at home. In the end, will probably add Docker support, run the NAS in a container or vm and leave it at that.

Why integrate the NAS?

I run a Synology full of spinning rust for backups, archives, and shared file storage. It works reliably and is stupid simple to maintain.

Adjacent to that is a server running Proxmox, and I'm currently working to add Kubernetes/Rancher as a layer on top of that. The Proxmox server has a local SSD array so the VMs have fast low-latency storage. VMs that touch larger files (media server, etc) have an NFS mount to the NAS.

IMHO the financial/maintenance overhead of a second unit is totally outweighed by the easy opportunity to have two performance and reliability tiers (fast homelab vs reliable slow storage), ability to upgrade/change the two units independently of each other, and ability for cohabitants to use the NAS without having to understand/deal with the virtualization stuff.

I have an older Synology NAS, it's really slow, and TBH need more compute for a few things at home... but don't want to spend on yet another box for this. The main role of the new server is NAS stuff, but if I can do a few other things in the background, so much the better.

I'll be running proxmox and VMs off of an nvme drive in a 4x pcie adapter, and the NAS drives using the hotswap bays in the front.. it's a used R710 (iirc), and has 6 3.5" bays (came with 2TB drives, but replacing the raid controller with one that will support larger drives and upgrading to 8TB as time/cost permits).

I could totally drop the home server compute requirement before the NAS requirement as my desktop could handle those chores.

TBH FreeNAS is a great NAS/SAN solution. I'm running a 4x4 RAIDZ2 (8TB effective) with a 10 year old quad core AMD and 16GB DDR2 RAM.

If you are okay with the limited VM hosting capabilities of Freenas (headless VMs and containers on the same network as the file server) then you can use it for that as well.

Unraid pretty much what does you're describing

https://unraid.net/product

downside is it's closed source and not free

Proxmox is pretty nice, except it has two glaring issues:

Setting up networks is a pain in the butt.

Setting up storage is even worse.

Both really require the command line, which shouldn’t be necessary.

I ran Proxmox in production for a year for a small ecom company, but if I had to do it again, I’d go vSphere. It’s worth it.

I just read all the release notes and this reduces the biggest issue with networking:

> If the package ifupdown2 of the Debian network interface manager is installed, it’s now possible to change the network configuration and reload it in the Proxmox web interface without a reboot.

What is your experience with major proxmox and OS updates?

We use it at work since Proxmox 3.

The only thing I miss is a truly compatible Terraform module that lets you create accounts, machines and infrastructure. That would be cool

What's wrong with using plain libvirt tools, like virsh? Is there some other advantage to Proxmox besides having a nice WebUI?

Nobody said it's wrong. It's perfectly OK. Proxmox is precisely a nice web UI for that and a lot of stuff integrated, such as LXC, CEPH, HA, Live Migration...

I cannot stand the Proxmox software. It’s a total pain in the ass to use.

XCP-NG is also a nightmare due to how half baked Xen Orchestra is.

VMWare ESXi is the only type 1 hypervisor that I would ever recommend to a fellow hacker. You can get a free license for personal use.

Can you give an example of the half baked-ness please?

Have you used it? It’s the perfect example of Twitter bootstrap being all the rope to hang yourself.

Reply on news.ycombinator.com