Proxmox VE 6.2
proxmox.com
proxmox.com
We once built an internal system to make network solution demos in it - a VM image running Proxmox that would be configured with any number of other VMs and containers all inside of that image, so you could literally take the while demo environment with you on a USB stick when you presented it, work on it while on a plane etc.
I also built a mini cloud on a Proxmox cluster using Ceph that would instantiate these images from a template using a web UI - a bit of Proxmox recursion fun.
That was a cool project to work on. :)
I also don't want to use aryufan patches to enable incremental backup. I feel like they could probably partner with duplicacy to ship that feature. Duplicacy already released a paid version specifically designed to backup VMware hosts so it should be possible to adapt it to backup Proxmox hosts.
(Thank you all for your comments ! They gave me some ideas)
https://git.proxmox.com/?p=proxmox-backup.git;a=tree
https://forum.proxmox.com/threads/proxmox-backup-server-and-...
snapshotting filesystems will be much more efficient in terms of time/effort compared to de-duplicating later, but other than that, all of these solutions are mostly equivalent.
For example if I have 1.5TB of VM and only 10-20Gb changed today I'm still sending everything to a remote storage. Snapshotting locally and sending the zfs volume seems to be a more efficient solution
bup and borg (and likely restic) keep local hash summaries that are synchronized with the repository if it is remote. They would still have to _read_ the entire 1.5TB VMS if every vm file has changed (there are no facilities on a regular file system that they can use to figure out which parts changed), but they WILL use those local hashes to find out parts actually changed, and only send those changes to the remote storage. The local hashes typically occupy 0.5% of the data size (for bup, by default: 20 byte SHA1 per 8192 bytes on average;
You can change the chunking parameters for bup/borg/restic for a tradeoff between cache size and granularity of change detected; for VMs it might make more sense to e.g. have 256KB chunks in which case you'll have 0.01% local storage overhead, and likely still negligible network transfer overhead.
> I don't argue that it would work but when you do that you're still sending all the data over some data link.
That's only true if you include e.g. the SATA link in that list; To avoid that, you need a filesystem that effectively tracks change ("damage") regions, or VM storage that uses lots of underlying files so you can track it on a regular file system (parallels on the mac used to do that so time machine backups are efficient). no way around it.
Doesn't ZFS do block level checksums that could be used for this, and backup tooling could use block pointer metadata for this purpopse?
At this point, I'd never use anything else. Proxmox "just works" in the best sense of the phrase. Never had uptime issues or anything (knock on wood). The only bare-metal install at this point is my plex server, which has a special level of importance in my close and extended family, so I baby it a little bit. A lotta bit.
I use it for a lot of things - a windows VM for running the mega client (so it's always on + isn't on a personal machine to snoop around), a postgres server that I use for the dev/qa versions of my personal stuff, a jenkins+wekan server, a vm just for minecraft servers, etc etc. It's so easy to just spin up a new VM when I need a clean slate for something random.
If I at some point actually start my own business, we'd be a proxmox shop for any non-cloud infra. And the cost-effectiveness of even a handful of 1-gen-old Dell servers on-prem (for things that don't need super high availability) is really hard to overstate when your budget isn't attached to VC funding. I would not run prod infra on-prem at this point, though: DigitalOcean makes the cloud far too affordable for that. But I'd probably have all CI, data analysis, etc etc local for a good long while.
One gripe I have is their subscription/license nag. It's annoying and tacky. I would gripe less if they had an option that made it feasible for me, a guy running it in his basement with 0 profit motive, to buy a license. Right now, it'd be (~$85 * 2 sockets * 4 servers) for me. Sorry, that's not happening, I could actually use DigitalOcean cheaper than that (and maybe even AWS). And it would be a huge % of my house's tech-related OpEx (compare that to ~80/yr for domains, $5/mo for my one droplet, $100/mo for internet, etc). If I had to name a price, I'd say $25/yr would be about the most I'd drop (as a guy with a basement server room, of course). Contrast this with Plex which I got a lifetime license for for like 150 bucks or something, and plex doesn't even really have an alternative. I could live without proxmox (libvirt is really annoying, but not that annoying), but I couldn't live without plex.
The pricing seems fine for businesses, though. I'd argue it's probably good RoI.
root@pve01:~# cd /etc/cron.daily/
root@pve01:/etc/cron.daily# cat proxmox-nosub
#!/bin/sh sed -i "s|if (data.status !== 'Active')|if (false \&\& data.status !== 'Active')|g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
I put it in daily in case I forget to run it during an update.
I used Proxmox in production on a decently sized ecommerce site, and it worked well otherwise.
I don't think there is another notable, proven, open source clustered manager of both actual VMs and containers. OpenNebula might be the closest: https://opennebula.io/get-your-hands-on-v-5-8-edge/
LXD is similar to proxmox if you only need containers, and pretty slick. They use their own distributed version of sqlite to propagate state, so it's much simpler than proxmox.
Googles Ganeti is strikingly similar to Proxmox, to the point where I suspect Proxmox started as a reimplementation:
http://www.ganeti.org/ (developed at https://github.com/ganeti/ganeti)
It does not come with a GUI, though there are a few third party ones. It has API endpoints for everything, a fully featured CLI client, and is extremely stable.
Googles marketing department does not want you to know this, but Ganeti runs most of its internal infrastructure (not public facing stuff).
No affiliation, just a happy user.
Thanks for checking the dates, I thought Proxmox was newer.
The basis of Proxmox VE is the real time replicated clustered configuration file system: https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html
I do not see anything similar for ganeti, also not much similar in general besides that it manages VMs?
Indeed the similarity with Ganeti is only on the surface: being a scalable infrastructure management tool, DRBD as a first-class supported disk backend, etc.
The cluster replication and consensus model is very different: Ganeti has a single "master" node that makes all the decisions and replicates cluster state to other nodes.