From Proxmox to FreeBSD and Sylve in our office lab

I have the same thing with proxmox especially after I realized how well it integrates with proxmox backup server. And I haven't even gotten into clustering yet. It really is a very solid product.

Same. Here's how I scratch the NixOS itch on Proxmox and/or libvirt[1]. One interface for both targets.

[1] https://github.com/EnigmaCurry/nixos-vm-template

Well it looks like we might soon be able to have the benefits of NixOS while also having bhyve (and presumably Sylve): https://github.com/nixos-bsd/nixbsd

Whether an appliance OS uses SystemD or not is as silly of a concern as “does the lead developer prefer cheddar or brie”

What about performance characteristics? Recoverability of workloads?

I’m interested in a FreeBSD base OS because it seems ZFS is better integrated and ZFS has a lot of incredibly useful tools that come with it. If Bhyve is at least nearly as performant as KVM, I’d be hard pressed not to give it a whirl.

What's wrong with systemd? It's easy to use and works well.

Do you have any opinions on how this works vs doing iSCSI to some other storage system using ZFS? That's how I've been handling Proxmox on the backend, and have mixed feelings. The GUI leaves a very great deal to be desired in honestly curious ways, have to touch the CLI a lot even for super basic networking or auth stuff, and of course neither side has the same insight to the data structures in question. Either you've got to do ZVOL instances and thus manual effort or scripting, or you give Proxmox a single big blob then let it manage that with LVM but that means the storage side can't give any granular help on snapshots and the like. It still can deal with data integrity and backups and storage redundancy and all that but no further, and some increased overhead. But on the other hand, I do feel like a really firm separation of concerns isn't without value. Having native support though is an interesting alternative I hadn't really considered.

I did actually notice the ZFS gui which is indeed something lacking in proxmox which doesn't default to ZFS in the installer. However once you do install it using ZFS it actually makes use of it pretty well and the user does not need to mess with the zfs cli tools much. Obviously it would be nice to have a GUI for all zfs operations too. Then again even TrueNAS refers you back to the cli for SOME operations.

On proxmox ZFS syncs do not require proxmox backup server, which actually has its own format which is very efficient in speed and disk space, but you do either need something like sanoid/syncoid or use of the shell.

> Sylve can support ZFS-on-root rollbacks better than hacking it into Proxmo

Can you explain your use case when you absolutely can't provide a separate M.2 drive solely for the OS?

See, people get things mixed and blame the tool.

Folks using TrueNAS or unRAID for backup instead of safe keeping, and then get mad when everything goes sideways and the data is gone. Your NAS must have a backup elsewhere, snapshot and what not won't save you if everything goes RIP.

ZFS is redundancy and redundancy only, but people see ZFS as some sort of backup. That is silly and wrong.

>A rollback capability is why I'm looking for Proxmox alternatives.

Your VMs and LXC container should have an automated backup. Proxmox itself takes a second to clean install it.

I had to change the motherboard and had to literraly install Proxmox 9.1 from scratch. BUT.... before doing that, I checked the LXC backups sent to a TrueNAS spool in mirror for safe keeping.

Reinstalled Proxmox, mounted the NFS share on Proxmox and voila, all the LXC containers were restored and started like nothing happened.

Right, so nothing?

Great to see we are back to counter jerking systemd after years of great work...

Sounds awesome! Looking forward to trying it out.

Excellent, this makes a lot of sense. I'll try it out

I get the impression bhyve does all that stuff too. Is sylve basically just a thin GUI wrapper on top?

(That'd be amazing if it's possible to do stuff like dump configs + check them into git from the cli, then stand them up on any bhve/sylve box later...)

One example: when learning Proxmox itself. I was able to set up a multi-node cluster with more complicated networking than I was normally comfortable with and experiment with failures of all sorts (killing a node, disabling NICs, etc.) without needing more hardware or affecting my existing things.

Outside of learning and testing I am not sure of what uses there might be but I'm curious to know if there are.

Nested virtualization can be very handy in both the lab and in production. In the lab, you can try out a new hosting platform by running one atop the other. IE: Proxmox on VMWare, Hyper-V on KVM. This lets you try things out without needing fresh bare metal hardware.

In prod, let's say you run workloads in Firecracker VMs. You have plenty of headroom on your existing hardware. Nested virtualization would allow you to set up Firecracker hosts on your existing hardware.

We run Proxmox VMs that are running Hashicorp's Nomad orchestration at $DAYJOB. The Nomad clients are then turning around and running the docker containers (Proxmox -> Nomad VM -> Docker). For us it's easier to manage and segregate duties on the initial metal this way.

WSL2 in a virtual desktop environment.

Or Windows VBS, if you need to run full-featured Windows VMs.

Do they have any near-future plans to resolve that?

> If you are unsure of bhyve's abilities then why not test yourself?

It is not possible to come to a conclusion about everything in the world yourself "from scratch". No one has the time to try out everything themselves. Some filteration process needs to be applied to prevent wasting your finite time.

That is why you ask for recommendations of hotels, restaurants, travel destinations, good computer brands, software and so on from friends, relatives or other trusted parties/groups. This does not mean your don't form your opinions. You use the opinions of others as a sort of bootstrap or prior which you can always refine.

HN is actually the perfect place to ask for opinions. Someone just said bhyve does not support nested virtualization (useful input !). Someone else might chime in and say they have run bhyve for a long time and they trust it (and so on...)

So I can't agree with your viewpoint.

I see, based on your comment and others, Sylve seems to be heavily GUI for everything.

There is nothing wrong with that but if an user cannot perform the same tasks via CLI, I see that as a big blocker for a project to be fully adopted with exceptions. OPNSense, there is zero reasons to manage the whole network and what not via CLI, GUI makes life so much easier. I would hate it having to do everything via CLI.

The other thing is LXC, Sylve seems to call it jail.

I would expect this jail to support something like below.

Ansible only automates what you do manually, the server itself only sees the command and it will never run Ansible itself, so intead of manually creating a LXC, Ansible would send:

   - name: Deploy LXC
     ansible.builtin.command: >
      pct create {{ lxc_id }} {{ template }}
      --hostname {{ hostname }}
      --unprivileged 1
      --cores {{ cores }}
      --memory {{ memory }}
      --rootfs {{ storage }}:{{ rootfs_size }}
      --net0 name=eth0,bridge={{ bridge }},ip={{ static_ip }}/{{ cidr }},gw={{ gateway }}
      --features nesting=0

Of I wanna exec into the LXC container to restore a backup and start the system, I would expect Sylve to support this.

  - name: Import lists and hotfixes
    ansible.builtin.command: >
     pct exec {{ lxc_id }} -- bash -c "
     pihole-FTL --config ntp.sync.interval 0;
     systemctl stop pihole-FTL;
     sqlite3 /etc/pihole/gravity.db < /tmp/adlist.sql;
     systemctl start pihole-FTL;
     sudo pihole -g
     "

All of that from my PC without having to go to a browser. That is the friction that your team should look into automating, there is always a way, it is just easier to go to the browser.

>How many times

It less about how many times and more about used to automate everything, spend less time doing boring things and more time doing fun stuff.

For example, when I first deployed a Jellyfin LXC container with GPU and what not, the container itself hosts nothing, Proxmox mounts the NFS shared from TrueNAS to it, and it uses a local NVMe for transcoding.

And yet, novice me picked a small storage size, 5GB or something because I only run Debian Netinst which uses 200MB of ram and 0.00001% CPU. Debian Netinst itself requires what 1-2GB of disk??

Back to your question, I had to redeploy another Jellyfin container coz it ran out of disk space with:

1. the GPU passthrough

2. mount all the NFS shares once the LXC is up

3. the transcode folder

4. rsync from TrueNAS and restore the metadata with all the movies and what not.

Had I planned to do it?? Nope.

One command line and I have a brand new Jellyfin LXC with much bigger storage, and working like nothing happened, fully automated from my PC via Ansible.

> Terraform I am not and do not intend in becoming a Kubernetes expert, many companies run Kubernetes and they don't know why they do it, some hypes make things so much harder.

But I do have a single cluster at home which allowed me to learn both Kubernetes and Terraform, I also hate Docker so much that I prefer to convert a Dockerfile into a Terraform template and voila, I do not use it to run my stuff.

I enjoy Terraform very much with Terragrunt. Terraform alone is too messy, Terragrunt makes the house cleaner.