Ditching Vagrant: VMs with KVM and Virsh on Debian | Hacker News Reader

Ditching Vagrant: VMs with KVM and Virsh on Debian

benjamintoll.com

101 pointsfanf245 comments

I never got into Vagrant but "lima" has worked well for spinning up (temporary) VM's quick: https://lima-vm.io/

There's plenty of images for different distributions, automatic file sharing between host and guest, etc.

Mama, what a read! Nice work! Same story on macos: recently ditched the big (corpulent, distended, ossified) vagrant + virtualbox _repositories_ for some spritely lima + apple virtualization framework scripts, and been happier for it.

It was a forklift, to be sure, but the machines're snappier, & you buy yourself a little headroom.

For many use cases you'll do fine without the libvirt tower of babel, with just the qemu/kvm cli.

Oh dear. I got called a "cave man" for suggesting such a thing. Good luck with that.

Running a VM is just a qemu-system-whatever command. They usually get pretty lengthy, but it's nonetheless an ordinary shell command.

What are people using Vagrant for these days? The last time I encountered it was before Docker become widespread.

Anything that uses virtual machines (perhaps you're running FreeBSD inside them, for example), or where you need complex network topology.

My homelab consists of several baremetal hosts, Vagrant lets me instantiate a full testing environment from a text file and kick off provisioning scripts.

Same. And now, my AI agent uses Vagrant to validate it's ansible yaml before it creates pull requests. It even does `vagrant up` inside a running vagrant when making changes to Vagrantfiles.

These days I'm just using plain qemu directly and managing the bridges with systemd-networkd.

virt-builder is an alternative to installing a guest OS from scratch. It has a selection of popular guest OSes template images that you can quickly install non-interactively.

It's convenient when you don't want to manually install a guest and also don't want to figure out preseed/kickstart/etc.

It makes creating new KVM guests easier and has handy features like the ability to copy in ssh keys so you can connect to the guest right away.

I think this is a backwards way to set up VMs. "Installing a guest" is the wrong concept. Use a VM image pre-prepared from your favourite distro to skip that step. cloud-init is the standard way for it to specialize to your use case.

There used to be virt-manager

Wonder if it’s still around ? Hope it’s doing well !

Unfortunately it still exists. Virt manager drives me crazy because it hides the VM files in its own directory with permissions that aren't yours forcing you to use sudo to manually manage your own fucking vm files. Creating a new VM? You're forced to pick an OS by typing the name of your OS into a search box which is tedious and doesnt give you an option for generic x86 machine. I hate it with a burning passion and instead manually manage VMs by reading the qemu man page and writing a script to directly invoke qemu. I'd recommend VirtualBox over it any day.

I can't say I share your hatred. It's my go-to management interface for VMs like this. Especially because it allows managing a remote libvirt install over SSH, handling things like forwarding the screen and input for you.

If you don't want to pick an OS preset, you can always just go for "manual install" and a "generic" OS and pick your own preferred configuration later. Or you paste the URL for an online install directory, which is even easier.

To manage libvirt machine without root, you can add your user to the libvirt group.

You can add directories to the storage, including ones in your home directory. Generic is actually the default option, all you have to do is to disable auto-detection.

It's fine to run qemu directly, but virt-manager ain't bad.

I don't share your antipathy against libvirt, but I do the same. To configure a qemu vm via libvirt you need to learn two concepts: the qemu internals, and how they're mapped to libvirt properties. And since the qemu internals are mostly documented as command-line switches, you can skip learning the libvirt mappings by just using shell.

> Virt manager drives me crazy because it hides the VM files in its own directory with permissions that aren't yours forcing you to use sudo to manually manage your own fucking vm files.

I just checked my `~/.local/share/libvirt/`. It doesn't do this for me, and I don't think it ever has.

I do remember having to set this up at some point. Looks like this is it:

https://unix.stackexchange.com/questions/799034/whats-the-di...

There are some limits around network ports in User Sessions, but it should suffice for anything you'd use Vagrant for.

> Creating a new VM? You're forced to pick an OS by typing the name of your OS into a search box which is tedious and doesnt give you an option for generic x86 machine.

...There is though? It's in the dropdown under "Generic or unknown OS. Usage is not recommended (generic)". Here it is in the code if you don't believe me:

https://github.com/virt-manager/virt-manager/blob/c3df2ba/vi...

And a random tutorial which makes use of it:

https://cyberlab.pacific.edu/courses/comp178/resources/virtu...

There are two ways to run the Libvirt daemon, which are unprivileged and privileged aka system. You are using unprivileged mode, the parent is using system mode which is more powerful and provides better isolation but does hide stuff in /var.

For example, running QEMU as its own user and using PCI passthrough is only possible with the system daemon.

You also need the system daemon to set up bridged networking, though the unprivileged daemon can use it through a setuid helper.

Nah, just add your own directories to storage. I've always done it that way and it works just fine.

You have to give your user libvirt rights (i think it's a group) and connect as user rather than system.

"around" is the best way to describe it; the libvirt/virt-manager ecosystem isn't dead, but redhat killing off ovirt/rhev support drained a lot of resources out of it.

And for some bizarre reason people decided that the much less mature (both organizationally and technologically) proxmox VE is the best thing since sliced bread, so everyone who does care about linux virtualization is now trying to hammer some homelabbers' collection of perl scripts into a replacement.

It would be funny if it wasn't so sad.

Red Hat is still using and developing libvirt (though the user facing layer is Kubevirt instead of oVirt) and virt-install, and even though virt-manager is not growing new features libvirt takes backwards compatibility extremely seriously, so new libvirt works with relatively old virt-manager.

kubevirt is a cute joke, but hardly a replacement for ovirt or vmware. It's a real shame redhat keeps sabotaging itself to chase hypes with half-assed non-solutions.

> It would be funny if it wasn't so sad.

Well it has been replaced. We don't need anything now.

Until recently, I used it routinely for VMs, and it worked solidly and reliably. There is a ZFS storage backend as well, always nice to see since I've loved to use zvols for VMs, even when I did VirtualBox on OpenIndiana back before ZFS on Linux was viable.

But I found that Proxmox fit my needs much better than wrestling vanilla Ubuntu or Debian into a VM server, particularly for things like backup/restore and instrumentation, or setting up a bridge on a desktop-based installation. Since both are based on QEMU/KVM, it wasn't too hard to move my VMs (one thing you might need to look out for is changing network interface names).

I still use it for home projects and it's great!

Little wonky to get the config files all setup for a VM but LLMs make that a breeze these days.

It's usable (?) as a toy: numerous gui inconsistencies and lack of security isolation [0].

Can't say whether the of lack funding (move to Cockpit from Red Hat) is the reason, or maintainer is just an obstructionist (strictest feature policy).

[0] https://github.com/virt-manager/virt-manager/issues/358

That is infuriating to read. I have been using virt-manager and sleeping soundly that it was a secure isolation level.

VM guests should be considered trusted? Are you kidding me? That’s my number one usage -to operate software I do not trust.

This is a shockingly common sentiment in open-source maintainers, that all software on a machine should be trusted.

Just to name two recent (and thus inexcusable) examples on top of my head:

Systemd refused to fix a privilege escalation through their use of "less" for years because they argued any user getting to that point should be trusted. The real world doesn't work that way unfortunately. They relented when someone took the initiative to fill a CVE.

Caddy's admin interface is system-wide opened, any local user can inject configuration and take over everything. Doesn't have to be a local user; think of any service you run that can makes web requests on behalf of people (custom api endpoints, avatar fetching, etc), it can be used to take over Caddy.

I use it multiple times a day, and it's great! It doesn't try to hide details from you, but it does put a well-thought out GUI in front so you don't have to stumble around. It's not perfect, does take some getting used to, but it's stable and relatively comprehensive. I love it. Caveat: I'm mainly using it for home usage in my home lab.

I also love that I can drop to the CLI easily for scripts and it plays perfectly well with virt-manager, and it can be used easily over SSH to manage VMs on a remote host (which is key to my use case).

vagrant, docker, qemu, lxc, libvirt, incus, proxmox, nixos, guix, incusos, talos, microVMs, distrobox, packer, distrobuilder, ansible, terraform, cloud-init... Just crazy layers of choice and combinations

I am trying vanilla qemu with cloud-init. Images: https://images.linuxcontainers.org/images/

This is the first time in my entire life I've ever seen someone use a Plan 9 formatted disk. Deep respect.

I might be wrong since it has been a long time since i used WSL, but i think that WSL2 uses 9P for mounting between the VM and the host.

9P is a filesystem protocol (like NFS or CIFS) rather than a local file system (like btrfs or NTFS). It's used to share files between the host and the guest. That way there's no need to copy files between the host and the guest.

Thanks for the clarification and another example of my ignorance about it. The only time I've ever seen it is when I was trying to build a "slender" kernel and I was removing all the stuff I've never used(eg: 9P ).

The only thing I did know about it was that it was named after the movie "Plan 9 from Outer Space". Kinda cool reference.

Have you tried distrobox/toolbox instead of having to spin up VMs? Also microvm looks nice

Most of my vagrant usage is from when a coworker has set up a vagrant file for a project.

This is another area I hope I'm able to migrate to systemd. I already use nspawn for containers but vmspawn is still a bit new and limited in the options. Once it gets there though it'll be nice to have system+containers+vms under one consistent roof.

I've been using Canonical's LXD for this, under Debian Trixie. Nowadays I can usually get away with just running containers for things I need vs full-blown VMs. I guess the only downside is that in order to get the latest version of LXD installed easily you'd have to do it via snap, which I understand is a dealbreaker for many.

Damn, I didn't know about vmspawn, will keep an eye. But I've been running VMs inside nspawn containers by binding /dev/kvm and using plain qemu. Works great.

Honestly though... why not just systemd containers with minimal mounts, and if you're needing an ip, what not an unprivledged LXC container? The surface area is far smaller and they're way easier to inspect.

Systemd-nspawn is great. You can have any networking you like with it too, no need for LXC.

I haven't dove into that yet! But it does look pretty danged neat. I'm really starting to realize that containership should be handled by the process manager

wouldn't incant/incus be a better drop in?

Membership in the libvirt group is root-equivalent, not a permissions fix. Through qemu:///system it lets you attach arbitrary host block devices to a VM, so anyone in that group can mount and read the host's own disk. If you want to manage VMs without root or that group, use qemu:///session instead: unprivileged, images live under ~/.local/share/libvirt, no sudo. Tradeoff is you lose bridged networking without a setuid helper.

Reply on news.ycombinator.com