QEMU v4.0.0 released | Hacker News Reader

QEMU v4.0.0 released

qemu.org

310 pointsingve159 comments

Last week, I intended to replace VirtualBox with QEMU for my personal use. The spartiate documentation made it really hard. I lack the competence to seriously contribute.

The official documentation just explains the numerous parameters. It also points to tutorials (wikibook, etc), which unfortunately are very basic and out of sync. Even Arch Linux does not provide a good documentation (lack of clarity, some obsolete syntax, "Networking" section is a mess...). Like often, Debian's wiki is obsolete, and even their QEMU images are 5 years old. I had to combine the following 3 sources and a lot of try and guesses to build my VMs.

QEMU Gentoo : https://wiki.gentoo.org/wiki/QEMU/Options

QEMU official : https://qemu.weilnetz.de/doc/qemu-doc.html

KVM Performance : http://www.linux-kvm.org/page/Tuning_KVM

Welcome to the Open Source experience... sadly.

https://www.virtualbox.org/wiki/Licensing_FAQ

As opposed to what? Most proprietary tools don't have good documentation either. And then you may not even be able to look at the code to see what it does.

I don't get why he's downvoted. Virtualbox (which is not proprietary), VMWare, Hyper-V are way easier to use.

That's a fact.

And not many people need to look at the code in real life...

You should compare VirtualBox to virt-manager or GNOME Boxes, not QEMU, just like you shouldn't compare Visual Studio to gcc.

Yeah, but VirtualBox & co. come as a package. virt-manager and Gnome Boxes, as you can see from the other comments, have been missed by various people trying to use these tools :)

I am on my third confusing error using virt-manager to create an armv7 build environment. First one was some ca-cert error, which meant I forgot to type a slash somewhere in some URL, second was something about not being able to enable firewall, which required firewalld and dnsmasq to be installed, now I'm having an error adding a qcow2 image to my vm setup, it probably doesn't like my mounted storage. We can compare this experience to terrible vendor lockins.

I have not used it, but Drew Devault had a post explaining how to get started with Qemu : https://drewdevault.com/2018/09/10/Getting-started-with-qemu...

After a decade of only using qemu via libvirt on the server, and lazily using VirtualBox on the desktop, this post convinced me to start using qemu locally. I run unprivileged guests for development and testing and it fits my workflow perfectly for about six months now.

The Gnome Boxes app is excellent

https://wiki.gnome.org/action/show/Apps/Boxes

There are several higher level frameworks that wrap QEMU-KVM and make it easier to use. oVirt, libvirt and proxmox are among my favourites, with the latter being quite polished and with advanced features for HA use.

Using QEMU directly is a bit tricky and it's not really intended to be used in that fashion by most users, unlike VirtualBox.

Both gnome-boxes and virt-manager use QEMU under the hood though, and they provide a point-and-click experience. I use virt-manager myself and I do not think it's any more difficult than VirtualBox is.

We have several windows vms running in virtualbox. All have rdp/vnc for specialised programs. (These are offline).

Everytime I tried to migrate to qemu or virt-manager or gnome-boxes it was a real nightmare especially installing SPICE+virtio drivers+ networking. I read in Phoronix the performance of virtualbox is subpar and all KVM/qemu is the way to go.

Please, could the OSS community build a easy to use GUI that SPICE+virtio drivers+ networking is made simple. Any one from qemu, I really appreciate the work but still very challenging for sysadmins.

I confess that I haven't run Windows with virt-manager on my own desktop in a long while, but it worked just fine after installing the required drivers at install time. IIRC Redhat distributes signed drivers for Windows for free.

The problems I did have were with VMs I tried to convert from VMware to KVM, and those stemmed from Windows' inability (version 7) to properly handle changing hardware. Perhaps it's better at nowadays.

Did this recently. It runs ok, but input and responsiveness latency is very noticeable.

Virt-manager seems to be written in Python. So if libvirt supports the bits you want then with some tinkering in GTK you might be able to bolt the controls onto Virt-manager, at least to get things started.

About 12 years ago when I had access to Microsoft MSDN I got a big SuperMicro box, set it up with CentOS 4 (RHEL) Linux, and used QEMU/KVM to set up a Windows domain controller and several other Windows VMs, and several Linux VMs, on a couple of virtual subnets, along with iptables port forwarding, Windows Remote Desktop, VNC, and many services. Some of the Linux VMs services authenticated with the Windows domain. I wanted to set up at least one non-x86/x64 emulated-CPU VM but never got that working very well in QEMU.

The exercise was a good (and incremental) learning experience and increased my general networking and sys-admin knowledge, which I've used well as part of my software engineering career. I would have been lost in many professional situations without it. QEMU/KVM are awesome.

EDIT: more and better words

So basically what you're saying is RTFM, in a polite fashion :)

Install the guest tools for windows from here

https://www.spice-space.org/download.html

It will contain everything you need for a fairly seamless experience on windows, and libvirtd and virt-manager make host side a cakewalk as well.

TBH, I tend to give up on the accelerated video drivers on Windows inside of virt-manager. It is often easier to just use RDP for basic office applications.

I mainly use windows + libvirt for gaming. But I also have an instance on a dedicated server for a remote development environment.

I would say forget spice + libvirt graphics layer. Use RDP, if on linux remmina work's great. RDP will outperform in my experience SPICE. I've also had constant troubles with the vga/qxl/virtio graphics.

Secondily for virtio: https://docs.fedoraproject.org/en-US/quick-docs/creating-win...

There is an iso at the above link. Grab that and have it added during install. When you get to the select a drive. None will show up until you load the virtiostorage driver.

There are a lot of nuances to first setting up a windows vm. But the performance is great. It has removed the need for a windows install for gaming at all.

Rather than using QEMU by itself, it might be advisable to use libvirt..? As somewhat of a newbie to virtualization, that's how I started - googling how to use libvirt and virsh, and I found a lot of resources that helped me to deploy what I needed... What are your painpoints in particular? I.e. which topics would you like to explore deeper and can't due to poor documentation/resources? We could work on improving the resources.

> What are your painpoints in particular?

I believe his pain point is that with Virtual Box you get a very simple GUI where you can get VM up and running very quickly without having to use Google or read any guides at all.

For that, I use 'Virtual Machine Manager' [1], which works for me extremely well, and is available for major distros.

[1] https://virt-manager.org/

Well, qemu + libvirt + virt-manager works pretty well.

Virt-manager adds a very nice VirtualBox-style GUI front-end to libvirt.

You can use QtEmu (https://gitlab.com/qtemu)

I personally dislike libvirt a lot, and virsh in particular. I think that it's pointlessly complicated and burdensome to use, and that in general the hypervisor-agnosticabstractions provided by libvirt are nullified by the fact that no backend except Qemu really works well, this making them a pointless exercise in futility.

I agree with you whole-heartedly. Instead of running one simple qemu command for each vm you instead have hundreds of lines of xml split into many different files. Files you need to run through libvirts session manager that needs to implements its own pretty pointless state management (this could be handled by systemd instead). If there is a nice feature in qemu you want to use, not only do you need to know how to define it in libvirts xml, it might not even be supported at all.

My recommendation for that sort of use would be to look at using a 'management layer' app that sits above QEMU and handles all the fiddly command line details for you -- https://virt-manager.org/ (which uses/is part of libvirt) is a popular one, I think. The QEMU project has kind of settled on the philosophy of being the low-level tool which provides you "full manual control" and interfaces for a higher-level UI to make use of. User-friendliness is not really our forte and so we let other projects provide that part.

This is the context I was missing when I switched to QEMU. I had not heard about libvirt. A simple "apt search qemu" had shown a few unsatisfying GUIs, but nothing about other layers above qemu.

I suggest QEMU's documentation could mention libvirt and its user-friendly interfaces.

FWIW, QEMU's upstream README does mention libvirt and higher-level managment tools:

"QEMU aims to fit into a variety of use cases. It can be invoked directly by users wishing to have full control over its behaviour and settings. It also aims to facilitate integration into higher level management layers, by providing a stable command line interface and monitor API. It is commonly invoked indirectly via the libvirt library when using open source applications such as oVirt, OpenStack and virt-manager."

But I get it, not everyone goes in search of a README buried in a Git repository. I myself am not sure if this REAMDE is rendered in an HTML format somewhere.

[1] https://git.qemu.org/?p=qemu.git;a=blob_plain;f=README

Modern GUI for QEMU: https://gitlab.com/qtemu

Note that libvirt isn't Qemu-specific; it supports multiple backends, including VirtualBox and Xen.

This reminds me of: One clear documentation page is worth thousands hours of man work.

> I lack the competence to seriously contribute.

actually, having gone through the process of successfully setting up the vm’s make you eminently suitable to contribute documentation knowledge that is sorely lacking in the online resources you have pointed out.

i am sure it will be greatly appreciated and will get improved upon areas that are missing etc.

Launching QEMU directly is not merely "tricky", but incredibly inefficient and ineffective once you need something that's beyond bare minimal.

E.g. here is a minimal QEMU command-line that gives you access to a serial console in the guest:

`qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio`

Simple, yeah? Now here (it's too long to post in this comment) is a real world QEMU command-line (as launched by the libvirt[1]):

htttps://kashyapc.fedorapeople.org/Fedora-28-QEMU-command-line-by-libvirt.txt

Compare and contrast the number of devices that libvirt adds.

FWIW, I strongly recommend to use libvirt to launch your QEMU-based guests. It is far more effective, and more importantly, provides security infrastructure like running the QEMU process as an unprivileged user, protecting the QEMU process (and its associated disk iamges) via the SELinux-based sVirt mechanism, and so forth.

Check out these[2] slides (recording is online, too) on "Security in QEMU" from one of the QEMU maintainers, Stefan Hajnoczi, at last year's KVM Forum in Edinburgh.

[1] http://libvirt.org/

[2] https://vmsplice.net/~stefan/stefanha-kvm-forum-2018.pdf

> like running the QEMU process as an unprivileged user,

Anyone can run your example "qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio" from the command line/bash, as non-root user, yet libvirt which requires root to install, and is expecting root to configure it, is better? How?

Libvirt is one of those softwares which ads a layer of complexity for the sake of complexity, yet offers nothing of value in return.

Indeed. Instead of contributing a patch to qemu with better defaults, or even support for config files, someone wrote tens of thousands of lines of complex code.

People forget that software is malleable. Bend it to your needs, instead of adding fragile wrapper layers.

libvirt-the-software is tens of thousands of lines of complex code that doesn't really add much. libvirt-the-project does contribute patches in to qemu, and a lot of the reason that libvirt-the-software doesn't add much of value is because libvirt-the-project worked to get capabilities in to upstream qemu.

The libvirt team definitely does view qemu as malleable, and bends it to their needs.

The problem is that the libvirt authors contributing to qemu look at qemu as a machine-interface (to be consumed by libvirt), not as a human-interface. Many of the more poorly documented or otherwise difficult-for-humans flags were contributed by the libvirt team. And they're useful! They make it so qemu can do things it couldn't do before. But they're also kinda the reason qemu is difficult to use without libvirt.

Good observations! I can't say I like it when a wrapper (loosely said here) starts to exert too much influence over what it wraps.

its not true that libvirt adds nothing of value, it adds easy networking support, easy snapshots, a nice gui interface, a very nice remote protocol and clients (that seamlessly support both graphical and serial output, aswell as on the fly reconfiguration of VMs without restarting them), template VMs (including one time initial provision setup), easy support for storage pools including remote ones, an easy way to manipulate RAM usage on live VMs, easy PCI and USB passthrough, live migration to other hosts, etc etc.

Sure its possible to do all this without libvirt, but it comes as one integrated, easy to manage package instead of hundreds of disparate parts.

You might not need (all) these features, but to claim libvirt is useless is extremely parochial.

Its also untrue that it needs root, you can virt-manager for instance without a root libvirtd instance at all, and it will run VMs without any permissions needed.

> its not true that libvirt adds nothing of value, it adds easy networking support, easy snapshots, a nice gui interface, a very nice remote protocol and clients (that seamlessly support both graphical and serial output, aswell as on the fly reconfiguration of VMs without restarting them), template VMs (including one time initial provision setup), easy support for storage pools including remote ones, an easy way to manipulate RAM usage on live VMs, easy PCI and USB passthrough, live migration to other hosts, etc etc.

everything libvirt does can be done without it by talking to qemu directly. and yes, Ive done many of the above "by hand", it was easier to read man qemu and other qemu docs than to swallow libvirt.

No, not everything it does can be done by qemu directly, plenty of it lies outside of the scope of QEMU entirely, like managing networks (not just network devices), or storage pools, setting up permissions for device redirection, or doing the legwork for VM provisioning and configuration (including net booting and automatically running appropriate scripts), or managing access to VMs both locally and remotely.

Its fine if your use cases are covered by plain QEMU, but libvirt does a hell of a lot more than plain QEMU alone is capable of.

Again all of that is possible via other means, but when other means involve recreating half of libvirt in shell scripts or other code, id rather let somebody else do the work for me.

QEMU is designed to run virtual machines, libvirtd is designed to manage VM server infrastructure.

The alternative is not that many commands: https://blog.elastocloud.org/2015/07/qemukvm-bridged-network...

For one static network, what about complicated inter VM networking schemes (including firewalling) that need to be dynamically adjusted as VMs go on and offline?

Libvirtd is much better suited to managing that sort of complexity.

I have the luxury of opining that such complicated networking schemes should be reined in, and not served with baroque management software. Complexity is the arch enemy of security, because systems must be kept easy to reason about.

> Libvirt is one of those softwares which ads a layer of complexity for the sake of complexity, yet offers nothing of value in return.

Isn't that sort of like saying the system startup scripts/subsystem adds complexity for the sake of complexity, and you should just use ifconfig and route manually in some script after startup?

In both cases, that you can fall back to the underlying commands is great. That you should use them instead of the normalized interface provided is debatable.

Unless the default changed one of these last few years, I'm pretty sure if you don't use `-enable-kvm` then qemu emulates the selected architecture. That's probably why it doesn't need root.

EDIT: Nevermind, `-enable-kvm` also doesn't need root. I'm not sure what libvirt uses root for.

Hear, hear. QEMU is excellent software, and I don't find that libvirt adds any value to its use.

Instead, one of the more annoying things I have ever done was foolishly installing libvirt and having it do all kinds of whackiness to a previously excellent QEMU set-up. I still cannot figure out where it managed to change all QEMU invocations to be VNC-based.

There's a lot to be said for running qemu from the command line, vs the complexity and opaqueness of libvirt. For example you can put the invocation in a script and check it into version control, and it will work on anyone's Linux box. You can easily debug the config and be suire about what options qemu was given.

Yes, there may be many switches in the invocation, but at least they are in one place, the invocation syntax rarely changes and works accross distributions, and they are applied from scratch in each invocation rather than having lots of implicit state hidden in daemons and their configurations / databases.

Your example is nicer on the eyes written in a config file style and leaving out the unnecessary options:

  qemu-system-x86_64 \
     -display none \
     -m 2048 \
     -serial stdio \
     -drive file=/export/cirros.qcow2,\
            format=qcow2,\
            if=virtio

I can see the value in "putting the command-line in a script and launch it anywhere". Many of us use bespoke QEMU command-line scripts for test and development. And I even know a small hosting provider who manages their production guests entirely via QEMU command-line "config file" syntax (-writeconfig and -readconfig options—they're little-known, not least because they don't cover all the options; upstream has some proposals to address it).

I appreciate that libvirt may not be a suitable option for some. That said, you might want to consider the below points on where it adds value (for production VMs; not talking about test and development setups):

• It's not about having all the QEMU "switches available in one place". The dizzying array of switches make it trivial to let you shoot yourself in the shoulder (e.g. getting fiddly details like PCI device addressing right manually).

• Knowing the "best practice" command-line will get you optimal performance out of your (production) guest—libvirt handles that for you. And if the best practices change, libvirt will automatically handle that for you, as it keeps track of "QEMU command-line best practice evolution".

• Launching the command-line is only the first step. If it's a production guest, at some point you might need to live-migrate the guest, or take a live disk backup, or if your guest has a long disk image backing chain, you want to "merge" the disk images into a single, coalesced image—all of this without the guest going offline. These tasks can be done manually (assuming you launched QEMU the 'right away' upfront) via extremely tedious and error-prone ways using QEMU's JSON-based run-time interface, QMP (QEMU Machine Protocol). But that's too risky and a colossal waste of time.

• Speaking of QMP, the run-time interface, it has even more dizzying array of commands—to be more precise, as of 2015: QMP had about 126 commands + 33 events. And more than 700 named arguments and results. Try keeping track of all of that manually. It reminds me of how once a QEMU sub-maintainer memorably compared (in his 2015 talk, "QEMU interface introspection: From hacks to solutions") the volume of QMP schema to old religious books: "QMP schema is larger than "Gospel of Luke", but smaller than "Genesis". :-)

IOW, I'm yet to see anyone who launches multiple guests (on the moderate scale of hundreds of VMs) use direct QEMU in production. There are odd examples, as I noted earlier, but it's certainly not the norm.

You're splitting the option value for -drive into multiple arguments. Here's an idea to fix that while keeping the same style:

  qemu-system-x86_64 \
      -display none \
      -m 2048 \
      -serial stdio \
      -drive "$(printf "%s"
          file=/export/cirros.qcow2, \
          format=qcow2, \
          if=virtio
      )"

If you don't mind using bash:

    args=(
     -display none
     -m 2048
     -serial stdio
    # -disabled-option
    )

    qemu-system-x86_64 "${args[@]}"

The array syntax means you can ditch the tedious backslashes and easily comment individual lines.

Ah, indeed. Thanks for the nice trick to avoid the tedium.

You're right about the ability to comment, but it doesn't save you from the tedious commas:

  drive_args=(
      file=/export/cirros.qcow2,
      format=qcow2,
      if=virtio
  )

  qemu_args=(
      -display none
      -m 2048
      -serial stdio
      -drive "$(printf "%s" "${drive_args[@]}")"
  )

  qemu-system-x86_64 "${qemu_args[@]}"

Personally, I prefer it with the backslashes if I don't need to comment.

Sometimes I feel like all I do is plug Nix.

But NixOS has a neat feature where any (?) NixOS configuration can trivially instead be built as a "vm", which will assemble an initrd for the described system along with a script which will run a tailored qemu invocation to launch the system, leaving a qcow2 file of any changes you make to the filesystem in your cwd. Makes the whole thing extremely usable, and is done in a very extensible way.

> minimal QEMU command-line"

> -nodefaults

Once you add `-nodefaults`, the list of arguments grows because now you must add a bunch of things that were there by default. If by "minimal command-line" you mean "the simplest list of arguments", then your example is wrong. If by "minimal command-line" you mean "command-line that gives the minimal VM", then you're being misleading, because that's not how people will take it.

Most of the devices added by libvirt would be included by qemu automatically if you didn't say `-nodefaults`. libvirt uses `-nodefaults` and specifies its own defaults, so that it doesn't need to keep track of changes in Qemu's defaults between versions of Qemu (since libvirt cares about the nitty-gritty more than most humans do).

PS: You have too many "t"s in the "htttps://" in one of those URLs.

This. Also, most of the example command line has nothing to do with serial consoles.

> Once you add `-nodefaults`, the list of arguments grows because now you must add a bunch of things that were there by default [...]

Partly true (refer below).

> If by "minimal command-line" you mean [...]

I appreciate your corrections, but please don't read too much into the word "minimal". I just quickly pasted it from one of the "subjective scripts" named `min-qemu.sh` lying around on my file system. By the time I realized, it was too late to adjust it.

> Most of the devices added by libvirt would be included by qemu automatically if you didn't say `-nodefaults`

Not quite; I just double-checked with an upstream libvirt dev: when you remove `-nodefaults`, some of the built-in devices get removed, whether those equate to "most" or not depends how many devices you've asked for.

> PS: You have too many "t"s in the "htttps://" in one of those URLs.

Yeah, realized too late that there was a typo :-(

There's not really much choice than trying to use qemu command line directly on windows. I didn't find any single useful GUI that was usable or newer than 3 or 4 years ago. It's a pretty sad state of affairs. I spend hours until I discovered that I needed to add some magic flag called machine for acceleration even work.

> E.g. here is a minimal QEMU command-line that gives you access to a serial console in the guest:

> `qemu-system-x86_64 -display none -no-user-config -nodefaults -m 2048 -device virtio-scsi-pci,id=scsi -device virtio-serial-pci -serial stdio -drive file=/export/cirros.qcow2,format=qcow2,if=virtio`

That's far more than I typically put in my qemu calls. You can technically be fine with just:

  qemu-system-x86_64 foobar.qcow2

but I always add `-enable-kvm` and use `-monitor stdio` or `-nographic` depending if I want a graphic display from the vm or not. `-nographic` is all I need to give me access to a serial console in the guest (besides configuring the guest kernel with the paramater `console=ttyS0`). I also add `-m` because the default amount of memory is just 128 MiB, which is typically not enough for my use. Sometimes I add network related arguments, but we're going beyond minimal then.

When I was running a Fedora desktop and was doing a lot of dev/test work on VMs+Foreman+Puppet, I used Libvirt+QEMU as the VM hypervisor.

I ran Foreman inside a CentOS VM, which talked to my desktop's Libvirtd to provision new VMs, hand out IPs via DHCP, boot via TFTP, then kickstart. It worked pretty well, and helped me to spin up a vcenter foreman controlled dev environment for a few hundred users in just a few days.

My recommendation would be Vagrant + Libvirt.

You can always document your experience, what worked, what didn't work, things you are still unsure about.

We need to spin up a lot of VMs of various Linux distributions for testing Flockport and we found the existing tooling too heavy and involved.

In the end we rolled our own solution that is much more lightweight, uses QEMU directly and provides better visibility, ease of use and even some potentially useful networking features. Here are a few screencasts [1].

This is mainly designed as a module for Flockport and we will preview this soon.

[1] https://www.youtube.com/watch?v=uEnk9m-egeA

I'm impressed nobody mentioned libvirt (https://libvirt.org/) here, an open source framework for managing all kind of virtualization platforms. It is especially well suited to take off all heavy work of managing QEMU. From the features, Virt-Manager (https://virt-manager.org/) can compete with VirtualBox -- and is probably well beyond, thanks to network transparency, the Spice protocol, real guest console access, powerful CLI interfaces such as `virsh` and other goodies.

In Linux distributions, you basically install `virt-manager` and the package manager will resolve all dependencies to provide you a fully working virtualization desktop software which is well capable to be used on servers.

Fun fact: Virt manager (plus the good stuff like e.g. splice) runs also natively on macOS and makes it easy to control remote Linux VMs:

https://github.com/jeffreywildman/homebrew-virt-manager

Precisely so. Couple of us have mentioned libvirt as a recommendation in the comment threads above:

https://news.ycombinator.com/item?id=19736609

https://news.ycombinator.com/item?id=19736722

LTT [1] did a video recently about running macOS on a Linux host using QEMU. Anyone here tried that? If so, what's the experience like? Thanks

[1]: https://www.youtube.com/watch?v=ATnpEOo3GJA

I do this using Proxmox (QEMU under the hood), and it's been mostly a nice experience. I currently run 3 VMs on a single Linux box with 3 GPUs (one per VM via VFIO passthrough). I've used both NVIDIA and AMD graphics cards with success. Before Mojave, NVIDIA cards were easier with their web driver, but Apple and NVIDIA don't get along in Mojave (both companies are blaming each other) and there is no web driver. So for macOS, I've switched to an AMD GPU which works mostly out of the box, apart from needing a small tweak in the Linux host's kernel parameters ("disable_idle_d3=1" to avoid a bug preventing the device from being used after a VM restart).

To get macOS to see my GPUs with full acceleration and audio over HDMI, I use the Clover bootloader to inject a small section into the DSDT, mapping the emulated PCI address to a device that actually looks like a graphics card to macOS. This same DSDT patch works for both NVIDIA and AMD GPUs (the ones I've tried anyway).

The best thing about Hackintosh in a VM is that you no longer need to fear macOS upgrades. Just create a snapshot, try the upgrade, and if it doesn't work out, you can restore, investigate, and try again.

LTT's video is a nice overview, but his claim that "because this is a VM... it will run on any hardware" isn't quite true when talking about GPUs. Unlike the CPU, memory, and some other devices, the GPU is seen (nearly) directly by the guest VM, so guest OS support is in fact required to see the advantages of GPU passthrough.

In summary, it's never going to be super easy, but it's not too difficult either, and the performance (and flexibility) is actually really great.

Thank you very much for sharing your experience.

Thats sounds pretty interesting. Ive been running a hacked together windows VM gaming setup via qemu with vfio passthrough for the past 1.5 years on my local linux server. Im pretty pleased with it. Ive got it down to simply executing a small script and everything comes up. It could be even more automated using evdev potentially (recognizing mouse/keyboard events).

I was thinking about expanding the setup to include a beefier linux host (more gpus and better hardware) and more seats -- windows gaming for me and osx for my wife. But you mentioned several possible osx specific caveats: amd gpus, clover (?) bootloader w/ dsdt (?), disable_idle_d3, etc. Can you link to some info or READMEs regarding your setup or how you came across these solutions?

I don't have any specific guides about my setup, but here are a few things that have been helpful to me:

- Good starting point for setting up macOS under QEMU/KVM: https://github.com/kholia/OSX-KVM

- DSDT/SSDT patching: https://www.tonymacx86.com/threads/ssdt-gpu-graphics-card-in...

- AMD reset bug: https://www.reddit.com/r/VFIO/comments/5h351m/gpu_stuck_in_d...

- Kernel extension that can help with some GPU issues: https://github.com/acidanthera/WhateverGreen

The DSDT/SSDT patch was probably the trickiest thing. I've posted my particular patch here if you're interested: https://pastebin.com/ngvkVZYN

Also, a few of my other scripts/config: https://pastebin.com/9Nh5rheZ

Thank you for the links. Time to build a new server!

In case anyone runs across similar AMD GPU issues, the "disable_idle_d3=1" option in the second pastebin link (when used as a kernel parameter) should be prefixed as "vfio-pci.disable_idle_d3=1". Even then, it can still be hit-or-miss with some cards (it works most of the time for me, but some VM restarts still find that the GPU has entered D3 and won't come back without rebooting the host).

is this a macOS guest specific issue? would I have similar issues on a Windows or Linux guest?

I believe it happens for any guest OS (search for "amd reset bug"). Some people are using a workaround that involves removing the device (in the sysfs), suspending the host, and then rescanning, but for me, that's not much better than a host reboot, so when the AMD GPU occasionally does get stuck in D3 (preventing starting the VM after a VM reboot), I just reboot the host. It doesn't happen very often, and when it does, it's only when I'm in the process of rebooting the VM anyway.

Previously, I also attempted using a kernel patch that's been floating around (DECLARE_PCI_FIXUP_HEADER ... quirk_no_bus_reset), but it just gives me corrupt graphics when rebooting the VM.

I am building a VFIO build this weekend. one of the goals is to power a macOS VM for work. if I remember I will update you on my experience.

I had success getting macOS running on QEMU with my existing PC, but its performance suffered without a dedicated GPU and CPU pinning, which my new build will be configured to handle

Thank you, hope your build goes well!

Note that this isn't a "major" change from v3.x.x; the major version is year-based (${year}-2015). This is v4.0.0 because it is the first release of 2019.

QEMU's killer feature is user mode emulation [1]. It's never been easier to test/experiment with cross-target binaries.

[1] https://wiki.debian.org/QemuUserEmulation

Exactly! We are using this in our compiler test farm for the quick test runs. Sadly, the release prior to 4.0.0 broke floating point handling of NaN on (at least) PowerPC e5500 :( I'm looking forward to test qemu 4.0.0 once I'm back in the office again.

Edit: Tested it, aaaaaand NaN compares are still broken :(

Anybody here using QEMU? What is your use case?

I used it before to dabble in BIOS coding. Just for fun.

Screenshots? :)

Virtualbox doesn't work with Linux 5.1 so I moved a bunch of VMs to Qemu.

Huh, surprised it broke that badly.

But support appears to be there as of a week ago, fyi.

It does now for me at least in version 6.0.6.

Interesting, I installed 6.0.6 and got issues generating the kernel modules.

Mainly playing with old operating systems, I'm a retrocomputing enthusiast.

While years ago I would mainly use Virtual PC (I do miss its great GUI) and VMWare (and rarely bochs), qemu improved a lot and has (somewhat decent) support for so many more architectures. You can run Solaris for SPARC and Mac Os 9, for example.

I'm interested in your is cases. Do you have any good write-ups for Mac OS 9 and others? I never considered QEMU for those older systems.

Well, usually the internet is large and you can find lots of guides :)

I'm an avid follower of virtuallyfun.com and right now the latest post is about installing AIX on QEMU. If you look into the qemu category ( https://virtuallyfun.com/wordpress/category/qemu/ ) you'll find OS X Server, Unixware and more.

The author, neozeed, is very friendly and you can leave him comments or shoot him an email and he usually answers pretty in depth.

As for OS 9, I'm not sure it is as refined and functional as SheepShaver on Windows yet, but I'm on OS X, where SheepShaver takes more effort to set up. There's a modified QEMU for enhanced OS 9 compatibility (I think more G3 emulation?) here, and people managed to install 9.2: https://www.emaculation.com/forum/viewtopic.php?f=34&t=9028

"Solaris for SPARC"

Link?

https://astr0baby.wordpress.com/2018/09/22/running-solaris-2...

Or do you mean a copy of the ISO? For that I'd try archive.org or winworldpc

We build our own Linux images (using Yocto) for various embedded devices and run the same images on QEMU for testing purposes.

Is there any great resources to learn QEMU for ARM embedded system? I googled them bit it's a little bit mess.

I'm using it extensively, but as a build tool: https://github.com/multiarch/qemu-user-static

(See https://taoofmac.com/space/blog/2019/04/21/2300 and https://github.com/insightfulsystems/alpine-python)

Same here. Use it to build tools targeting arm architecture (rpi, etc) on a x86_64 machine, that too in docker.

My experience is that building for ARMv7 is relatively quick, but ARMv6/armhf is MUCH slower. I have no idea why, really.

Will be trying ARM64 soon, as soon as I have a suitable development board.

i use qemu/kvm almost everyday. i also use qemu-img tools quite often. and as other people have mentioned, building inside qemu is quite convenient, especially with tools like packer.

I'm on NixOS and QEMU is well integrated in Nix.

You can write a system configuration, generate a QEMU VM and run it. Also all NixOS automatic tests[1] are running in QEMU VMs that are generated from a Nix configuration.

The most beautiful thing is that you can create multiple VMs and run them simultaneously to create a virtual network. For example you can test a client-server application by running the client and the server on two hosts or test some network configuration[2].

[1]: https://nixos.org/nixos/manual/index.html#sec-nixos-tests

[2]: https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat...

You use NixOS as your daily operating system? What’s that like?

I've been using it on my personal laptop for the past two months.

There are a lot of rough edges to the desktop experience, but it generally works. And while that's faint praise, I have no current plans to switch back to Fedora (3 years) or Debian (12 years prior to that).

It's comforting to know that my entire base system configuration is declarative, and it's pleasant to not worry about libraries conflicting or unnecessary packages lingering around.

Immediate, ephemeral access to programs via, e.g., `nix-shell -p pdftk` is extremely convenient: they're only available in that shell session, and they (along with their dependencies) get cleaned up the next time you run `nix-collect-garbage`. Super handy.

What do you use as a web browser?

edit: nevermind, I didn't realize it was a Linux distro.

What do you think about Darch?

https://godarch.com/

I use it to run aarch64 unit test builds on my x86 system. Pretty cool with binfmt support, allowing me to run aarch64 executables just like any other x86 executable: https://wiki.debian.org/QemuUserEmulation

QEMU is one of the core building blocks for many of the KVM-based data center virtualization and infrastructure projects (e.g. OpenStack, oVirt, KubeVirt, etc). And is used by a wide variety of other projects for various use cases (e.g. `systemd` upstream uses QEMU to speed-up development/testing).

When you launch a KVM-based guest, most often you would see QEMU doing the major user space heavy-lifting by providing a PC-like environment—it handles guest memory management, network cards, emulated storage devices, and so forth. QEMU also has a robust Block Layer; not to mention its versatile, and widely-used, native disk image format, QCOW2.

PS: I am biased, as I work with QEMU on a daily basis :-)

I have a NUC sitting in a closet that I've set up to serve virtual desktops using LXC or libvirt (depending on what features I need, or if it's a Linux or MacOS or Windows guest). I made some scripts to build the various types of virtual environments I need [1], and then connect to them using x2go or Chrome Remote Desktop.

This has really freed me up because I'm no longer tied to a single machine. My home dirs are mounted volumes that are also backed up, so I can tear down and rebuild the environment with a single command (which means that my environments are also deterministic). I can connect to them from anywhere on the planet, using any machine as a client (even my phone or a tablet). My main laptop is now a weaksauce 1080p chromebook that acts as a glorified thin client and web browser.

[1] https://github.com/kstenerud/virtual-builders

We've a shared beefy Threadripper workstation in my lab that has a boatload of RAM and cores to spare. We set up Qemu + KVM in order to run a big Windows VM that we use mainly for MS Office via RDP connections and when we really need Visual Studio. We also have a FreeBSD and a Aarch64 Linux instance that we use for testing. It's amazing how flexible Qemu is, and how reliable the whole stack can be.

Website browser compatibility testing on Windows/macOS under Linux.

I'm using it in conjunction with PCI passthrough instead of dual boot. I've been running Windows 10 for more than a year now and played all kinds of AAA titles successfully. The performance-loss seems minimal. Recently I managed to get a second GPU and spin up another VM, so now me and my partner can play on the same machine, heaving each a keyboard, mouse and screen.

Do you have any hints on how to set up a similar system? Links, guides, gotchas, whatnot? I've seen someone mention this setup before (perhaps it was you?) and I'm sure I'm not the only one interested in running something similar.

You might be able to find some info here https://www.reddit.com/r/VFIO/.

This seems very helpful, yes. Thanks a bunch for posting!

I have written a "guide" for myself to replicate my setup. It is by no means perfect, but maybe it helps: https://gitlab.com/snippets/1678554

An overall good introduction/guide is: https://github.com/saveriomiroddi/vga-passthrough.

The most complete guide is probably the Arch Linux one: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVM....

I posted about this on my blog here: https://1125.io/#/2019-01-21_21:58_Playing_within_a_Windows_...

Also I recently discovered some very in-depth blog posts here: https://heiko-sieger.info/iommu-groups-what-you-need-to-cons...

Another whole website about this topic (no joke): https://passthroughpo.st/

I do the same, I used to run my VMs off of ZFS with spinning disks, but that was too slow at times, so my last upgrade has been giving the VMs their own SSDs and that has worked out great, I then backup the VMs to ZFS and snapshot them.

My employer provides me with an iPad Pro and rather than carry around an additional laptop I run x86 systems (Linux and Windows 10) using QEMU-KVM on my ridiculously overpowered home server and remote desktop to them from the iPad Pro.

What’s your mouse/keyboard situation like?

What remote desktop app do you use?

What’s your latency like? What are the drawbacks to a setup like this?

For software I use Jump Desktop. On Linux I connect via VNC and Windows I use their custom Fluid protocol. For keyboard and mouse I use an Anker Bluetooth keyboard and mouse is a Swiftpoint GT.

Latency varies a lot and I’ve found hotel WiFi to sometimes be unusable. ATT LTE on the device generally works pretty well. I’m primarily running this setup just to have access to a full browser environment for my tinkering with web scraping so having a super responsive low latency connection isn’t as vital as it would be to do something like gaming. The main drawback is always needing a solid connection which is mostly solved with LTE where I go.

I currently have a messy side project[1] where I'm writing bare-metal RISC-V machine code (i.e. writing in hex after assembling instructions by hand) targeting the SiFive HiFive1. QEMU is handy for testing since it supports that board pretty well - it lets me skip the "code upload" step and just run a binary with gdb right away.

I did a similar thing with the Raspberry Pi a few years ago, but the QEMU support wasn't great back then which made debugging incredibly hard. I think the support has improved a lot since then, so I should probably give it another go.

[1] https://github.com/SgtCoDFish/bedrock-bootstrap - I'm writing the steps as a kind of tutorial to myself to aid learning, but fair warning that it's all a bit of a mess right now!

Recently I started playing with classic Forth systems like figforth and pygmy.

If you are using MS-DOS on QEMU there is a open source utility called DOSIDLE.EXE that keep the CPU usage low and stop the annoying CPU fan from spinning.

Running PowerPC Mac OS X on Talos (and OS 9, but this doesn't work with KVM-PR ... yet).

So that's OSX Tiger 10.4 with OS9 emulation layer you want to run on your Talos under a linux-ppc build of QEMU?

Running linux as Host and optionally starting my gaming vm. With KVM and various hardware passthroughs (pci network card, usb devices, gpu, etc.) it is as fast a native installation and sometimes even faster compared to my old bloated windows host. Ofcourse it was not easy to setup and some games needed tweaking (e.g. Just Cause 2 wanted to use unsupported cpu features and therefore needed an extra flag). But if one has the time and patience this is awesome.

I use QEMU every day for testing my new operating system: https://github.com/awesomekling/serenity

QEMU has just the right mix of performance and hackability that I need for my "printf debugging" workflow :) (I've used Bochs a lot, too, which is more hackable, but far less performant.)

Trying to run Linux and Windows guest on macOS host with hvf acceleration. Found that FreeBSD won’t boot perhaps due to lack of support of v86 mode IVT in hvf part.

Also trying to setup gvt-g with dmabuf but corner cases still need fixes, e.g. video playback doesn’t work on overlay sprite of yuv encoding.

Currently using it as part of a testing workflow in making my own linux distribution. It makes booting kernels with initramfs, as well as live images, super easy and painless. QEMU is great software! https://jrl.ninja/joy/1/

We use it here exclusively at my current place of employment in the OS and BSP team. We create pre-built, bootable ramdisk images inside a QEMU instance which are hashed and saved for fast provisioning, as well as having multiple QEMU instances on targets to prevent cross domain pollution (being mandated by NSA). I work in defense, for context.

I use qemu to run multiple virtualized machines on a single beefy (32-core) headless box; including a shared build-server, and my own remote-workstation.

I got sick of trying to figure out which thing I needed to click in virt-manager to get it to pass the flags I wanted, so I wrote my own minimal qemu-wrapper that covers all of the functionality from libvirt that I care about (vCPU pinning, binding to NUMA nodes, running as non-privileged user), and integrates better with the other system (systemd) monitoring than libvirt does. https://git.lukeshu.com/systemd-qemu/

----

I'm also involved with the Parabola GNU/Linux-libre distribution. Qemu user-mode emulation is essential to our support of non-x86 platforms (ARM and PPC). It lets us run cross-architecture chroots without a full-blown VM.

We use it with a wrapper[1] for continuous testing of an operating system[2]. The wrapper will perform the installation process as users do, then run the testsuite and upload the results. It's nice because it lets us catch regressions in the installation process, too.

[1] - https://www.gson.org/netbsd/anita/

[2] - http://releng.netbsd.org/test-results.html

I use it as a VirtualBox replacement via libvert - no thanks to Oracle setting it (VB) up as an IP licensing minefield. Some VirtualBox extensions are non-free in commercial environments, and installing them creates liability for your organization.

VFIO! I play videogames and use Lightroom. It's rock solid (at least, with my usage pattern), however, v4.0.0 hangs on boot, so I use v3.1.0.

I have a library that needs to target, among other platforms, big endian MIPS. It does a lot of endianness sensitive stuff but all the CI workers are naturally little endian, so I run the testsuite through qemu-user simulating a MIPS system.

I used QEMU for VFIO, GPU passthrough to a Win 10 guest for games. It worked really well. I passed an nvidia card, an ssd with the os/programs, and an hdd data disk from a Linux host (Gentoo).

Setting it up was somewhat difficult. When I did it there weren't many good guides for just exactly my situation, so I had to use many, many different resources to get it running. Based on some of the links others have posted here, there seems to be more out there on it now.

This was my last working startup script. I no longer recall why or even if any of these options were necessary or beneficial. At the time, it was just what worked. (Dual monitor, separate video cards for host and guest, L-CTRL+R-CTRL to pass mouse/keyboard control back and forth.) I haven't used this in quite a while, so I can't say if all of these options are still legit anymore.

  #!/bin/bash

  cp /usr/share/edk2-ovmf/OVMF_VARS.fd /tmp/my_vars.fd

  qemu-system-x86_64 \
        -enable-kvm \
        -m 16G \
        -cpu qemu64,hv_vendor_id=whatever,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,smep=off \
        -smp cores=8,threads=1,sockets=1 \
        -machine q35,type=pc -vga std -display gtk \
        -drive if=pflash,format=raw,readonly,file=/usr/share/edk2-ovmf/OVMF_CODE.fd \
        -drive if=pflash,format=raw,file=/tmp/my_vars.fd \
        -drive file=/dev/disk/by-id/ata-Samsung_SSD_860_EVO_250GB_XXXXXXXXXXXXXXX,format=raw,if=virtio \
        -drive file=/dev/disk/by-id/ata-ST3000DM001-XXXXXXXXXXXXXXX,format=raw,if=virtio \
        -boot order=d \
        -object input-linux,id=kbd,evdev=/dev/input/by-id/usb-XXXXXXXXXXXXX_USB_Keyboard-event-kbd,grab_all=yes \
        -object input-linux,id=mouse,evdev=/dev/input/by-id/usb-XXXXXXXXXXXXXXXX-event-mouse \
        -device virtio-mouse-pci \
        -device virtio-keyboard-pci \
        -netdev user,id=user.0 -device e1000,netdev=user.0 \
        &

I have not upgraded to the latest VMWare Workstation -- should I take a look at QEMU first or is this something meant for a completely different use case? (I use VMWare Workstation for contracting/consulting and build out a workstation to match the client engagement rather than pollute my base OS with five different versions of Visual Studio; IntilliJ, Eclipse, AND Android Studio; multiple version of SQL Server/PostgreSQL/Mongo/MySQL... and I can dump those VMs off to network storage when I'm done with the engagement and reclaim the storage space on my laptop.)

Yes, qemu can do that.

Can QEMU be used to run a small (classic) ARM binary with simulated RAM/Flash and can I somehow plugin my own peripheral simulations? On Windows.

I always wanted (but never succeeded) in emulating SPARC guests using QEMU. Is there a comprehensive list of systems and hardware emulated? I suspect a lot of people in the HN crowd would be curious about emulating Alphas, even if for only running Genera.

Having a collection of emulated Unix workstations is an upgrade in terms of space saved.

I got Debian for SPARC64 running under QEMU a few months ago. I wanted a big endian system for testing. I'll post the link on steps if I can find it. It was on GitHub IIRC.

My memory was faulty - I did MIPS on QEMU to get big endian.

The steps are here for posterity: https://github.com/kholia/mips-hacking

Here's a pretty good tutorial.

https://astr0baby.wordpress.com/2018/09/22/running-solaris-2...

I used a Solaris 7 .iso from archive.org.

https://archive.org/details/solaris_7_1199_sparc

Ooh really? That works?

Yes. Here's a screenshot of Netscape 4.76.

http://www.w6rz.net/netscape.png

That was with a Solaris 8 disk image that I found here:

https://github.com/cclark64/qemu_solaris_sparc

Has anyone got a Modula-2 compiler for old Solaris versions? Then I can re-live the monochrome (2-color) xterm experience from the 90s.

Note that the versioning has changed to time-based, v4.0 denotes that this is the first release of 2019.

I clicked on the story because of the major version number bump, then belatedly realised they had moved to time based version numbering. The major version number gets incremented every year.

So the version number is essentially meaningless now - the year of release (eg, QEMU v2019.0.0) would give you more information. I can't say I'm a fan.

I use Virtualbox on home PC to dev. I ssh into it and use tmux/vim as my IDE. Is there a better setup for performance on windows? Should I bite the bullet and get myself a Windows Pro upgrade for Hyper-V? If my memory serves well you Hyper-V can't have Hyper-V and Virtualbox at the same time.

True. I use vmware and my ubuntu vm is faster than my windows host. This makes hyper-v and docker on windows incompatible, though. I run i3 inside vmware. I would expect virtualbox to be ok too, for this use case.

You can use `docker-machine` and its vmware driver to use your VM as the docker backend

Did anyone manage to solve the problem of virgl rendering failing with

    Error starting domain: internal error: qemu unexpectedly closed the monitor
    ....
    libvirtError: internal error: qemu unexpectedly closed the monitor

It's pretty great and has been useful for many use cases for me, except too bad it lacks good 3D graphics support for modern gaming

It all depends on how and if someone will implement a Windows driver for VirGL. Given how complex video drivers are nowadays, it looks like a pretty daunting task.

The ReactOS folks might do it.

Intel has support for KVMGT, but AMD/NVIDIA won't support it. This would really be the correct approach to get support from the graphics manufacturers:

https://wiki.archlinux.org/index.php/Intel_GVT-g

There will always be a performance loss unless you go the extra mile and passthrough the gpu. It already works great, but is not a breeze to set up.

You don't really need VMs for gaming, at least if you are ready to ditch a few broken games that don't work in Wine. For the most part they work very well these days.

I don't use QEMU and I'll always confuse it with QEMM. So every time I see a QEMU article title I think to myself, What year is it again?

Reply on news.ycombinator.com