In Praise of QEMU | Hacker News Reader

In Praise of QEMU

drewdevault.com

242 pointsTomte72 comments

I agree.

Few years ago there was an uprise of blog posts explaining how to run Linux as a primary system and virtualize Windows machine with GPU passthrough for gaming. ~ 1 year ago I took a dive and did a lot of reading and attempts to make it work, the final results made me really happy. QEMU/KVM/VFIO is one of the best software combinations I've ever used.

  Witch CPU isolation and scheduling properly setup my Linux part of machine runs Docker with 25 services, while Windows part happily plays games without stutter at the same time.

  That said, quite a few gaming companies are anti-virtualization (with reasonings being that mostly those who use it are using it for cheating purposes) and ban your account, without rights to appeal (ex. Rainbow Six). I honestly find this stance blunt and it feels like discrimination. But currently such use of virtualization is really niche, so what one can do?

I've been using VFIO and maintaining a guide for a few years*, then I've realized that VFIO kinda works, but it's not a reliable technology, and I moved (back) to dual boot.

There are very significant pain points, specifically:

1. if one reserves the video card for VFIO, it won't have any power management; this means that it will run hot while doing nothing; in order to work this around:

1a. first has to battle with X, which has an option not-to-take-over-a-card-but-it-takes-it-over-nonetheless

1b. then one can give exclusive access to the graphic card driver, which can be switched out/in when starting/stopping the VM; this unfortunately works, but not reliably

2. the points above apply to nvidia; AMD is worse, as it hasn't supported soft GPU reset until very recently (I think it was added on 5.19 or so)

2a. this means that one starts the VM, then stops it, and most of the times the card will hang

2b. there resize BAR functionality is not supported by VFIO (at least, last year it wasn't), which means, one loses additional performance (I could be ok with it, as the loss is not significant, but performance losses compound)

The problem is that all the points above are not in control of the user; the problems happen at driver level (if, say, there is no reset support, one can't add it out of thin air).

If one uses Nvidia, and they're ok with the card running hot all the time, then definitely, VFIO works wonder. But this lead me to abandon VFIO, as I don't want that (and the alternative of the card having a most-of-the-time-malfunctiong driver was not appealing, either).

Big shame! I loved VFIO :)

* https://github.com/64kramsystem/vga-passthrough

I built my current desktop with the whole goal of being a tank of a VFIO setup and I have no regrets. Best of both worlds for me as I used to just have two separate computers.

QEMU/VFIO with Looking Glass is just such an amazing setup for my use cases.

Thanks, Drew, for calling out the author of QEMU and linking to his sight: Fabrice Bellard https://bellard.org/ I didn't know he had done so much! Incredible!

Calling out on what specifically?

"shout out" was what he meant.

Ah, thanks. I thought I was missing something.

He mostly seems to list his hobby projects on his website, not his paid work (except https://textsynth.com and I don't know what else). E.g. he is CTO of Amarisoft. His actual productivity is greater still.

From Bellard the things I like most are the "toy" projects. Qemu, tinyemu and such began like that.

His list of hobby projects includes QEMU and ffmpeg. I'm sure Amarisoft is a fine company, but mentioning them before those two would be burying the lede.

He also wrote QuickJs a complete Javascript VM.

https://en.wikipedia.org/wiki/Fabrice_Bellard

Wow.

QEMU is great, I'm just annoyed by its UI. I'd much rather have a config file I can point it towards than an endlessly long shell script I can't comfortably use comments in (maybe this is a criticism of shell scripts in general).

I guess libvirt would be the "docker compose" front-end, but I remember considering that more complex than would be necessary.

I know Proxmox has some QEMU management tools (which I find decent), but they don't seem to be very popular outside of it.

One cool QEMU thing I recently discovered is the microvm machine type (https://qemu.readthedocs.io/en/latest/system/i386/microvm.ht...), which enables it to spawn VMs just as quickly as Firecracker!

It would be nicer yet if there was some way to say "give me a shell in this ephemeral VM" like docker does it, but sadly Kata Containers appear to have dropped docker support and I haven't been able to get it to run with the alternative runtimes.

Virt-Manager (and virsh) has option for both config files and scripts creation.

> I'd much rather have a config file I can point it towards than an endlessly long shell script I can't comfortably use comments in (maybe this is a criticism of shell scripts in general).

I end up designing a lot of CLI tools and this is a design problem I run into a lot with my own stuff: how many flags are too many? Introducing a config file works, but now it makes your tool much less feasible to run in a shell script but, like you say, too many flags makes your tool unwieldy to run anyway. The problem is harder if you have a venerable tool that's always been flag heavy. Do you keep lumping flags in or do you steer the community toward the use of a config file? Or a hybrid?

OpenVPN's approach is to have every config key imply a valid config file entry and a long-arg. So if you see the line `tun-mtu 1500` in a config file, you know you could use the arg `--tun-mtu 1500` on the command line, and vice versa; if you see somebody use the CLI arg `--proto udp` you know you could put the line in your config file `proto udp`.

I think mysqld has a similar approach, though I don't think it's as complete. I'll bet both of those projects would answer that there's no such thing as a tipping point where you tell users to use a config file. They're just two ways of doing the same thing; use whichever makes sense for your deployment.

That's an interesting approach.

The mpv media player takes a similar approach.

Several Python tools (Mypy, Black) also use this approach and I like it a lot.

I like tools that have 1:1 mappings for config file and command line flags because, as you say, both have advantages and disadvantages.

Zsh (and to a lesser extent Bash) has support for arrays, which you can use to represent long command line invocations in relative safety. E.g. instead of this sort of madness:

    #!/bin/sh

    set -eu

    # Argh I can't write comments in between lines... just email me if you have q's: nerdponx@mail.example.net
    exec /usr/sbin/boop \
      --enable-foobars \
      --this-is-a-really-long-option \
      --sound=quack:.:loader=org.example.soundfactory-lib.SoundFactory.wrapped_loader \
      --sound=woof:.:loader=net.quibbler.QuibblerWoofMaker \
      --jvm-opts='-Xms512M -Xmx1G -Xquux=1234 -Xbootclasspath:.'

But you can preserve what's left of your brain cells without switching to a "non-shell" language if you use Zsh, and then you don't have to sweat over shell quoting and parameter expansion either:

    #!/usr/bin/env zsh

    # Enable "strict" features for typo-safety
    emulate zsh
    setopt err_exit pipe_fail warn_create_global warn_nested_var no_unset

    boop_args=(
      # We always want foobars, why isn't this on by default?
      --enable-foobars

      # See: https://stackoverflow.com/questions/4973
      --this-is-a-really-long-option

      # Use the good sounds! Please see README for obtaining these deps.
      --sound=quack:.:loader=org.example.soundfactory-lib.SoundFactory.wrapped_loader
      --sound=woof:.:loader=net.quibbler.QuibblerWoofMaker

      # Trust me, don't touch these.
      --jvm-opts='-Xms512M -Xmx1G -Xquux=1234 -Xbootclasspath:.'
    )

    exec /usr/sbin/boop ${(@)boop_args}

aaah, that's a great idea for interleaving comments, I'll have to use that! thanks!

previously I've done this, which works, but is a bit....... yeah.

    some command \
    `# did you know comments work inside backticks like this?` \
    --complex_flag_needing_descriptions=... \
    `# literally any ffmpeg flag` \
    --lol no

now if only shellcheck would support zsh....

For completeness, the other option is:

    CMD="some command"
    # Some comment
    CMD="$CMD --verbose --complex-flag=..."
    # Other stuff
    CMD="$CMD --lol=\"no thanks\""
    eval $CMD

yeah, I just don't like dealing with the possibly-multiple-levels of escaping that sometimes requires :\

there are ways to deal with that of course, but it's kinda unavoidably more complex.

> now if only shellcheck would support zsh

Since the default zsh behaviour is much more reasonable (e.g. just $foo won't do globbing or word splitting) and you don't really need POSIX compatibility if you're going to use zsh-specific features, the need for shellcheck is hugely less than what you need with bash or POSIX sh.

Still, it would be pretty handy. For example a good Zsh linter could warn you if you use a Bashism that isn't valid in Zsh, or it could switch to Posix mode if you set `emulate sh`, or warn you when an autoload function file is missing its function call at the bottom, etc.

Sure, but that's "only" useful warnings/errors of that kind that are useful for any language, rather than footguns that work on first approximation and then don't ($*, unquoted $@, unquoted $var, stuff like that). A zsh LSP would be nice as well.

Very fair! I bet you could use the C "module" interface to write the Zsh LSP in Zsh itself...

There's https://github.com/zsh-users/zsh-syntax-highlighting which does some parsing, but I never looked at the details of it (I just know it exists).

I ran OSX on QEMU/KVM on a Ryzen emulated to be an IceLake Intel cpu and got close to ~90%+ performance of my native machine according to geekbench.

So my QEMU OSX instance got better performance than a Mac Pro max, very well near the top ~5 scores for macs in geekbench.

Props

How did you get the Mac VM created? I tried this once but eventually gave up because whatever path I tried it was challenging at every point to create a macOS VM from scratch.

Checkout https://github.com/kholia/OSX-KVM. The README is pretty complete.

I'll try it out, thank you!!!

What version of MacOS? Did your IceLake emulation require OpenCore and modified kexts or did you achieve by some other process? Were there any notable compatibility issues with any specific hardware or software?

Take a look here https://github.com/kholia/OSX-KVM. Post install, I use the "Opencore-Boot.sh" script to boot the machine and inside of it modify the cpu "Penryn" to Icelake one assuming you're running a Ryzen one.

I did get some warnings from QEMU along the lines of "X register doesnt exist" or something but the machine runs fine and geekbench runs fine as well. Play with the cores/threads settings as well. If you go too high (for me was around 8), it doesn't boot up and just hangs.

The QEMU project and it's developers are responsible for making my research and hobby OS development on RISC-V and Aarch64 platforms possible, where I learned so much at a pace I otherwise couldn't. Thanks.

Agreed. I'm currently using qemu and gdb from an x64 Linux system to debug an aarch64 interrupt problem I'm having. Not the sort of thing you can easily debug on real hardware. I've done similar things with VMware, but as far as I know, that can't do cross architectures like this. It's pretty amazing.

Yeah embedded debugging is an absolute PITA, even with OpenOCD + JTAG exposing a remote gdb server to the target it's just always been so fragile in my experience. Like reloading a changed executable causing gdb to segfault sometimes, and you need to restart every part of the debug chain (device being debugged, openocd, gdb) in order to get back to working on the problem you were originally trying to solve.

I hadn't seen OpenOCD yet, so thanks for that. On the boards I'm currently playing with, while they technically have JTAG, it's pin muxed with pins used for the sd card. So I'd have to build a new u-boot to reconfigure the right pins, and get an adapter to break the right jtag pins out of the sd slot, and figure out booting from emmc or spi since I'm currently booting everything from the sd card... QEMU seemed much easier!

OpenOCD is a massive godsend for any kind of board bringup/embedded development.

While I can't speak for your board specifically, if the pin assignment is software reconfigurable (and is reset to sd-card access on reboot!) and you're not using the SD card once your program is running, you could reconfigure the pins in software and then wait for a JTAG debugger to attach (by having an infinite loop you use openOCD to continue past). Of course if you can debug your issue using QEMU that's indeed much easier, thanks Fabrice!

It's truly awesome and has nearly become synonymous with emulation on Linux.

Of course without things like KVM, and assorted accelerators, etc. it would lose a fair bit of luster. We all stand on the shoulders of giants to some degree.

Nonetheless, I don't know of another free emulator that even comes close... are there any to speak of?

QEMU is still amazing even without KVM; KVM is just a performance improvement for native platforms.

But with QEMU you can emulate extremely exotic CPUs on standard x86's; that's the stuff that Console Emulators struggled with immensely!

Not to trivialize QEMU (I use it often and I love it) but usually the challenge with console emulators are A) emulating supporting hardware such as graphics chips which are often undocumented and B) achieving cycle accurate emulation for platforms which need that. QEMU emulates relatively standard hardware for each platform. Even for x86, sometimes I find 86box or PCem more desirable for certain older hardware/software vs QEMU.

Well, it depends, the i440FX chipset with an SB16 will run most if not all DOS games.

... and, since you need the OS, there's always FreeDOS.

One of the best programmers I know always say that QEMU is the greatest piece of C code that he has ever come across. It seems that there is some very cool preprocessor magic as well, but I've never dug any deeper than hearing him out :)

As someone who works with QEMU, I must say QEMU is a shining example of how to do object oriented programming in C.

That said, trying to follow the QEMU code using static analysis is impossible. Every function call is a function pointer stored as a parameter in a struct that was set up by a macro called in initialization. The only way I could figure out how things are tied together is with GDB.

I feel like using C++ would have been a better language choice.

The modern QEMU is magic, but I find the original QEMU even more magical: It did not contain any assembly, and was easily portable to new architectures while still providing (relatively) decent performance -- the new TCG backend delivers much beter performance, at the cost of requiring more porting work for new host architectures.

The original QEMU had an implementation of the required machine ops in C, but instead of calling those ops, it memcpy()d their implementation, thus enabling binary translation without interpreter overhead -- basically using the underlying "gcc" as a code-generator at runtime without including it. Pure elegance and/or madness.

And it worked very well.

Nice. Copying (concatenating) the opcode implementations into an executable memory buffer is well-known. There are variants but it seems to generally be called inline threading or dynamic superinstructions or the memcpy method [1]. If you can manage to do that in C (probably with the GCC computed-goto extension?) without asm or editing machine code I think it's a very good complexity-performance tradeoff, but unfortunately it doesn't seem to be too popular. It's faster than direct-threaded VMs but only eliminates one part of the instruction dispatch overhead compared to a real JIT (e.g. LuaJIT 2 vs LuaJIT 1) which loads operands directly or inlines them and does register allocation. But once you do inline threading, you have opportunities to eliminate more of the interpretation overhead, and the payoff becomes even greater e.g. [2].

[1] Markku Rossi, Kengatharan Sivalingam, 1996, A Survey of Instruction Dispatch Techniques for Byte-Code Interpreters, https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.53....

[2] M. Anton Ertl, David Gregg, 2004, Combining Stack Caching with Dynamic Superinstructions https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90...

I recently discovered the magic that is qemu-user-static on an Ubuntu system. Install that and you can instantly run binaries compiled for other architectures. No need for a big VM or anything (*), just run the binary standalone. Combined with a cross-compiler toolchain and you have a nice way to do development for other platforms.

(*) the trick being the environment. I'm writing assembly code that just invokes Linux system calls. If you need shared libraries or other OS services, it gets more complicated.

QEMU is incredibly helpful when debugging issues very early in OS startup (like before console prints work). This sort of debugging was invaluable when working on FreeBSD NUMA stuff and doing things like backing the vm page array with node-local physical memory.

My colleague, who is working on kboot for FreeBSD (to run the FreeBSD bootloader as a linux binary that kexec's into FreeBSD from a linuxboot environment) also found QEMU an invaluable tool.

In both cases, I think QEMU probably cut development time by 50% or more.

Virt-Manager it's a must over Qemu, too. I mean, scripts are nice, but qemu has zillions of options.

Also:

    virsh domxml-to-native qemu-argv --domain "your_wm_name"

To get the Qemu launching script if you plain to deploy a prototype over thousands of machines.

What’s wrong with scripts anyway? I always call qemu from shell script and don’t understand why would I need more layers for that.

For the same reason you don't develop front-end JavaScript without a framework, or write a modern video game without using an engine. Unless you're willing to sign up for added friction and pain, you just don't.

If all I need is to write 100 lines of code, framework will add friction while vanilla JS is perfectly fine.

Well, sometimes it helps to visually see your device tree setup and then when everything it's perfectly configured, you connect the virsh shell to your KVM domain and you begin to spit out all the Qemu scripts so you can edit them for tweaks not seen under Virt-Manager.

https://github.com/lima-vm/lima is also excellent

QEMU is fantastic, I use it pretty much every day through UTM on macOS (and it doesn't need any 3rd party kernel extensions because it can take advantage of Apple's Hypervisor framework). I used to just use shell scripts to launch VMs but configuring it via the UI is just much easier, maybe I'm getting old.

https://mac.getutm.app

I always preferred a UI when messing with VMs. It's why I've used VirtualBox for the longest time. Part of my hesitancy on using QEMU has been how command line heavy it is where it has just enough friction that I don't want to devote too much time to it. I'll check out your suggestion!

Drew is one of my internet personalities where I just follow and read everything I can.

I still have to install sway though!

I started reading drew when he had a Gemini capsule. He ditched it, I was disappointed, but I understand why. I still read everything I come across.

I didn't even realize he was the maintainer of sway, I was reading his stuff while using his software and didn't even know it! Try out sway, it's fantastic.

I never understood the interaction between QEMU and KVM. I use the libvirt tools and it works great, but the internals confuse me.

I dont know why but many of the problems I have with podman/docker running on M1 seem to stem from QEMU.

E.g. problems suspending/unsuspending the mac, problems starting/restarting machines, etc.

Digging out logs to figure out what went wrong seems to be a pain too.

Quickemu: Quickly create and run optimised Win-10,11/macOS/Linux on Linux

20211008 https://news.ycombinator.com/item?id=28797129

QEMU is fantastic. I think for spinning up a VM just to try out something, virtualbox is still a good option because of its gui and its simplicity. QEMU is extremely powerful however, and I'm glad it exists.

It's a bit of an apples vs. oranges comparison though. AFAIK Virtualbox is a hypervisor for x86-64 only. Qemu, otoh, can emulate a number of different hw platforms.

I've either never heard of or never bothered to pay attention to SourceHut. https://sourcehut.org/

How did GitHub ever get so big when this exists? Is it because GH predates SourceHut? I'm not clear on when SH came into existence, but they have entries on their blog from 2019.

> * Absolutely no tracking or advertising

> * All features work without JavaScript

> * Many features work without an account

> * The fastest & lightest software forge [links to https://forgeperf.org/ - which is very impressive]

> * 100% free and open source software

Edit: here's the actual "forge" site: https://sr.ht/

I'm a paying customer, so I like SourceHut a lot, but it's easy to see why it won't be nearly as big as GitHub.

* GitHub was much earlier

* GitHub was backed by venture capital, and is now backed by Microsoft

* GitHub doesn't force you to adopt an email-only workflow. Pull request are much maligned in tech circles (following Linus Torvald's rant), but people all over dev teams love them. They are a huge argument why companies don't just stand up a git repo and call it done, devs want all those workflows and issue trackers and stuff. SourceHut is very barebones even where it has them. (The exception would be build actions, where SourceHut has no real integrations into big tools, but the core build action system is superior to GitHub's, IMO).

Incidentally, the build system depends on qemu: https://drewdevault.com/2018/09/10/Getting-started-with-qemu...

I am definitely a big PR user, but I believe that PRs could be implemented as a stand-alone app.

That link was very interesting, I read it all.

The footnote made me laugh, which is good, because apparently I make poor life choices, and am an embarrassment to us all. ;)

For macos people, that's what your docker runs on.

If you are emulating another architecture (e.g. running x86 images on top of an M1 chip) then yes. For native images, Docker Desktop uses xhyve, which is a port of the FreeBSD bhyve hypervisor to macOS

You're right never realized that as I was running mostly x86 images.

Reply on news.ycombinator.com