Show HN: QEMU front end for M1 and Intel Macs | Hacker News Reader

Show HN: QEMU front end for M1 and Intel Macs

mac.getutm.app

418 pointsosy122 comments

Hey, I need to thank you for this!

This is what I use to run QEMU on my M1 Air. I had a few bumps setting it up but now it runs flawlessly. I SSH to it via VS Code and basically I have a fully functional Linux dev environment with MacOS as the UI, just the way I like it.

Honestly, I haven't been this excited about a new setup since Moore's Law was in effect :-)

QEMU is amazing. I run Linux, ans being able to dockerize Windows with it, and set up a selenium testing environment I can run locally or on CI servers is genuinely mind-blowing. Headless chrome, Firefox plus a windows in docker setup means pretty great automated test coverage without relying on third party infra.

Just to confirm - This project has a UI over Qemu which in turn uses this https://developer.apple.com/documentation/virtualization hypervisor API to create and manage VMs on a Mac ?

It (QEMU) actually uses the older https://developer.apple.com/documentation/hypervisor apis which has lower level control. QEMU provides device emulation and Hypervisor.framework provides access to CPU’s virtualization hardware. Virtualization.framework is the new macOS 11 api that emulates the entire system but has much worse device emulation support and also lacks framebuffer emulation (I think)

Right, so the older API is akin to "/dev/kvm" and ioctls on Linux ?

It’s actually a bit lower level than KVM.

Basically a hook in the kernel ?

Hey man, care to elaborate? Seems like what I am looking for. How do you handle SSH'ing into the local VM? Are your files on the VM or on Mac?

Files are on the VM. SSH just goes through an exposed port. Feel free to email me if you need help!

I do this except the Linux is on a separate box and the dev envs are docker containers.

Decided to spend my money on decent, upgradeable hardware instead of shelling out 5k for another MB Pro. My next MacBook will probably be the most low-end I can get.

VSCode's remote/docker support is a godsend. Seems like we're heading back to the dumb terminal paradigm.

I did this too and that is very pleasant as well. But the new Air performs well and has no fan and the feeling is, quite frankly, the same.

And the way you're doing it sounds like you avoid the docker mount slowdown issue. Sounds like quite a good solution

Do you recommend any tutorials online to get such a setup?

Wondering the same thing. I'd like to explore a setup like this but trying to cobble it together myself without a good guide is daunting.

I've been considering writing one myself, admittedly it's not an easy road at first, and a step by step guide would've helped a lot.

Except that the terminal is thousands and thousands times more powerful than the machine you connected to with the terminal back in the days.

Can't get that with AWS though? Some kind of spinup-able VPS?

Like a normal VPS but you can get dat 64 cores + 64gb ram only the day you need them and otherwise pay a low monthly fee.

I have a Dell R720 that I bought fairly cheaply ~$600 came with 192GB of RAM. I shoved some SSDs in it with a GPU. Sits in my garage. If I'm not home I can still remote into my terminal via wireguard. Good deals can be found using lab gopher:

https://labgopher.com/servers/Dell_R720/

Granted it's not 64cores, but still fast.

I saw some VPS providers offer hourly billing, which can get me a 64cores + 64gb ram at like $30 a month. I'm going to go with that. I actually don't have landline/fiber internet at home so your solution wouldn't work for me, but I'll keep it in mind I didn't know about such deals!

I train ML models on my Macbook (Intel GPU) by SSHing into a machine with a Tesla A100.

In this case it really is more than thousands of times more powerful.

Do you have your projects cloned in the remote host?

I ask because I was excited about this prospect only to learn that I could only use Remote: Containers if codebase and container are on the same machine as the project workspace is just volume-mounted in the container.

Yeah I dusted off an old licence I had for resilio sync and use that to sync with my local machine.

You can get around this by running git clone inside your container as part of the startup process. But 1. It can add quite a delay to starting a container depending on the size of your codebase, and 2. Not all of my code is necessarily committed and pushed.

For me, syncing works well but I can see how it would not suit every use case

Fair enough. I don't want to clone from within the container because I like the idea of being able to nuke it at any time and just start over from local code.

Probably similar reasons to what you mean by not having all code committed and pushed.

There are more options here. For example, you can attach to fully remote containers: https://code.visualstudio.com/docs/remote/containers-advance... or just attach to existing local containers that you can set up however you'd like.

Yeah I have been using local containers and I really like the workflow.

I wasn't aware of that setup you linked. I'll give that a go, thanks!

> the dumb terminal paradigm

Hehem. The paradigm you're referencing is letting you do what you want to do (because it's a powerful paradigm). We need more of that if anything.

Also worth mentioning that UTM developers has been contribution patches to QEMU to get TCG (code emulation layer) to work under iOS / M1 Mac[1] as well (whereas I think TCG is still broken as of latest QEMU release due to mprotect[2]). So huge, huge thanks for this as well!

[1]: https://lore.kernel.org/qemu-devel/20210126012457.39046-1-j@...

[2]: https://lore.kernel.org/qemu-devel/161259505296.24527.772997...

There's a tiny patch mentioned in your second link that makes TCG work again on M1 Macs

https://bugs.launchpad.net/qemu/+bug/1914849/comments/3

  $ qemu-system-alpha -nographic
  PCI: 00:00:0 class 0300 id 1013:00b8
  PCI:   region 0: 10000000
  PCI:   region 1: 12000000
  PCI: 00:01:0 class 0200 id 8086:100e
  PCI:   region 0: 12020000
  PCI:   region 1: 0000c000
  PCI: 00:02:0 class 0101 id 1095:0646
  PCI:   region 0: 0000c040
  PCI:   region 1: 0000c048
  PCI:   region 3: 0000c04c
  >>>

This is a very nice frontend. Someone should definitely port it or rewrite it for Linux. All the existing UIs are either ugly or use libvirt, which is a nasty pile of hacks that has no place on a desktop machine.

Not sure what GNOME Boxes uses, but it looks good: https://wiki.gnome.org/Apps/Boxes

Boxes uses libvirt too. (It would be foolish not to.)

Thanks but I think a port is virtually impossible as this was designed in SwiftUI and depends extensively on Apple APIs. GNOME Boxes is a good UI on Linux.

libvirt has put a lot of time into figuring out how to run QEMU in the most secure and performant way (specific low-privilege user, seccomp, all the right options for virtio, etc, etc). If you aren't using it then that's a lot of effort to reinvent, and you probably missed something out.

But they also hid it behind it layers of XML-laden complexity. If you know exactly what you want, using QEMU directly may be preferable.

https://pypi.org/project/libvirt-python/

That doesn't help one bit with the impedance mismatch that you get when you want to do something that goes against libvirt's abstraction layer.

libvirt wasted a lot of effort into abstracting everything behind a ton of pointless hypervisor-agnostic cruft, only to then deprecate basically every single backend except QEMU.

I have used libvirt for almost a decade before going back to plain QEMU; I must say that using it from the CLI is an herculean task, you often have to edit plain XML in order to do whatever you want, and the cherry on top is that `virsh`is a huge pile of cow dung that often drags you into a Sisyphean fight against the aforementioned pointless abstractions - you find yourself configuring "domains" and "virtual networks" when you really only wanted to launch Windows to run Word.

When I had to run stuff on a big server which only does VM, maybe with OpenStack and other orchestration systems on top, libvirt was often _fine_. I still use it in those use cases (I usually prefer Proxmox this day, though), it works fine, I can connect to remote instances using SSH and Virt-Manager and I like how it often just works.

My use case, 90% of the time, is that I just want to run a damn Windows or FreeBSD VM on my workstation. I want to put disk images wherever I want because my root FS is small (often on ZVOLs, which are a big massive PITA to use with libvirt) without having to read a million different websites and man pages. I want it to work with my firewall configuration, which often changes due to the fact I have to do stuff with this machine, and I don't really care about 90% of the features of libvirt - I just want to run Windows every once in a while and I want my VMs to keep working without too much effort.

Thus, I run plain QEMU with the following configuration, which took me a solid 10 minutes to perfect:

- `nsbr0`, a bridge with a static IP configured using systemd-network;

- `qemu-bridge-helper` set-up in order to attach taps to nsbr0 automatically, using `-netdev bridge` (echo 'allow nsbr0' > /etc/qemu/bridge.conf);

- IP forwarding set to 1 and `masquerade` in nft.conf;

- OVMF downloaded from my distribution's repos;

- `spicy` to attach to a running VM;

- a 30 line script that spawns QEMU in a tmux. If I want to configure a new VM, I just copy paste this shell script and I put it in a directory together with a copy of OVMF_VARS.fd for the efivars.

Et voilà, I get VMs that just work, without having to wage any pointless battle against libvirt in order to get what I want. This is basically just a way to run KVM VMs that does not create a million firewall rules together with useless network interfaces, and doesn't force me to configure "storage pools" when the only thing I care about is running 2, 3 VMs max on my machine.

There's a reason why VMWare sells both ESXi and VMWare Workstation, and why people still love VirtualBox despite how shitty it is.

It's because more often than not something designed to run on servers is hardly be the best solution for a single user. The same features that make libvirt interesting on servers make it clumsy and overcomplicated on workstations, and the more I use it the more I realize it's true.

I find VirtualBox a pretty good solution for a lot of stuff. What’s wrong with it?

It's a type-2 hypervisor, so its performance is often quite worse than KVM. Also, its storage drivers are quite wonky, so I/O is often choppy and underwhelming. It has often a better desktop experience, though, except when you run Linux with VirGL. Also, virtualbox cannot emulate other platforms; for instance, I often run AArch64 operating systems in full emulation using QEMU on my x86_64 workstation.

Isn't KVM also a type-2 hypervisor?

No, KVM (and FreeBSD's bhyve) are kernel modules that for all intents and purposes transform the host kernel into a type 1 hypervisor itself. The hypervisor is not, like in the case of VrtualBox or VMware workstation, running on top of the kernel, it is the kernel itself. The VMs spawned on KVM or bhyve have potentially full access to any hardware capabilities exposed to them. You can passthrough PCI devices to KVM VMs, for instance, using VFIO; you can also fully expose the host CPU allowing nested virtualization

A "true" type-2 hypervisor does not require any modification to the host kernel; at max it ships with a kernel module or two to help implementing things such as custom networking or to get a speed boost kernel side. For instance, it's trivial to share USBs with a guest on the fly with VirtualBox, because the VM is basically a process running on the host, so it can just tap into the USB stack and expose the device to the VM. This is not so simple with KVM.

Also the license situation with the extension package.

Yeah I didn’t say ideal. But if you are on Win/Mac (there are a lot of those people) and need a Linux box without much fuss it’s a good go-to solution.

LXD recently added support for QEMU virtual machines in addition to its existing support for LXC containers, and it's almost completely replaced libvirt for me. It's great, pretty much just a matter of adding a `--vm` flag to normal `lxc launch` usage, e.g. `lxc launch images:debian/buster --vm`.

Yes, but the main reason I use VMs is to run different OSes. When I need to run other linux distros I normally use systemd-nspawn (or rarely LXC) and it's enough.

Thanks for creating this, it looks fantastic, I just bought it! I love that it's a nice layer over QEMU / Apple's Hypervisor, so i know the performance will be reasonable. One bit of feedback you might consider: Most of the VMs in your gallery are for ARM64, and the x86 one is quite an old version of Ubuntu. Would you consider making a couple more modern x86 versions? My requirement is to have an x86 VM for deployments to Lambda, cross compiling Rust using Docker on M1 simply doesn't work yet and I'm hoping your app will do the trick :) Thanks!

I have the same requirement, but ended up building the final bundles to lambda in the cloud instead. However I am quite excited about the new container image support for lambda meaning I guess you do not have to compile for musl anymore.

> UTM was created for macOS and only for Apple platforms

And it shows, the UI is beautiful. Will try this out.

Oh very interesting. I only need Vagrant support and then I can fully switch to M1 in the future. Looks like this could be the successor for Virtualbox on Mac M1.

Wonder if this also helps developing for x86 Docker too (which is a major pain point on M1 Mac).

This is insane. Best software find in a long time. An open-source alternative to Fusion is very welcome for me.

And, lovely work on the UI!

Where have you been OP? I've been waiting for years.

I will totally pay for this. Tired of paying yearly subscriptions to Parallels Desktop Lite which is another frontend to the OS hypervisor.

Wondering if there could be a Metal -> Vulkan -> Zink -> OpenGL way of doing HW acceleration. Or Metal -> Vulkan -> DXVK -> D3D for Windows. AFAIK VirtualBox used some of Wine's D3D code to do D3D on Virtualbox.

In theory virgil3d is the way to go as it has existing support in QEMU but there hasn’t been any work porting it to macOS/Metal as of this post.

The commercial virtualisation tools from Parallels and VMware support creating macOS guests on a macOS host by making a clean install, i guess from the macos recovery system/service?

Are you planning on macOS guests too? It's sometimes really nice to test a macOS application on an absolutely clean macOS install to ensure it's not dependent on any library on your macOS development machine.

macOS support isn’t perfect in QEMU but there’s been people who got it working: https://khronokernel.github.io/apple/silicon/2021/01/17/QEMU...

Seems to support QXL graphics driver and SPICE? I ran into problems compiling QEMU on macOS with the above mentioned. Did you solve this?

https://stackoverflow.com/questions/59636198/how-to-compile-...

Check out https://github.com/utmapp/UTM/tree/master/patches

Awesome! Thanks.

Well it is great in many ways, but today I downloaded it and played a bit and found it hard to use compared with “directly using qemu cli” or vmware fusion for Mac free edition, but I am looking forward to its improvements in the future. Please do keep the good work ;-) thanks!

Thank you for making this available for download from the website too, and not making it app store exclusive. Do you have any plans to support macOS Mojave? What is needed to add GPU support in the future?

Since it uses SwiftUI heavily, I doubt it.

GPU support will probably come from implementing virgil3d in Metal.

Is this legal to deploy on the Mac app store? Not sure if the Mac app store has the same license incompatibility issues as the iOS app store.

afaik there is nothing preventing this from being on the mac app store.

The iPhone app store is so strict because they really want to keep the platform locked down, with no possibility for 3rd party app stores. Although even then there's weird exceptions - Roblox seems to be allowed.

That cat is out of the bag on MacOS, though, and at least for now it seems Apple is keeping it that way - iPadOS is going to be their locked down work OS.

Well, it seems controversial, to say the least (unless qemu and other GPL components are not actually bundled? I don't know... I don't have a Mac):

https://apple.stackexchange.com/questions/6109/is-it-possibl...

It's possible that those replies are out of date and the Mac App Store TOS have changed.

Random question but when I tried to install Debian on this I got an error that no cdrom drive was detected. Is there a workaround?

Use the Debian netinst iso instead?

I used exactly that

When the Debian install asks you what device to use, give it /dev/sda or /dev/vda. For some reason /dev/cdrom wasn't working for me either. I ended up switching the ISO's device to virtio and tried /dev/vda and it worked.

The ISO was on /dev/vda and the target disk image was on /dev/vdb. Just make sure you install the bootloader on the disk image. Then when I removed the ISO, everything was just fine.

From one dev to another, can't thank you enough! Just bought it on the App Store to support you.

This looks amazing! Can us dinosaurs on macOS Catalina expect a compatible build in the future?

This is great, thank you.

Is it currently possible to have snapshots, like in VirtualBox, for instance?

Not yet but it’s planned for the future but after other higher priority features like USB, network bridging, and running from other paths.

What is the plan for network bridging? Tuntaposx isn't an option anymore and Apple has plans to ban all kexts.

https://developer.apple.com/documentation/vmnet

interesting - thanks!

Just a silly question here but what's the UTM acronym meaning?

I have been a QEMU user for over a decade now. Virtualization was fun back then, but so slow forcing me to experiment with many. All is so decent now.

100% uncertain since I couldn't find it to be sure but "Universal Turing Machine" seems likely considering the quote from Turing on their GitHub.

Yes, that’s correct.

Nice page, very clean and straightforward, yet not too dull. Cool project!

What does UTM stand for? Couldn't find it on google or in the FAQ...

https://en.wikipedia.org/wiki/Universal_Turing_machine

The price isn't that big, I have to wonder if this product will be supported for the next, say, 5 years?

I am willing to buy. Just wondering what the author's plans are.

Please try out the free download and only purchase it if you feel that you’ve already gotten value from it and would like to support the project. There is no difference in the binary for the free and paid version. It’s really hard to promise anything 5 years ahead as the intent is not to make money off of it.

I did not realize there was a free version. Thank you!

Running UTM with Ubuntu Server 20.04 ARM64 ISO image, it prompt UEFI and I couldn't install like it did work a few month ago.

I thought virtualization doesn't work with M1? Like docker and such... did something change?

No, that was never the case. Even Docker is running, albeit still in WIP status (https://docs.docker.com/docker-for-mac/apple-m1/).

A12Z didn’t do virtualization. M1 does it fine.

It's understandable you may have missed some information, so to correct the record (and ensure other readers are aware), Apple highlighted virtualization in their announcement of M1, specifically highlighting Linux. It's been supported from the moment it was public.

Well WTF... I know 100% for a fact I saw this somewhere, that's why I didn't buy one............. can you think of what the article I read might have been referencing? It might have even been a huge thread on HN how developers/programmers were waiting a year+ to buy it until certain things were supported.

I think maybe you saw a bunch of FUD? I’ve seen a ton on here.

There was also a lot of speculation that Apple would quash booting other OSes, and maybe that got into the mix?

FWIW I’m a lifelong Apple user and even if I had the budget I’d hold off a year on ARM Macs, because I’ve learned to avoid first revision models in general.

There were two common misunderstandings which were widely promulgated people whose confidence exceeded their actual knowledge:

1. The developer preview systems didn't support it, but that was because they used the older A12X chip. Lots of people spread that on social media as this not working on ARM at all.

2. Docker for Mac did not have ARM support at launch. Docker does have full ARM support so this was just a function of needing time to develop the desktop app (and not having access to the M1 hardware much ahead of anyone else) but some people panicked and portrayed this as something which would require a huge amount of work. There's a preview release out now: https://docs.docker.com/docker-for-mac/apple-m1/

Well fuck me, that was the entire point of buying this current mac... thanks for the info.

I use Parallels. What is the benefit of QEMU compared to Parallels?

It’s FOSS :) other than that, Parallels only does virtualization and does it really well. UTM/QEMU does virtualization not-as-well-but-still-pretty-good in addition to emulation.

How is the x86 performance on say windows and linux?

Do you mean the emulation performance on M1? In my experience Windows XP is “usable” and Windows 7 feels like a 10 year old desktop running with a spinning drive.

Linux arm64/Windows 10 ARM should be near native, minus graphics acceleration. x86 will be about as slow as you would expect with QEMU. Pretty slow.

There is no hardware assisted virtualization for x86 on ARM.

Which parts of QEMU on M1 are hardware accelerated?

Emulation, which goes through a hypervisor. I don’t believe graphics are yet.

Technically the AArch64 instruction set and the memory system are the only parts that are 'hardware accelerated'. QEMU handles everything else in software.

Specifically arm64 guests, otherwise it's just QEMU's TCG.

  $ nm /Applications/UTM.app/Contents/Frameworks/libqemu-aarch64-softmmu.utm.dylib  | grep -i hv_
                 U _hv_vcpu_create
                 U _hv_vcpu_destroy
                 U _hv_vcpu_get_reg
                 U _hv_vcpu_get_sys_reg
                 U _hv_vcpu_run
                 U _hv_vcpu_set_pending_interrupt
                 U _hv_vcpu_set_reg
                 U _hv_vcpu_set_sys_reg
                 U _hv_vcpu_set_vtimer_mask
                 U _hv_vcpus_exit
                 U _hv_vm_create
                 U _hv_vm_map
                 U _hv_vm_protect
                 U _hv_vm_unmap

https://github.com/utmapp/qemu/tree/apple-silicon

https://developer.apple.com/documentation/hypervisor/apple_s...

Just bought it on the app store, very nicely done!

Looks like it's built with SwiftUI too, nice!

Any benchmarks?

Looks good. How is the speed compared to vmware?

The underlying tech is QEMU, so probably equivalent or better.

It is infinitely faster than VMware on M1 Macs :)

Very nice! Any way to 'buy'/donate without giving 30% to Apple?

Nice for Apple users but how/when can I run Mac and iOS applications on Linux?

AFAIK you can't, and there's no indication that will change. You may be able to virtualize a whole macOS install, though that's pretty difficult even on Apple hardware (source: I tried several times, and didn't get it to work well)

I think many open-source efforts are heading in the wrong direction. Let Apple make sure our software runs on their hardware, and let FOSS make sure that Apple software runs on FOSS platforms. Not the other way around.

Dream on one can. But sorry mate reality bite.

https://github.com/foxlet/macOS-Simple-KVM works pretty well for me (not for daily use though as it uses software rendering, but fine for compiling/testing apps)

Use this equivalent of Wine for macOS apps called Darling:

https://www.darlinghq.org/

Try this: https://github.com/myspaghetti/macos-virtualbox

Be careful, I think you need something like 50-60GB of free space to run it. And the script does not check the free space beforehand. After the script is run, I think the OS X VM will occupy at least 25GB.

PS: the graphics performance is ok, but not great. I could not manage to install the extension package of VirtualBox, so may be if you can install it, then the graphics performance of the virtualized OSX will improve.

Saw UTM for iOS, got excited, only to find out it's not available in the App Store, and I have no Mac to build it with.

Oh well.

You can sideload it without a jailbreak using AltStore: https://getutm.app/install/

Thats interesting. But it seems like the AltStore requires running a AltServer which is only available for Mac or Windows. It would be great if they provided a Linux build too.

It is kind of ironic though that I need to bring up a QEMU VM to run the AltStore binary which lets me run QEMU on my phone :D

Reply on news.ycombinator.com