Development on Apple Silicon with UTM | Hacker News Reader

Development on Apple Silicon with UTM

rkiselenko.dev

102 pointstodsacerdoti72 comments

Maybe I'm just really out of my depth on this, but it feels like there's not a lot of information about _why_ these particular steps and tools are used. Why are 4 different Linux images needed? Why are there all of these steps to create a one-time use "init.iso"? Is it just so the cloud-init script can run? I see the call to mkisofs is referencing "cidata", but it's the only place in the whole page that "cidata" shows up. Does that mean mkisofs is cloud-init aware? And why use emulation instead of virtualization?

I guess part of why I'm asking is because I've set up virtual machines in UTM on Apple Silicon before, and I never had to go through all of this just to get Linux installed and configured. The post makes me wonder if there's something I'm maybe missing, but it doesn't give any explanation for me to be able to figure out if that's the case. Maybe the post is meant more just as a checklist for the person that wrote it, for their own reference? But the way the post reads doesn't quite sound that way.

Hmm... that's all coming out sounding more critical than I mean to. I just want more info and I am curious about the approach.

If you just need a single Linux VM, you don't need to fiddle with cloud-init. If you want repeatability or automation, that's when you'd reach for it, whether for setting up per-project VMs, shareable VM configs between developers etc.

Also, you don't need all 4 Linux images, just the one you want to run as your guest OS. Emulation / virtualization depends on the guest OS CPU architecture.

Yep, I just install from the Fedora netinstall iso and do a normal install. I might be missing some features but it works for me.

The easiest way is to just choose from the UTM gallery. But I wanted Fedora 41, not 38 which is the latest in the gallery.

From my understanding, you really need only one image – the article is just providing four for your tastes (Fedora/Ubuntu, aarch64/x86_64). The images are linux installers, which “understand” cloud-init (because it’s “industry standard”), so if you place the {user,meta}-data file in the right place (a volume named CIDATA, it seems[0]), they can configure your installation without really having to go through the tedious process of configuring through the installation process, install packages, and so on.

I don’t understand why anyone would go to the route of emulation in 2025, but if someone wants to run an x86_64 image with UTM, well that’s the only route – I’d suggest just going to an aarch64 image. Things were a bit more rough back in 2020, but stuff got much better and I don’t remember any compatibility problems these days.

[0] https://cloudinit.readthedocs.io/en/latest/reference/datasou...

My thought was why use UTM? Most of this can be achieved with qemu alone :). But it showed me something new. The cloud init tool was new to me. From my toolbox I would have used ansible or something. But I think it very interesting that this runs all automatically during first boot. But I agree one needs to read between the lines to understand what the purpose of this post is. As you said it reads like a overcomplicated install setup.

> My thought was why use UTM? Most of this can be achieved with qemu alone

Afaik UTM uses Qemu under the hood, but provides a nice UI on top for the basic use cases. It also has a library of prepared images, so that your VM is a few clicks away from intention to have one.

It can also modify the VM, resize storage after creation etc.

Of course all of it can be done with QEMU alone, but this makes it easier to deal with than remembering tons of QEMU command line arguments.

Guess you misunderstood me. I know that UTM is built on top of qemu. I use it as well. I mean when already using this init image tooling etc why clicking through the UI to setup a VM. One would think to offload this also to a script. Because in the posts steps UTM is just a means to start the resulting image.

Maybe I did misunderstand, I agree the post makes it overly complicated! You definitely don't need cloud-init just to run Linux on Mac OS

> Afaik UTM uses Qemu under the hood

During installation UTM asks if you’re willing to use Apple virtualization rather than qemu

> My thought was why use UTM? Most of this can be achieved with qemu alone :)

qemu needs to be studied a bit, UTM is fairly intuitive.

I recently decided to learn how to create VMs with bare qemu (using the command-line).

As I have an arm macbook for work, UTM helped me a ton with aarch64 virtual machines because I could enable debug log and see what qemu options/flags/switches would UTM use.

Unrelated: I have some ideas about writing a tool that aims at being a "spiritual successor" to vagrant (from hashicorp), but focused on targeting qemu rather than virtual box.

Anyone interested? Please let me know (upvote or comment)

UTM uses a custom fork of QEMU, so not all flags will apply to the non-forked QEMU.

https://docs.getutm.app/settings-qemu/qemu/#qemu-arguments

Here is some advice:

Theoretically the entire docker workflow can be translated to VMs. In practice it is a shit show.

The biggest problem by far is building a VM image, because it consists of multiple highly irritating steps.

1. Building custom packages for the target distribution.

Since we aren't using containers, we would in principle need a full VM per application. This is not a good idea in practice. We want to avoid containers, but we still want something very much like docker images on the application level. The obvious answer is building distro specific packages.

Building distro packages is annoying, because the developer machine doesn't necessarily run the same OS as the servers. This means that building the package requires you to spin up a temporary virtual machine or a docker container. Let me tell you, it is by far easier to build your packages inside a docker container and that's why I never even bothered with the VM route, even in situations where I'm deploying VM images. There needs to be a VM based alternative to "docker build" that doesn't necessarily spit out a VM image, but rather it spits out the result of your build (e.g. packages) onto a mounted directory on the host.

If you never built your own alpine packages. Try writing an APKBUILD. It is very easy.

2. Building VM images

Now let's say we are done and just want to build our VM images. There are already distro specific tools like https://github.com/alpinelinux/alpine-make-vm-image. What you want to do is install the packages created in the first step, run a simple bash script for finishing touches and setup cloud-init for the first boot. Unlike a Dockerfile, this should be kept very simple, because the packages are already doing everything the Dockerfile is expected to do. The only thing I would overcomplicate here is directly integrating a package repository into the tool to make it effortless.

3. Running the VM

At this point everything should be quite simple. The primary use case is to run the VM image locally on a developer computer before deployment. Some quality of life features like docker style port proxying and mounting directories would be nice. This is by far the easiest part because tools like virt-manager already exist.

Is this whole wall of text slop ai-generated content? It’s completely irrelevant wrt what i posted.

UTM does use QEMU, so really it's just "bare QEMU" or "QEMU with a nice UI"

In my dreams I make time to use qemu for entirely user mode Docker on Windows.

Cloud-init isn't a replacement for ansible unless you're building your own VM images. cloud-init only runs on the first boot. You would use it to install and setup ansible.

From the title, I hoped so much that this was about using UTM on Apple Silicon iPads – don’t really want the person who always complains about Apple, but it’s really a pity that the iPad has such a strong CPU inside (I own an M4 iPad Pro) and can’t do any realistic development on it.

Last time I checked, there really isn’t any way to virtualize ARM machines on the iPad with UTM (unless you’re on a lower iOS version), and emulating x86_64 machines were slow enough (even on M4 iPads) and not really usable.

I’m on an environment where iPads are fine but equivalent MacBooks aren’t (stupid rules), and got this machine for hope that I might tinker development with iPads, but I gave up running stuff locally and just boot up an EC2 instance whenever I really want to do something. It’s a pity.

(BTW, from the article, why would anyone really emulate a Linux machine with UTM on Apple Silicon? From my experience ARM64 Linux images are really good at compatibility…)

The fault here is Apple having exclusive access to JIT, UTM on iPadOS/iOS can't fix that.

Yeah it’s pretty unfortunate that such capabilities are locked to special entitlements.

Ever since I discovered OrbStack, my M1 has become the perfect laptop for me. OrbStack has great UX, massive FS performance, Docker & Kubernetes support, and a bunch of really clever people behind it. Everyone doing linux development on a Mac should try it

What are the real-world benefits over Colima?⁽¹⁾ (Besides having a GUI, which I can do without.)

I'm interested in an up-to-date comparison when running the VM with Rosetta 2:

    colima start --vm-type=vz --vz-rosetta

⁽¹⁾ https://github.com/abiosoft/colima

Edit to add: While researching OrbStack, I found this comment in a post from 7 months ago that mirrors a lot about my experience with Colima: https://news.ycombinator.com/item?id=41424044

I use Colima as well. Haven’t found anything that works better, TBH.

It's really good but the fs caching has bit me a couple times before I realized what was going on.

Can you tell more about this?

Orbstack is great! I'm wondering why there is not an open-source equivalent? Did they have to reimplement a lot of the native calls to make it work fast enough?

oh wow. i hadn’t hesrd if orbstack. i am sick of docker issues on my og m1.

Or you could just follow the guide on the UTM site & use the installer. https://docs.getutm.app/guides/ubuntu/

UTM is great when you’re using native ARM versions of Linux or Windows. Where it (or QEMU) is absolutely terrible is trying to emulate x86 for working on legacy apps or for retro computing.

For example, try installing a Windows 2000 vm. It won’t even get past the initial setup screen because it runs so slowly.

My saving grace has been that Windows 11 ARM also has its own x86 translation layer, so I can still run many (but not all) 32bit windows apps. But it’s been frustrating having such a powerful machine that can’t run an x86 vm.

WINE also runs under Rosetta and can handle 32-bit apps, which can help improve x86 32-bit Windows app coverage. Not everything runs well under it but it has settings for which version of Windows it simulates which allows it to run some things that 11 can’t.

I really like UTM, I especially like UTM remote on the iPad.

The issue is that of course Apple computers that are large enough to run VMs are… expensive.

So I have instead found myself using proxmox on a cheaper (by comparison) threadripper machine.

The added bonus is that xterm.js (the default LXC console viewer) works wonderfully on the iPad- Though of course you can use prompt3 for heavier duty stuff.

The other uses I have for VMs on the mac is Docker and Kubernetes, but those are solved by colima and minikube respectively.

> The issue is that of course Apple computers that are large enough to run VMs are… expensive.

Expensive compared to what? UTM uses Apple Virtualization Framework, which lets the guest VM only occupy as much memory as it's actually using (instead of reserving all of its available memory). This means it's viable to run a Linux VM on e.g. an 8GB RAM MacBook Air.

How can I verify that? it definitely looks in htop and Activity Monitor that the RAM is occupied and unavailable.

Maybe memory compression helps, but it doesn’t appear so to me right now.

> How can I verify that?

Good question. I was speaking from memory, and all I can find when googling is this: https://docs.getutm.app/settings-apple/virtualization/#ballo...

EDIT: I recall vaguely something about it being a problem with Linux guest VMs because it aggressively uses RAM as a file system cache. So it'll just fill up RAM to the max when reading from disk, and overwrite this if it's later needed by applications.

balloon memory is supported by qemu on linux too.

The cost of RAM on Apple machines is far above the market price for the same RAM in the PC world. Apple quotes a loss leader with a minimum configuration, but as soon as you up the specs to something usable by a developer for VMs or LLMs, the price has ballooned.

Love it. Also using UTM with Amazon Linux 2023 x64 on an M1 works, so you can create a local HashiCorp Packer Pipeline using the Packer UTM plugin (similar to the QEMU Packer plugin)

source "utm-cloud" "this" { iso_url = "${path.root}/dev_images/al2023-kvm-2023.6.20250303.0-kernel-6.1-x86_64.xfs.gpt.qcow2" iso_checksum = "sha256:0dc2797fe19847f6c75878dd344ab478ac0657077d9a15f2907bb2df41d8c3de" vm_arch = "x86_64" cpus = 2 memory = 4096 display_nopause = true boot_nopause = true export_nopause = true skip_nat_mapping = true communicator = "ssh" ssh_clear_authorized_keys = true ssh_private_key_file = data.sshkey.this.private_key_path ssh_host = "192.168.64.100" ssh_port = 22 ssh_username = "ec2-user" shutdown_command = "echo 'packer' | sudo -S /sbin/halt -h -p" use_cd = true cd_label = "cidata" cd_content = { "meta-data" = <<EOF local-hostname: vm-hostname EOF "user-data" = <<EOF #cloud-config ssh_sftp: enabled: true password: ec2-user ssh_pwauth: True chpasswd: expire: False users: - default - name: ec2-user lock_passwd: false plain_text_passwd: ec2-user ssh_authorized_keys: - ${data.sshkey.this.public_key} EOF "network-config" = <<EOF #cloud-config version: 2 ethernets: enp0s1: # dhcp4: yes addresses: - 192.168.64.100/24 gateway4: 192.168.64.1 nameservers: addresses: - 10.0.2.2 enp0s2: dhcp4: no state: down EOF } }

I've had a really good time using https://github.com/lima-vm/lima

I also suggest https://github.com/abiosoft/colima for containers, "Containers on Lima"

Big fan. I will say Orbstack is superior - and can be more performant with mountpoints but colima does pretty much everything I need and is FOSS.

Not sure why this recommends emulation instead of installing the native ARM images referenced near the top of the page?

Yeah I was wondering too. The big advantage of UTM, apart from that it's free, is IMO that it can run natively on Apple ARM cpu's.

Yeah but VMware Fusion is also free these days and can also run natively.

I think that UTM is the Apple way to gain kernel access for other OSes. Mach was originally designed to host multiple OSes simultaneously. I think UTM is meant to develop those capabilities.

A lot of dev tooling still expects x86_64. Example off the top of my head, Cosmopolitan will not build ARM binaries and will not compile on ARM. (But it WILL build x86_64 universal binaries that will run on Apple Silicon and macOS via Rosetta.)

There is also the issue of wanting to have your dev environment be as close to your prod environment as possible, and the vast majority of cloud-based hosting is still x86_64.

With virtualisation you can use Rosetta 2 for Linux to run x86_64 binaries. That's what I do and it's more efficient.

UTM uses Rosetta via qemu. It’s the most direct way of emulating x86 on Apple Silicon, using native Apple emulation layer.

Then you're emulating everything that runs in the VM, as opposed to using an ARM vm and only emulating the x86 program you want to run. This makes things a lot slower.

That’s a good point… but does Rosetta in a Linux VM somehow use the native MacOS hypervisor?

I would like to see a benchmark of these various methods…

I don't know how Rosetta 2 for Linux is implemented in Virtualization.Framework - but it exposes it via a mountpoint on the (ARM) Linux VM. The binfmt executable that is exposed essentially does what Rosetta 2 does but for a Linux binary.

I think the binfmt executable can be used outside of Virtualization.Framework and can even be used in an ARM VM in say Asahi but people don't because it's not easy to do but also honour system as it's not licensed for use outside of MacOS.

You can't do Rosetta 2 for Linux thru qemu as qemu doesn't use Virtualization.Framework. UTM does support virtualisation via VF.

Building on a virtual x86_64 barely achieves the performance when building on host, if there was an applicable way.

Fair enough, but (speaking as a web developer), running a stack locally on Apple Silicon (especially if it has any "interesting" dependencies such as compiled Rust/C libs and whatnot) and expecting the same stack to run identically in the cloud on x64 can still expose differential behavior

At the very least, I could test mock deploys from the Apple Silicon side to the x64 VM running locally (over loopback) as an extra data point

I don't actually use it for this use-case but now that I'm thinking about it, I might try, because this seems useful

From a web standpoint it's possible, but when I tried building AOSP on macOS, it didn't turn out great.

Indeed, and here are a large variety of OSs directly from the source:

https://mac.getutm.app/gallery/

For a quick Linux VM on an Apple Silicon Mac I’ve had great results with Tart. Drop dead simple. X86 binaries run at full speed due to Rosetta.

I’n about to embark on my first Apple Silicon machine tomorrow, and UTM is a core part of my plans for it. The difference from TFA, for me, is that I already have some mature Nix flakes for doing development in WSL that I’ve built up over the last year. If I can’t have that environment bake a VM for me wholesale, it’s probably a short `nixos-rebuild switch --flake` away.

Keep Colima on your radar. Best way to run Docker on a Mac, IMHO.

Although I would'e loved to use UTM, it's still a resource hog. OrbStack (machines) does such a great job at being lightweight.

Is it already possible to connect an external screen to UTM and use that as the main screen? (Multi-port Adapter or iOS, or a displaylink usb dongle on Macbook).

That would enable some great stuff. Until then, I don't think the overhead is worth it for me

The most severe limitation that I've found is inability to work both with x86 (Rosetta) and USB passsthrough. qemu does not support Rosetta and Virtualization.Framework does not support USB passthrough. In the end, I abandoned Macbook and moved to Thinkpad because of that.

I use multipass for ubuntu https://themythicalengineer.com/how-to-install-ubuntu-on-mac...

I moved from that to Colima and never really looked back.

UTM is such a high quality mac app, up there with Preview.app. I use the free version distributed via homebrew. But if you find the software useful, highly recommend you either pay for it via mac app store or donate which can be done via github sponsors.

The docs for UTM seem to indicate that jailbreaking or sideloading is required, and with my current reliance on the Apple stack (macs, HomePods, tv etc) I really worry about getting my account banned - even though I’d only ever use this for development.

Is this an unreasonable fear?

UTM requires jailbreaking/sideloading to install on iOS because it violates App Store policies so it can't be distributed via the App Store.

On macOS the Mac App Store has much less restrictive policies so UTM is actually approved to be installed via the Mac App Store. https://apps.apple.com/us/app/utm-virtual-machines/id1538878... Also macOS allows you to install and run applications which aren't approved by the Mac App Store. So using UTM as described in the article doesn't violate any Apple policies.

Hmm I haven't tried x86 linux on utm on an arm mac (i have an x86 linux box i can ssh in), but i tried x86 windows. It was so slow as to be unusable.

Edit: would command line linux be usable, considering it doesn't have to emulate all those pixels?

An x86 Linux VM on an ARM Mac is roughly native speed because it can utilize Rosetta 2: https://docs.getutm.app/advanced/rosetta/

Ahh. Thanks, I didn't know that.

Looks like you need to install an arm distro and x86_64 binaries on top of it though. May be a lot of trouble in some cases.

UTM is the one thing I wish I could run on my iPad - with a JIT or proper virtualization (right now you can only run UTM SE with very slow and battery draining emulation).

Has anyone tried emulating linux/amd64 on Silicon with GPU support? I'm testing an engine to run LLMs locally that only runs on that, no support for arm.

UTM is OK for light workloads, but if you can justify the cost of Parallels, you’ll be far better off using that.

I hate most things about Parallels. I spun up a Windows VM using Parallels for work purposes, and it decided to change from using native Chrome to open html files to using Edge in the Windows VM. So double-clicking a html file would first start the Windows VM, if it wasn't already started, and then open it in Edge inside the VM. WTF is wrong with you Parallels?

I work at the company selling Parallels and I let one of the PMs know of your concern; this is his reply:

"Parallels doesn't change the file associations. Parallels registers Windows apps (like Edge) for macOS in case users want to open them/files using macOS interface. macOS then decides what apps to use for this or that file type or protocol. Based on our testing if there is already a file association, then a new app registered in the system doesn't change it. Tip: Parallels provides an option to disable sharing Windows apps with macOS. It is enabled by default as the vast majority of customers like it. File associations can also be changed anytime using "Get Info" in file context menu and then "Open with:" and "Change All..."."

In other words, it looks like .html files were associated to your (in-Windows) Edge, hence it booting up a VM etc. but that can easily be mitigated.

Reply on news.ycombinator.com