MacOS in QEMU – ARM edition [pdf] | Hacker News Reader

MacOS in QEMU – ARM edition [pdf]

kvm-forum.qemu.org

222 pointswatusername88 comments

KVM Forum 2023 Day 2 10:15 AM https://youtu.be/hyrw4j2D6I0?t=4684

thank you, without the video, slides do not make sense

Thank you. Slides don't really tell the whole story.

I recently (yesterday) ran into emulation issues when I tried to run my dedicated game server build for Linux (that I produced from Unity) inside an amazon Linux docker container on my M2 Mac. I use docker for apple silicon that supposedly allows you to run amd64 images on arm (which M1/M2 macs are). Yes even specifying the correct —-platform and upgrading the emulation software.

It was only when I opened up all logging in a dev build did I discover there was some syscall from the mono side of things that wasn’t emulated properly enough. I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles. Anyways, I just spun up an AWS EC2 instance to test this nothing part of the process.

I know podman uses qemu and maybe it could run this Unity based workload and maybe I could uninstall docker but eesh the boring image tech is starting to become a bit of a hassle. I’ll probably just keep using EC2 instances or write up a vagrant file to create a local VM. I’m sure eventually my workload will be fully supported eventually so I can optionally just wait.

I love and hate knowing so much.

Isn't Docker for Mac just using VMs in the background?

It is, and Docker for Mac uses QEMU for `--platform`, just like Podman AFAIK. The issue here is that it's QEMU user-space emulation, which does not support some syscalls, like `ioctl`.

The fix here would be to spin an amd64 VM and run the binary or install Docker there, instead of relying on the more convenient - but less compatible - user-space emulation of Docker. Not very related to this very interesting presentation on virtualizing ARM macOS guests on QEMU hosts..

> Docker for Mac uses QEMU for `--platform`

Only on older setups and on the default config. Updated MacOS (13+) and updated docker provides an option to use Rosetta and the new virtualization system. That, I believe, replaces QEMU for x86 emulation with the Apple emulator.

I've had substantial success with that method; not only does it seem a bit faster, but some containerized apps that very frequently failed to start (e.g. Apache Pulsar) work consistently in that mode. I suspect this difference in experience speaks to bugs in QEMU that trigger in the presence of certain application behavior (in Pulsar's case, something weird happening with the JVM on startup).

QEMU's still an incredible feat of engineering and something I frequently use on Linux, though. It just seems that the Apple emulation on Macs is better (which makes sense, given that it's built by the people who made the semi-proprietary silicon it's emulating from).

Can I ask a question? It seems like your post is combining two separate things. As I understand it and based on earlier comments in the thread:

1. To run a Linux container you need to be running Linux 2. This is why Docker for Max has always run a VM under the covers. It uses qemu. 3. This was true even before Apple Silicon when everything was x86_64 and nothing was emulated

Now you are bringing up Rosetta and x86 emulation. While I understand sometimes it’s necessary to run a container that hasn’t been compiled for aarch64 (or whatever they call Apple) so I can see why the actual cpu emulation capabilities of qemu or Rosetta would be necessary there, this seems to be beside the point.

if you have no Linux system involved because you have no VM, how would that work? What would the base system “under” the container be and what would respond to Linux’s OS-level calls?

This is the core of my challenges:

> sometimes it’s necessary to run a container that hasn’t been compiled for aarch64

I have several containers in that boat, for which no ARM versions exist at all. Those containers were previously using QEMU to emulate x86 on ARM linux (in Docker's VM on an ARM Mac). That emulation encountered the failures I described, and was slow.

Switching to use Rosetta 2 in Docker solved the problems; the process for enabling it is described here: https://levelup.gitconnected.com/docker-on-apple-silicon-mac...

Something I don't know and am curious about is how Docker-for-Mac is actually using Rosetta 2. Is it running an additional Linux VM containing an x86 Linux OS, and running that VM through a Rosetta-2-enabled hypervisor? Or is Rosetta 2 distributed as a Linux program that is being invoked inside Docker's pre-existing aarch64 Linux VM instead of (or inside of?) QEMU?

Edit: as for your question:

> if you have no Linux system involved because you have no VM, how would that work?

That's not my situation, so I'm not sure.

Yes, Rosetta 2 is being used by Docker’s pre-existing aarch64 Linux VM. So it’s only translating userspace x86 code to Arm, then running it on the ARM Linux VM. Syscalls, etc. are still handled by the (aarch64) Linux VM.

Containers can and do remove the "it works on my machine" problem when you're on the same platform, and preferably the same OS exactly. The big deal here at least in my experience is the ability to completely replicate your environment, however you still need the underlying OS, because docker is not a VM.

It sounds to me like you're running a container on a massively different architecture and I think it's still pretty cool that it runs at all. Obviously when trying to run a game, which needs access to hardware and system calls beyond the use case of a simple CLI tool, running the container on a different architecture becomes more of a long shot.

I think given the circumstances it's actually kind of amazing you got this far, and if you're able to go all the way with your game with emulation, that'd be pretty cool.

Docker is a VM on macOS

And on M1/M2 Macs that means an arm64 Linux VM. So whatever issue came up here probably would come up attempting to run amd64 containers on regular arm64 linux (assuming the right compatibility tooling is installed).

> I thought it ironic recalling how many people had sold containers as lighter weight VMs that remove “works on my machine” hassles.

Containers aren't VMs, they're Linux namespaces. If you want to run a container you need Linux. If you want to run Linux on MacOS you need virtualization. There's no "lightweight" VM about it - it's still a VM.

Containers are virtualization without the machine part.

When running Linux containers on Mac, the containers run within a full virtual machine.

Docker Desktop on Linux works this way, too.

While the machine part may not matter, the OS certainly does. Linux containers depend on the Linux kernel which is external to the container.

No, they are not. Containers are namespaced process groups (CGroups) with a sandboxed filesystem and (optionally) sandboxed network layer via bridge interfaces and virtual devices.

What is virtualization to you?

https://en.wikipedia.org/wiki/Docker_(software)

> Docker is a set of platform as a service products that use OS-level virtualization

https://www.netapp.com/devops-solutions/what-are-containers

> Containers are a form of operating system virtualization.

On Windows and macOS Docker runs Linux in a VM and then all the containers attach to that VM. I think this has changed to WSL2 on Windows of late but it's still the case on macOS AFAIK.

WSL2 is a Linux VM (albeit with some advanced features such as dynamic memory allocations, GPU & file sharing, etc.) on Windows hypervisor, and Docker for Windows just uses their own WSL2 VM.

It also 'magically' exposes Docker socket to your own WSL2 instances/VMs.

Windows actually has some kind of native containerization but I believe it's based on virtualization and not an equivalent of namespaces on Linux. I'm not even sure if such a feature would make sense on Windows in the same way it does on Linux.

But that's my point. It seems from this thread people don't understand what docker is except that they can kinda use it like a VM. But it's fundamentally different.

It’s worth noting there are different kinds of containers out there, and Docker is just one “ecosystem”. Docker on Windows and macOS runs in a VM, which is kind of a bummer because it means you can’t really allocate all your RAM for containers. By default Docker likes to max out at half the system RAM, which of course you can change post-install, but I feel like there’s a trade off or disadvantage in such a scenario vs Docker (or LXC or podman or something else) running on Linux.

It might be that the RAM is reserved up-front on Windows and macOS but not on Linux? I’m not sure.

Even when the arch matched, macOS was a bad platform for running containers. I'm not even sure you can call them "edge cases" because it was basic things like volumes not working right.

macOS is engineered for users and desktops, not servers and development. Apple's long history of blocking virtualizing macOS is a testament to this.

MacOS has been virtualizable in x86 since mac-on-x86 was released.

It also is engineered for development. Maybe not your particular type of development. Apple silicon isn't technically development-unfriendly, it's just apple silicon makes it less easy to do x86 development. But so does mips, sparc, ppc, and other architectures.

Do you understand the limits apple put on virtualization? Are you aware of how it's handled for cloud providers?

I didn't say it was technically impossible. I said apple has a history of blocking it.

maximum of two virtual machines running MacOS on a single Apple device, no matter how it is configured.

this is in the EULA and is enforced by Virtualization.Framework.

cloud providers get no exception to this.

MacOS can be simultaneously "engineered for development" and "a worse container host than a chromebook". They aren't mutually exclusive.

> I thought it ironic recalling how many people had sold containers as lighter weight VMs

That's your problem there, containers are by design not VMs. Anyone who sold them as such aren't understanding the tech and you shouldn't have blindly trusted them.

Why not just use native Amd64 Linux machine for development? Using emulators and syscall debugger is not very productive experience!

That’s kinda the conclusion I came to. Being apart of the bleeding edge is fun in some senses but I need to get moving quicker.

That’s why I just said I’d spin up an EC2 instance that will 100% run on amd64 cpu arch.

I’ll be exploring vagrant but if it too fails I have workarounds.

That said I’m not really planning on hauling two computers with me just in case I want to test something on an intel processor.

This should work, since it spins up a full VM to run your container.

I'm using UTM (nice frontend to Qemu) and the performance is okay-ish on a base mbp m1. The desktop is fluid enough for basic usage. I use it mainly for remote access client apps that I don't want running on my OS, like Citrix.

what operating systems are you running on utm? I've tried it, and for some it's great, for others, usually one of the bsd's, there's always some small problem. And the gui feels kind of incomplete.

Weirdly, virtualbox on linux was usually my go to for trying out other operating systems. It also felt like the most straightforward to use.

UTM is fantastic for macOS-on-macOS emulation on Apple Silicon devices. Historically, macOS VMs have been so slow on every other virtualization platform. UTM uses Apple's native virtualization framework and it's a godsend for testing older or newer OS's.

I used parallels in the past. Mainly to spin up macOS VMs for setup testing with ansible (I maintain a macOS build farm for our studio) And I found the performance decent enough. It all tanks the moment more than one instance is running though.

There is also a nice little guide on how to get macOS Ventura working on Proxmox (using QEMU), although x64 this is really working like expected:

https://www.nicksherlock.com/2022/10/installing-macos-13-ven...

Worth noting that you can’t use your appleid or icloud on a vm of macos

As in technically can't, or mere license-wise "you're not supposed to"?

Virtualization at least does not have a functional sign-in

i could be completely wrong here, but i do seem to remember something about there being some physical chip in the machine that needed to be present for e.g. iMessage to work. this was a while ago but iirc they also somehow incorporated local Apple ID account management into that system.

I wouldn't be surprised if there are fuses inside the processor that tie it to your Apple ID.

I don't use them on my real Mac why would I need them on my fake (VM) one

I think that means no iOS development so if you need that you need a real Mac

Well, QEMU has no graphical acceleration for macOS (no virtio drivers for XNU and Quartz) so it's basically unusable - software rendering has been historically borked on macOS, and never well tested.

I can see why that would block publishing apps to the app store, but surely you can still locally develop applications in the iOS emulator?

I'm sure it's possible, depending on the kind of application and the tools you use to build it. But xcode won't build your application if you don't link your account. I believe it needs that to "sign" the application with apple.

There may be ways to get around this but I don't of any.

You can manage your signing identity entirely using Apple's website and just not use automatic provisioning. (I do that anyway even though I build on macOS as I just don't want our build system tied to Xcode just so we can use the automatic provisioning.)

Note that the EU's Digital Markets Act will probably stop all that bs, as requiring to use one core platform service to develop for, and distribute to, another (iOS) is against that law.

Third-party app stores and outright website-based app distribution are coming next year anyway.

That's interesting, I did not know about this legislation. I'm not European, but it seems that even outside of Europe, people will benefit from this change.

https://en.wikipedia.org/wiki/Brussels_effect

If Apple was compelled to support a first-class iOS development experience on Windows, all this would do is immovably stunt the continued improvement of iOS development on macOS. There is a material degree of OS integration / cooperation in IOS development with Xcode. If Apple released a second-class development experience I’ve no doubt that you’d spare no time in making similar insinuations that it’s some sort of artificial constraint.

XCode is a hog, if there was a supported way to develop applications without it, that would be a first class experience.

like the xcode cli tools. all th e basic tools are there.

This is more than accurate. I wanted to start iOS development, but even opening Xcode is painful.

Been using Xcode since 2012 and I think it's great. It has it's flaws, sure, but overall I think it's a really good development experience. What do you dislike about it?

I've been using it since it was known as Project Builder and also don't have many complaints. It's not perfect, but it's not bad. I spend plenty of time in the IntelliJ-based Android Studio and haven't found it to be much of an improvement (and in fact, find that its "smarts" get in the way as often as they help).

Xcode and iOS Simulator on other platforms would involve porting AppKit as well as writing all-new Darwin virtualization for those platforms. Xcode is thoroughly a Mac native app, and the iOS/watchOS/tvOS/etc simulators currently just run the respective userlands on top of the underpinnings that macOS shares with other Apple platforms.

It'd be rather similar to the situation with Docker, where Xcode and the simulators would run markedly better on macOS than on other platforms due to fewer layers being necessary. Developing for iOS on Windows or Linux would technically be possible but it wouldn't be very pleasant.

You can, but only if you follow an hackintosh guide and spoof it with a "virgin" ID, it's tricky but doable.

So what is for rent on AWS or MacStadium are not VMs on genuine Mac hardware?

On Intel macos it did work with iCloud accounts etc. At least last time I tried (which is a few years ago admittedly).

They're not really VMs. They are metal instances, which run without a hypervisor, but have virtualized network and storage attached instead.

How are these secured against people messing around with flashable components?

There aren’t many.

I have Mac running on ESX signed into my Apple account. I haven’t used it too much but it’s on my list of devices on my account.

Even says “VMWare” for device type.

VMware passes through the id of the physical hardware into the guest VM. So if you're running on apple hardware, this works. If you're running on a PC, there are hacks to 'borrow' the serial number of some ancient imac and have it work still (even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using).

But if this serial number was shared between several people, all of whom might have logged in with a different apple cloud id, would that somehow cause apple to trigger something?

I think they log you out if someone else logs in with another appleid on the same hardware id.

Mine got logged out when I used a publically shared id, and stopped being randomly logged out when I picked a random number.

> even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using

The ancient random serial numbers are also in a standard format.

> Apple devices manufactured after 2010 generally have 12-character alphanumeric serial numbers, with the first three digits representing the manufacturing location, the following two indicating the year and week of manufacture, the next three digits providing a unique identifier, and the last four digits representing the model number.

> If you're running on a PC, there are hacks to 'borrow' the serial number of some ancient imac and have it work still (even ancient systems are still allowed to log into icloud, and those ancient systems didn't do any cryptography so all you need is a valid serial number that nobody else is using).

Hackintosh users have to do something similar to be able to sign into iCloud.

The serial number doesn't have to be valid (as in existing on a physical machine) to work, though. It just needs to look valid (be generated using the same methodology as real serials). In fact in order to prevent accidentally using a serial tied to a machine owned by somebody else, the recommended procedure is to generate a serial and check its AppleCare status to verify that it's not tied to a real machine, and if it is to regenerate and check until you find one that isn't.

Same here. Mac VM which is all signed in to iCloud/iMsg etc hosted on ESXi. Haven't had any issues

Yes, it works on Intel but not ARM VMs (it’s a conscious limitation by Apple)

You can't use AppleID or iCloud because Apple's virtualization.framework does not support this (yet) https://developer.apple.com/forums/thread/707459

If you run macOS without the virtualization.framework, which is only possible on non-ARM mac's, then it will work.

I’m using QEMU via UTM as my primary virtualisation solution for Linux and macOS guests on macOS.

What guest OS do you use for GPU accelerated Linux guests? I have not been able to get GPU accelerated full res Linux guests working on M1/M2 via UTM.

Alex's presentation was among some of the cooler ones at this year's KVM Forum.

HVF sounds like it has some unique features that make it nicer than KVM for some use cases.

It would be interesting for this audience to compare QEMU against VirtualBox..

My own bias is that I will be purchasing a Mac Studio around XMas as a flutter app build machine.

I know that VirtualBox has guest OS additions to pass through graphics acceleration from the Host OS to the Guest OS. Does QEMU have this for MS Windows and Linux *6 guest OSes?

The quick summary is VirtualBox vs kvm/qemu is a classic commercial product vs open source comparison. Without going too deep into the actual architecture, VirtualBox is much easier to get started with (although check out something like Proxmox), but qemu is much more powerful. AFAIK, there are no features in VirtualBox that can't also be done with kvm/qemu, but there's a lot of things that can be done with kvm/qemu that can't be done in VirtualBox.

I say kvm/qemu because the actual hypervisor is kvm, while qemu is used for control and device emulation. qemu can also emulate full machines, which is obviously much slower but can be extremely useful.

As a personal VM machine, vbox is fine. But for anything more involved, it's got a lot of rough edges.

We used to use virtualbox at work to run our CI workload (spawning a fresh VM for every job). Long story short: It does not scale well at all. Its scripting API is terrible, we ran into endless network problems (for instance, its builtin dhcp server would sometimes fail to flush expired IPs, leading to running out of IPs to give), and it would sometimes just freeze up entirely, requiring us to restart the virtualbox services.

We switched over to kubevirt[0] (essentially qemu/kvm inside a kube cluster) to handle our CI, and it's been a much, much smoother sailing since then. And it even allowed us to run macos VMs on linux hosts, something that we never managed to get working in vbox.

> Does QEMU have this for MS Windows and Linux *6 guest OSes?

QEmu has virgl[1] for paravirtualized graphics, but its windows drivers are unsigned afaict, so you can't really use it there. Works fine on linux, so long as your mesa and kernel aren't super ancient.

Semi-related (MacOS x86-64):

Quickemu: Quickly create and run optimised Win-10,11/macOS/Linux on Linux

https://news.ycombinator.com/item?id=28797129 (2021)

The command line strings for Qemu can be enormous and complicated because they have changed over time, whats shown on page49 is very simplified, but should get some people going.

Well, not just that, but Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox. Qemu is an emulator of general machine platforms that happens to offer virtualization APIs for performance boosts in specific configurations (x86 on x86, most specifically).

This is what allows it to simultaneously pretend to be a reference PREP platform, G3 machine, G4 machine, ARM9TDMI mobile device, ARM-Cortex EFI platform, SPARC server or even some hodge-podge machine of mixed emulated components. People treat it as equivalent to VBox or VMware; but it's not, it's a different tool that happens to offer a similar overlap of use cases (like using the butt of a screwdriver as a hammer).

> Qemu isn't a platform virtualization utility like Hyper-V or Virtualbox

I know, thats why I could run arm based OS on those twos on an x86 or x64 cpu.

Theres a lot of work thats gone into Qemu to emulate the different processors, question is have they found undocumented features of cpu's when trying to emulate them?

Technically, a good chunk of the ARM boards that Qemu supports are completely undocumented. So yes?

The emudev scene is pretty close-knit, there are shared contributors amongst MESS, MAME, Qemu, etc that spread research advances. In addition, a lot of the breakthrough work in undocumented/edge features is done by decappers, hardware researchers and computer engineers. This research then distills down into the more advanced emulator platforms before working its way into commodity documentation and hobby emulators.

how well does this work? Is this the new hackintosh?

It is MacOS only. So, no, this is not the new Hackintosh.

Reply on news.ycombinator.com