Apple Virtualization Framework
developer.apple.com
developer.apple.com
It's great overall, and fairly convenient, but it has a fair share of bugs and limitations. The Virtualization service crashes with various combinations of Linux kernel version, macOS version, and architecture (ARM/Intel), so making it stable in each setup takes quite a bit of work. Device support is limited (no USB, etc.), and workarounds have other limitations. Rosetta is really buggy.
The popular belief is that Virtualization.framework is inherently faster/lighter than other VMMs like QEMU, but I haven't found evidence to back it up. I spent a day prototyping a custom VMM (with an open-source base) and was able to get dynamic memory allocation working [1], as well as faster file sharing in some cases. There's a lot of other things I could do better (faster, simpler, more reliable) with a custom, tightly-integrated replacement.
But unfortunately, that's not an option. Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. There are too many users relying on Rosetta for fast x86 emulation for me to ditch it, despite all its bugs.
So yes, I still use Virtualization.framework, but only because I have no choice.
Happy to answer other questions about the framework!
Would that change with Asahi Linux host OS and macOS guest?
I can't find the exact post but I remember marcan saying that he doesn't want to poke the bear of using Apple's custom ISA extensions on Linux. So even though it's possible, I'm not sure whether it'll actually be done.
Wait why have I never heard of this? I just tried it out and it seems to work amazingly well.
the only thing that i'm afraid of is this:
- i work in an enterprise company, and they don't want to pay for docker. as a software engineer i want to be able to buy OrbStack pro license with my own money and be able to use it both in company or my personal environment. is it possible? or your future licensing plan would prevent me to use it in professional environment and only allow me to use it in personal environment?
- Professional : XXX $ - Business YYY $ - Enterprise ZZZ $
which XXX << YYY << ZZZ
I'll repeat my scenario: - I'm a Backend Developer - I don't have any share in the company. - I don't have any management position in the company. - I'm not forcing any one in the company to use OrbStack to progress my mission in the company. - I'm solely using it inside own device provided by the company. i don't have it installed in my colleagues devices. - this company device is only in my own control and is not shared with anybody else in the company. - this company device is not used in any automation CI/CD pipeline in the company. - I'm willing to use it as a docker GUI for the docker images hosted in companies account in Google cloud. - I'm also sometimes/occasionally, willing to use it in my personal toy projects using the same company laptop in my spare time using the same instance of OrbStack license.
based on these are you implying that i'm not allowed to use `professional` license and i need to use `business` license?
Or you are telling that it's okey for me to use my personal `professional` license in this scenarios?
Needless to say orbstack earned its place in my setup so thanks a lot!
Edit: sorry, I wasn't clear, what I'm using in orbstack isn't a container instance but a full VM. I don't expect to find a huge difference with containers but I figured I'd make it clear so I don't cause any confusions.
Every once in a while, the guest Ubuntu kernel will oops about smp related things. Every time it boots its clock is reset almost a month in the past and I have to force sntp to correct it. After sleeping the host, the guest clock will not advance so it'll be behind several hours.
I'm not sure where to assign these issues: Ubuntu kernel? virtualization framework? vftool? bad configuration?
https://github.com/evansm7/vftool appears to indicate the former, but I thought the latter was required for rosetta so interested to try this.
EDIT - looks like I had this backwards, vftool indeed supports https://developer.apple.com/documentation/virtualization which is the newer framework that supports Rosetta. The older framework is https://developer.apple.com/documentation/hypervisor
Have had a PR open for several months with no attention from the author. I almost forgot about it until seeing this HN headline.
[0] https://github.com/JamesDunne/vftool-rosetta
The Rosetta support is surprisingly trivial to use. The virtualization framework exposes a mountable volume containing a single `rosetta` executable. In the Linux guest, you just register binfmt_misc support that recognizes an x86 ELF image and point it at the `rosetta` binary. It works for docker too so you can run arm64 and amd64 images.
But, regardless, the subject of using a Mac to host a plethora of different platforms (in my case, for build-server duties) is indeed a fascinating subject. If I can get my target/builds set up on the next Mac Pro, whatever that is going to be, I'll be quite happy to do so .. especially if it isn't just doing MacOS/iOS builds, but whatever else my framework of choice (JUCE/Tracktion) supports (Android, Linux .. Windows .. etc.)
Alas, if only there were the hardware at scale which supported Apples' Mach implementation, at favourable economies.
One way this is favourable to this dev is if Apple finally give me a machine which I can use to Build All the Things™, including iOS and MacOS and Windows and Linux and Android, oh my!, virtually and/or with x86/ARM cores onboard, oh yeah!, without having to resort to the typically draconian bollocks on other platforms, or rent someone elses hardware, or whatever.
A new Mac Pro that can be loaded to the gills and host several competing OS's at once, for the purpose of software development? Sign me up, because I'm tired of build-box'ing rigs and fighting that darn license ..
Well except that, a new developer-focused Mac Pro with that centralized campus feeling, sounds great to me.
I can only wish it would have both x86 and ARM cores onboard, though, to get that true all-round Mach prowess...
The ability to move between hardware platforms also has very little do with the ability to run operating systems in virtualization. Case in point is that Apple is one of the last major OSes to include a hypervisor in the OS.
https://docs.getutm.app/advanced/rosetta/
And FYI, this is also available in Docker Desktop for macOS, which allows better performance when running x86_64 containers compared to the older solution where Docker was using qemu.
Last time I checked, Rosetta only worked for one-off binaries, wouldn’t really allow docker to run more x86 code inside.
Rosetta is a tool for running x86-64 binaries on an arm64 OS.
I think you're asking if rosetta lets you run an x86 kernel, to which the answer is no - the whole point of this framework is to support virtualization, e.g. the OS is running directly on the hardware. The moment the OS can't do that, there is no point in doing anything other than emulation.
The only thing I'd like to see added to the framework is the ability for nested virtualization (which is now available in the M2 chips, but isn't built into M1 chips).
Is there an example of this working in any context? Until then I don't see how it can be considered available.
EDIT - Confirmed below it does work in Linux on metal - Props to the Asahi team!
There could be knobs to twist but UTM doesn’t expose any, so…
UTM doesn't have as many settings as VMWare Fusion but stability and performance wise it's on par. It's also open source and free.
In the UTM page it says "Under the hood of UTM is QEMU, a decades old..." so are we talking about the same thing as Apple Virtualization Framework. I am confused qbout the QEMU mention.
Edit: I think the catch is that QEMU uses AVF if available [1].
[1] https://medium.com/code-uncomplicated/virtual-machines-on-ma...
I've got some Ubuntu VMs set up already, and as per [0] I cannot find an option called "Use Apple Virtualization" neither in UTM, nor in the VM settings.
I am on Intel Mac.
[0] https://www.jamesarmes.com/2022/12/linux-mac-utm.html#once-m...
- Introduction when Monterey was released: https://macops.ca/macos-monterey-apple-silicon-vms/
- New features added in Ventura: https://macops.ca/virtualization-updates-2022-06/
https://developer.apple.com/documentation/virtualization/run...
I tired of evaluating and learning various wrappers and helpful virtualization applications. Being able to configure directly in a normal language is great, and it eliminates software supply chain concerns.
Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap)
Can you explain more? Sounds like you are saying your Ubuntu (guest) VM reacts faster to your actions than the underlying (host) macOS?
The iPad could finally utilize the m1 processor to its fullest.
I agree if Apple implemented virtualization on iPad/iPhone this same way it would be a huge for unlocking their full usefulness and capabilities while still maintaining host isolation/sandbox.
^^ For the curious ones :)
More or less it's an abstraction on top of Hypervisor to make VMs much simpler in Swift.
From what I remember, Docker tried to use this but suffered greatly on disk performance.
OSDEV rocks!
I'd be quite happy to see how a fat MacOS machine is being used as a build-server for multiple generations of MacOS versions, as well as how it could be used (via Linux VM's) to also build for Android, Linux, Windows, etc.
Essentially, this subject is all about the build server destination; there is an award for how many different targets you can get from the same code-base, and being able to load up a fat Mac that approaches the nexus, is certainly of interest to us developers.
VM's are key tools in modern build environments. Having a fat Mac running multiple VM's to solve the target/build infrastructure problem is fascinating on all fronts..
I'm curious whether someone has found a good option for running x86 images on m1 chips.
I'm generally happy with the multipass VMs I'm creating but they are arm architecture so then if i use docker images inside them I'm limited to arm images. Maybe I just haven't explored how to run x86 correctly within multipass?
I'm finding that usually arm images are a step behind the x86 images if they are official, or they are built by someone outside the organization, so I'm never sure I'm using a safe image.
Do any of the options here make it performant to run x86 on apple silicon? Some interesting comments on Rosetta that I don't fully understand.
Regardless, performance on my beefy M1 Max MacBook Pro is pitiful compared to running natively on an x86 system.
Sure, you could put in a request to the project maintainer, fork it - or find a contribution yourself - but there will still be zero guarantees and support arrangements will be shaky at best.
Keep in mind Rosetta has a few limitations like it can't translate AVX instructions.
Having said that..
If you really want to run x86 images on arm, and while we're definitely not 1:1 with docker (and don't want to be) we're interested in people's experiences with https://ops.city as there is native hardware accelerated arm support and x86 on arm. (I'm with the company behind it.) It's definitely going to be more performant than a docker image since docker/k8s not only rely on a full blown linux but also abuse iptables, et al. It also has support for bridging (outside of the vm) and 9pfs so that would help alleviate your pain.
One big thing that's missing is better storage support. Right now it's just disk image and that's about it. No way to implement something like qcow2 with snapshots and whatnot. It's missing a lot of opportunity for VMs.
Another big thing that's missing is USB pass-through.
Other than that: it does absolutely everything I need. It even provides some kind of hook for network which should allow to build userspace router with any network architecture (didn't try it myself, though, but I think it should work).
I was able to implement thing that I missed for years on M1 mac. Basically I run Fedora VM, I installed Rosetta helper inside, and docker. And then automagically my x86 images started to work with reasonable speed. I don't know if Docker Desktop supports it already, I don't want to touch it with a longest stick in the world. But for myself I solved this issue.
It was rock stable in my experience and just worked. qemu on the other hand managed to crash my entire macOS (hopefully that bug was fixed).
Another thing that I liked about it: protocols are "modern" and open-source. Almost every driver is virtio. No ancient cruft. No proprietary things. Eventually other operating systems could be run on this solution.
https://github.com/lima-vm/lima can use Virtualization framework for creating VMs, there is also https://github.com/gyf304/vmcli as a very simple CLI utility for running VMs, though it's not very actively maintained.
The only concern currently, besides for virtualized macOS the lack of support for iCloud accounts, is that it contains a number of bugs which make it difficult to use as is for non-English/US configurations.
Thus, there is a bug with ISO keyboards, which are recognized by Apple virtualization framework as ANSI keyboards, which shifts or even makes a certain number of keys disappear (different depending on the language: French, German, Danish, etc.).
https://developer.apple.com/documentation/virtualization/run...
I used to use CCC (Carbon Copy Cloner) to make a mirror, but it doesn't like to boot from the mirror (it does, but I can't use any of the secure stuff). Also CCC destabilizes my system. If I have it running all the time, I crash -a lot-. Since I yanked it, I haven't had a hard system crash in months.
These days, I just make sure the Time Machine backups keep going, and I'll use CCC on a one-shot basis to make occasional disk clones (makes for a much faster restore).
Maybe I misunderstood you, but you can make a bootable USB stick from the installer app:
It's been a while, since that was possible.
The M-series just basically put a fork in it.
That, and CCC kept crashing the machine.
This is not a rhetorical question, I am trying to understand if I missed something important.