Hacker News Reader

Comment by bonyt | Hacker News Reader

What were they doing with the A12Z when they demoed Linux running in a VM at WWDC, then?

It appeared to be Ubuntu for ARM64. If it wasn't virtualized, could it have been running under QEMU? Or maybe some sort of WSL type shim into the Darwin kernel?

pardon my ignorance, but how is QEMU not virtualization? There have been MacOS builds of KVM/QEMU for a couple of years now and I distinctly remember seeing that it had either been added or was being added to the base OS (google-fu is failing me because of so many results about the reverse-- running MacOS under KVM/QEMU).

QEMU is an emulator, not virtualization. It allows the user to run an OS designed for one type of CPU on a host system of a different type. i.e running Raspbian (ARM Linux) through QEMU on an X64 Linux Host running on an X64 machine.

Virtualization, by contrast, allows the user to run a guest OS designed for the same architecture as the physical hardware. i.e running x64 Linux guest on an X64 host that runs Windows as a host OS.

But to add to the complexity, many virtualization systems use QEMU as a support library to achieve stuff.

I'm not nearly smart enough on that stuff. I fear I will need to get deeply into it in order to setup tooling that will run on the new Macs for embedded development.

That's hardly the point. Many virtualization systems do indeed use QEMU for device emulation, but they don't use it to emulate the entire CPU functionality itself. A key point about virtualization is that most code run native while privileged instructions are trapped and handled by the hypervisor, allowing multiple guest OSes to coexist. This gives guest OSes the illusion that it's in sole control of the hardware.

QEMU can do both the complete emulation you describe and accelerated virtualization similar to that offered by VMWare.

QEMU can either be used for virtualization (i.e. running x86 Linux on your x86 Mac) or emulation (i.e. running PPC Mac on your x86 Mac).

They must have not been using an A12Z for that.

They already said they were using the mac mini with A12Z for the whole presentation demo

Where?

In the presentation itself, it was towards the end

They said the demos were running on apple silicon. And they said the DTK runs a A12Z. They never said the demos ran on the DTK or on an A12Z.

They also mentioned pre-release hardware in the “undisclosed location” — if I recall correctly it was before they did a demo of virtualization.

Well, after Googling, I can’t find the exact quote that referenced new hardware in live blogs, but it’s telling that just before they did a demo of virtualization, Maya and so on, they said, “All of the Big Sur features demonstrated earlier were being run on the development platform” according to Anandtech — meaning the earlier Big Sur demo was on the A12Z but the next part would be possibly newer hardware.

They said a Mac with Apple silicon, not necessarily the developer transition kit. If they were running the demos on the dkt they would probably have shown the device and said so explicitly.

Parallels clearly has something significant under wraps: https://www.parallels.com/blogs/apple-silicon-wwdc/

My guess: Parallels on Apple Silicon will support virtualizing AArch64 VMs, and also x64 VMs through Rosetta 2. Support for AArch64 alone doesn't seem interesting enough to keep under wraps, and Apple did commit to supporting x64 JITs, so x64 VMs seems like a natural extension.

We'll see, but I doubt it.

A user-level emulator is a completely different beast performance wise than a full system emulator. With a user-level emulator the kernel, and possibly even the shared libraries are native code, and the code that needs to be emulated is relatively easy to translate from one architecture to another.

For a full system emulator not only you need to emulate the kernel also, but all the system-levels instructions have to be emulated as well (unlike user space instruction that can be JITed).

Just compare the performance of qemu-system-aarch64 with qemu-aarch64 running on amd64, and you will see a MASSIVE difference in performance. It's very likely rosetta will be more optimised than qemu, but still there is a fundamental problem here.

I'm sure qemu-system-x86_64 will run on aarch64 macs, but I doubt Apple/Parallels/VMware will touch this space.

Fair points, and on reflection you are probably correct.

But Apple has a history of weird but successful systems that nobody else tried, like 64 bit user space with a 32 bit kernel, or mixed-endian processes sharing memory. And there's pointed coyness around "Apple Silicon" and Parallels keeping mum. So I want to believe.

I need to believe because it makes a massive difference for me as an embedded developer. I can't use Macs anymore if I can't run multiple x64/x86 Linux VMs on them.

When did Apple do the mixed-endian processes? Was this with PowerPC Macs?

It would be interesting to see a blend of Rosetta2 for translating and running user-space x64 code and a WSL-style kernel "shim" to provide direct access to the native aarch64 Darwin kernel.

One can certainly write software that virtualizes another machine. Sure, performance might not be great. Sure, it's better if you have an implementation in silicon. But that doesn't prevent making a demo of a software VM running another kernel. What am I missing?

I would consider that a fraudulent demo, since it is selling something different from what it is showing.

No? There was nothing in the context of that video to suggest it was running on an A12Z. They were showing you the features that will be available on the chip they're going to bring to market sometime this year. This chip must already exist in a near-production form in order to meet that deadline.

I’m not saying they did, I’m pretty sure they were running it on an undisclosed chip. What would be fraudulent would be emulating it on the A12Z and claiming they were virtualizing.

Apple isn't selling anything at the moment. It was only an announcement, and it remains to be seen if what they deliver matches what they announced.

Not sure what you mean by "software VMs," but if you mean CPU emulators, they're fundamentally different from VMs. Emulation is not "virtualization."

No judgement of not knowing but para-virtualization is not emulation and it is not a new or niche technology either. I think IBM was doing it in the 60s (might be off base with that factoid because modern-ish operating systems like the RC4000 weren't even a thing until the late 60s). It has simply been superseded by hardware assisted vt that allows for multiple "ring-0" guests.

There is a greater overhead and a limited scope when using paravirtualization. But that doesn't mean it is a relic, in fact you can try it right now in Virtualbox. I also believe that the Linux KVM has para-virtualization optimizations if the guest is Linux and the bare metal doesn't support hardware VT.

I'm confused, are you sure you've responded to the right comment? No one is talking about paravirtualization in this thread. As I take it, the parent comment was suggesting that Apple was faking virtualization with some kind of software trick in the demo, which isn't what paravirtualization is about.

Paravirtualization is a general optimization technique for VMs that allows the guest OS to better communicate its intent to the hypervisor through the use of hypercall APIs, sparing the guest OS from having to issue many series of privileged CPU instructions that each have to be trapped and handled by the hypervisor. It saves time because going back and forth between the hypervisor and the guest OS is an expensive operation, and reducing the number of times it happens helps a lot. It's not a simple replacement for hardware assisted virtualization or the other way around.

Paravirtualization is relevant if the CPU does not support full virtualization - Xen was originally designed to use PV because it was very difficult to make x86 VMs fast because unmodified guest OSs required slow emulation. See section 2 of https://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xens...

You may be confusing the concept of hardware-assisted virtualization with full virtualization. Moreover, the famous Xen paper you've linked to never claimed that x86 didn't support full-virtualization because it did thanks to VMWare.

VMware didn't do full virtualization - it couldn't because x86 did not support it at that time! There were privileged instructions that did not cause a trap, and so which could not be virtualized using the normal trap-to-monitor technique. VMware used dynamic translation to JIT the guest kernel so that it did not execute privileged instructions, and so that it would run much faster than a simple emulator. This is explained in the Xen paper and also in https://inst.eecs.berkeley.edu//~cs252/sp17/papers/vmware.pd...

From the Xen paper:

> VMware [10] and Connectix [8] both virtualize commodity PC hardware, allowing multiple operating systems to run on a single host. All of these examples implement a full virtualization of (at least a subset of) the underlying hardware, rather than paravirtualizing and presenting a modified interface to the guest OS.

And no, I haven't forgotten about binary translation. As you mention, it was only used to replace privileged instructions and not a full-blown CPU emulator. VMware VMs still ran native CPU instructions, and the overhead incurred by translation is a whole different matter unrelated to my original point.

Perhaps I misunderstood. Here is the logic flow I followed:

(1) Apple has a virtualization framework that can "...boot and run a Linux-based operating system on an Apple silicon or Intel-based Mac computer."

(2) The virtualization framework for arm-based Macs will virtualize arm-Linux and x86-based macs would virtualize x86-linux. There maybe a semantic issue here if you believe that this api would also allow the "virtualization" of x86-linux on arm. That would not be considered virtualization to my best understanding of the definition.

(3) Looking at Apple's VZVirtualMachineConfiguration, VZVirtualMachine and "Virtualization Constants" there is nothing apparently exposing the underlying virtualization mechanism. So we don't know if Apple uses hardware (ring -1 in x86 parlance) or software virtualization (almost always para-virtualization) under the hood. Likely it uses both depending on the context.

(4) We know that Apple's current dev kit hardware (A12z) doesn't support hardware virtualization. Hardware virtualization is commonly referred to as the "Virtualization Host Extension" in arm64 parlance.

(5) We know that Apple demoed virtualization on arm with what appeared to be a arm64 ubuntu guest.

(6) Your original message said that you believed they weren't using an a12z for the demo, likely because its aforementioned lack of hardware virtualization.

(7) The comment below yours suggest that "One can certainly write software that virtualizes another machine". They were more than likely referring to para-virtualzation in their comment. I don't know any other blanket type of software virtualization.

(8) You responded saying you didn't know what software virtualization was, it is almost always para-virtualization. Para-virtualziation does not require a hardware hypervisor and predates all hardware virtualization technologies.

While there is full-hardware virtualization, it basically never a thing that happens anymore because of the rather large overhead. I don't know of any modern virtualization software that offers full software virtualization. Especially if your guest is Linux.

> (6) Your original message said that you believed they weren't using an a12z for the demo

I never claimed this

> (7) The comment below yours suggest that "One can certainly write software that virtualizes another machine". They were more than likely referring to para-virtualzation in their comment.

Well, yes and no. Virtualization (in the context of VMs) is about sharing the hardware between multiple OSes, so the statement about "virtualizing another machine" didn't even remotely make any sense. This led to my original comment, "emulation isn't virtualization."

And one small thing, it was the comment above mine, not below.

> (8) You responded saying you didn't know what software virtualization was, it is almost always para-virtualization.

Where can I find refereces to this term that supports this statement? I looked, and found the term "software virtualization" being used to refer to containerization and programming language runtimes, but not VMs.

> While there is full-hardware virtualization

I couldn't find anything about this either.

You are correct about (6)! Apologies different user!

7/8 together. A Google scholar search of "software virtualization" or "software virtualization para virtualization" returned the following (plus many more) that refer to software virtualization in the same way I meant:

https://dl.acm.org/doi/abs/10.1145/1168918.1168860

https://ieeexplore.ieee.org/abstract/document/4709159/

https://ir.library.oregonstate.edu/dspace/handle/1957/9907

http://www.cs.toronto.edu/~demke/2227/S.14/Papers/p2-adams.p...

Finally I dopishly wrote "full hardware virtualization" when I meant "full virtualization", but for posterity here is VMware doc on it vs paravirtualzation vs hardware-assisted virtualzation: https://www.vmware.com/content/dam/digitalmarketing/vmware/e...

A virtual machine does not imply virtualization, and can use an emulated CPU.

If you look at the original post, it says "virtualization" in big bold letters. The precise definition of the term "VM" may perhaps be debatable, but I don't think it's fair to market your system as supporting Linux VMs, when in fact, you're emulating the CPU instead of virtualizing.

More importantly, I still haven't got the slightest idea what a "software VM" means, either. It's a term that I've never seen before. I even did an online search and found nothing.

Visit Wikipedia, in the search field type "virtual machine" but do not hit enter or search. Notice the text in the immediate results says "software that emulates an entire computer." Now, visit the page[1]: "...a virtual machine (VM) is an emulation of a computer system." This says nothing about whether the virtualization is entirely software, assisted by hardware, or entirely hardware.

A "software virtual machine" is a disambiguation that I chose indicating that the "machine" is implemented entirely in software with no help from special silicon (contrast with [2]). I can't fathom why that would be so controversial.

The entire thread comes down to this: the demo of x86 Linux running on Apple Silicon could very easily have been running in a virtual machine made entirely of software. No one claimed, as I recall, that Silicon implemented any hardware assistance for executing x86 code. There might even be IP issues doing that (IP - intellectual property, not "internet protocol".)