Intel Virtualization and Apple Silicon | Hacker News Reader

Intel Virtualization and Apple Silicon

highcaffeinecontent.com

207 pointszdw107 comments

Surprised this is so new to people - ESXi has been free for years and is the basis for every VMware setup out there. People have been virtualizing MacOS since it was called OSX :-)

Using VMware Fusion in this way does sound useful in that it sort of replicates the experience you get with vSphere (for managing the VM's) which is really only something corporate clients would be using and is definitely not free.

That being said you could really do this with almost any hypervisor running on a machine at home (or in the cloud, or anywhere) and then connect and control those VM's from a laptop with internet. RDP, VNC, Teamviewer, Citrix, Horizon, whatever - it is just basically us coming full circle back to the dumb Terminal to Mainframe connection but over the internet.

> Surprised this is so new to people - ESXi has been free for years

> Using VMware Fusion in this way does sound useful in that it sort of replicates the experience you get with vSphere

VMware's first big product, long before ESX (and later ESXi), was VMware Workstation, released in 1999, which provided x86 virtualization and ran on (and virtualized) both Windows and Linux. It was the first commercial product to easily enable people to run Windows without having to leave their Linux environment. VMware Fusion is essentially the same product, just the spiritual successor to that product that runs on macOS (along with a bunch of modern optimizations, of course) going back to 1999.

I'm using second product - Connectix VirtualPC (for Windows) because at this time this was only that can run on Windows98SE. VMW require NT/2K at this time. But i couldn't migrate to NT platform due project limitations.

VPC works flawless in version 4, but for version 5 they require NT platform too.

Connectix VirtualPC started as a Mac product.

I installed Windows 95 on a Mac with VirtualPC back in... 1995. All 26 floppies of it. Installation done, it rebooted the virtual box, and I had to go to a meeting. Cane back an hour or so later, and it had just finished booting up. Good times.

Then Microsoft bought them, and I think afterwards the Mac version was no longer supported.

(PowerPC was a consortium of Motorola, IBM, and Apple, and was made out of silicon... so technically, that would be "Intel Virtualization on Apple Silicon". :-) )

Virtual PC on Mac got at least one major release after Microsoft bought Connectix, but that was right around the same time as Apple's transition to Intel. MS apparently didn't consider it worthwhile to port VPC to run natively on Intel OSX (which would've been a pretty big change in direction for the product-- moving it to native virtualization like on Windows rather than the dynamic recompilation approach that the Mac version used).

Also at this time processors for Windows comes with virtualization extensions - VT-x and AMD-V. So Microsoft focused on making their product better just for Windows.

Later they're expanding into Virtual Server and famous "Windows XP Mode" on Win7.

At the end everything was merged into Hyper-V.

Even though I do not use it professionally I think Hyper-V is very underrated - it is built in for free to many versions of Windows 10 and is pretty easy to get a VM running even by the average power user type person.

Running your VMs on another machine is not just good for virtualizing Intel while using an Apple Silicon machine -- it's also awesome just to offload resource utilization to another machine. Throw Wireguard on your home network and you can work at the coffee shop all day without plugging in to charge... and you can close your laptop and your long-running tasks keep going.

What's your recommended wireguard setup?

As a happy user, I can highly recommend trying Tailscale. https://tailscale.com/kb/1086/tailscale-vs-wireguard/

Vanilla WireGuard is not hard to use at all, it's by far the easiest to configure VPN software.

Some routers have it built in, and some have packages to install it, and the official GUI clients work well: https://www.wireguard.com/install/

If you can't run it on your router, you can just set it up on a linux VM and forward a port as necessary.

The configuration file is quite simple once you get the hang of it, you've basically just got:

1. a public/private key that you exchange between connected machines

2. the address of the tunnel, locally

3. the address/port to get to the other end

4. what traffic you want to send over the tunnel

An easier method is to just use PiVPN https://www.pivpn.io. It's much faster to setup and easier to manage with extremely simple commands.

I've looked at PiVPN, but it looks like more work than many of the packages available for routers. e.g. dd-wrt, pfsense, opnsense all have point-and-click configuration pages.

I'm not sure how this is possible. With pivpn:

install pivpn, open router port & forward (automatically configured and optionally changed during setup), type 'pivpn add' at the command line for every device I want. Copy the outputted config files to device and import into wg client. Done.

Not at all. Maybe three terminal commands tops to configure and add a user for yourself. Once the text config file is generated just import it on your client and you are done. As long as you have forwarded the appropriate port through your firewall that's literally all it is to it.

Of course if you have a firewall with it built in, then use that. However for pfSense and OpnSense wireguard runs in user mode and not the kernel, so depending on your traffic the pi may be faster since wireguard is compiled into the kernel with piVPN. Which can cause fun issues if there is a kernel update you aren't prepared for - was my only real problem with piVPN/Wireguard but it wasn't too hard to recover once I figured out what was going on :)

ZeroTier is even easier. 1. Install the package, 2. zerotier-cli join (your network id), 3. Flip the new node state to enabled in the control panel. No other config need unless you want specific network config/filters.

You might want to look into Tailscale if you're just starting out. Configure a small VM to act as a Tailscale subnet router for whatever network your macOS VMs are on, after which you will be ready to access them remotely from the coffee shop. If you prefer to have access to ESXi from a separate VLAN/network, then get the Pro plan and setup a 2nd subnet router for your hypervisor, NAS, etc.

1.) Install wireguard.

2.) Install pivpn.

3.) (Optional) Modify makeCONF.sh/removeCONF.sh if needed.

I use ZeroTier for that use case. Is Wireguard similar?

ZeroTier does a lot more for you than pure Wireguard. OTOH, Wireguard tends to have a lot more bandwidth than ZT in my experience.

If you want a similar product to ZeroTier but one that uses Wireguard under the hood, Tailscale is what I've been using to connect to my home network.

Last I used ZT, it was pretty flaky on Windows. Network would just stop working sometimes, client would bug out and I would have to uninstall/reinstall. Slow speeds also; wireguard is much faster.

Except now you require 2 machines and internet access instead of just one machine. You could offload your VMs before too. Keeping them all local and working offline are still important capabilities.

That depends on your use case, of course. I'm not going to be working offline at a coffee shop anyway.

Those cases are few and far between for me - usually limited to catching flights and my laptop can be used locally as a back up.

You can also dockerize your workloads and use an environment variable to use your ssh remote as the docker machine.

Catching a train home from the office and using a lightweight, cool to the touch, thin laptop that is offloading commands via 4g to my desktop PC with 24 cores such that I can't tell that workloads are not running locally is such a fantastic development experience.

Sure I need to be always online on 1 extra device, but the second connection can be significantly lower bandwidth while the server will access things over a hardwired 100 megabit connection.

All this is fine and dandy if you don't care about keyboard latency and writing code. If you're running a large workload, this is the way to go. However, using a VM to write code is just painful.

I don't write the code in a VM, I just run the code in a VM. It depends on your development stack, but many stacks have options for running code on remote machines. For example, you can configure a Docker context running on another machine.

    docker context create remote ‐‐docker “host=ssh://user@remotemachine”

Yes, PyCharm also has remote options.

But I see a lot of people writing code on remote machines. A good keyboard + baremetal FreeBSD (no GUI) machine is an absolute pleasure to write code on.

At 80x25 or are you doing some 'mad tricks' with your kernconf to get it higher res?

Got a diff on /usr/src/sys/amd64/conf/GENERIC (or whatever you call yours)?

mosh + tmux makes this largely a non-issue.

(source: I've spent the last 5 years working primarily on a physical server in a proper dc and just connecting into it from thin/fanless laptops)

Try the VSCode remote SSH extension. You'll often forget that you're actually editing on a remote machine.

The interactive debugger works and everything.

I have been using this for the last year or so, I was surprised finding out that it's not a standard feature of other IDE/editors.

Intellij doesn't support this and they are releasing a new development editor designed to offload compilation to their own build servers.

I don’t think developers write code in VMs. I use vagrant and mount folders as needed between the host and the guest.

Did this in university, where I carried a lightweight disposable laptop and had a desktop workstation (and later a server) at home. I used OpenVPN to remote in and got to smirk at the students who either had slow builds, a heavy gaming laptop, or had the money for a top-line MBP or similar.

Due to the slow upload speed of my DSL connection, this forced me to become comfortable with TUI apps and a multiplixed terminal environment, with the occassional use of Xpra when absolutely necessary.

If you like MacOS, that is the most affordable performace too. I minimize the Apple premium cost by buying an Macbook Air, then have custom build PC at home. Add all the CPU, GPU and storage space at home. Also allows gaming when I want.

I’ve been doing this for many years now—happiness is not keeping one or more beasts of machines in my backpack—and I still do this over SSH tunnels (maybe I should finally migrate to the WireGuard hotness).

I was doing a lot of work with SSH tunnels, but it seems Tailscale is an easy way into WireGuard.

For now I just have one node running on my office setup to access everything there and it seems to work pretty well from home. More convenient than mapping ports anyway.

Although...I don't quite grok how it might be more secure than something like certificate based SSH, seems like the Tailscale attack surface might be bigger even though you can go back and close off the firewall stuff that allows SSH to work. Would seem that instead of concentrating on breaking your (probably obscure) public IP address / port combo, an attacker would simply go to Tailscale and attack that instead.

It sounds like you and I have a similar setup. I recently set up WireGuard using the linuxserver/wireguard image, and although it was really easy (took ~5 minutes to grok and get running) and is useful for my phone to get pihole dns from my home network, I don't think I get any benefits when using wireguard on my laptop vs just using sshuttle, which doesn't require any extra software at home.

+1 I have been doing this for over a year now. Great workflow and tools like VSCode SSH extension make it appear that everything is happening locally.

I'm always paranoid about things catching fire though.

I rarely leave anything turned on when I leave the house.

This article has a really misleading feel to it. It appears like it was designed to lure clicks through the title, but in fact it's simply suggesting a bare-metal hypervisor on another machine.

And a hackintoshed one, at that!

This is pure gold, thank you! I've been wondering what to do about my VMware machines. Some run various windows apps, and one runs Aperture, the only way to access my photo library after Apple screwed me royally by abandoning Aperture. I didn't know what ESXi was.

If you only need to run Aperture I've had good success using Retroactive[1] on my M1 Max machine running Monterrey to patch it to run under Rosetta. Almost everything works fine except anything tied to iCloud and some of the video and slideshow features. It's been great in letting me access my older photos which have adjustments that I don't want to re-create in Lightroom.

It will copy over the missing frameworks and modify the app so the OS will run it even though it's unsupported, if you're interested there is a technical breakdown that goes into detail[2].

[1] https://github.com/cormiertyshawn895/Retroactive

[2] https://medium.com/@cormiertyshawn895/deep-dive-how-does-ret...

Nice - thanks for sharing this!

Alas, while Retroactive is fantastic, it doesn't solve the problem of Aperture, because RAW decoding doesn't seem to work. My TIFF and JPEG files are visible, but RAWs show up as black previews and can't be decoded.

Yes, I wish Apple would sell Aperture to someone who will pick it back up. There still is nothing remotely like it.

> One advantage very specific to the low-level way ESXi works, because there's no underlying operating system to get in the way, is the ability to 'pass through' real hardware to the virtual machine — say a specific network card, or a USB hub. Or most importantly for macOS: a GPU.

This is not at all specific to ESXi... you can do PCIe and USB passthrough with qemu running on Linux. The operating system doesn't get in the way.

Quibbling a bit with the details here.

> This is not at all specific to ESXi... you can do PCIe and USB passthrough with qemu running on Linux.

This is true.

> The operating system doesn't get in the way.

This isn't, really, at least not qemu does quite a bit of fiddling under the hoot -- without VFIO or legacy KVM pass-through, the operating system very much gets in the way. Linux now provides facilities that allow qemu to pass through devices -- that is, it provides the APIs necessary to move it back out of the way.

I don't know to what extent ESXi looks like pure-kernel setup of passthrough vruss KVM_ASSIGN_DEVICE versus VFIO -- that would be quite interesting.

GPU passthrough specifically is tricky - I know ESXi does it well but you of course have to dedicate a physical GPU to a single VM. No sharing.

I know they were working on it with Hyper-V awhile back but who knows howe well it works

Came here looking for a way to run Windows VMs on Apple silicon (Parallels and VMware can’t boot old win7/10 images on M1 ARM chips) and instead the solution was to use a second PC to run the VMs (with ESXi) and serve these to the brand new M1 based Mac. Oh well, we’ll have to wait until Parallels/VMware sort out the emulation and we’ll be able to run x86 bases OSes at a decent speed on M1.

For those thinking they can use a cheap old Mac Pro 4.1/5.1 or other old 2009-2012-ish machines as an ESXi setup which is exactly what I attempted to do... don't if you need anything involving PCIe passthrough - the Intel 5520 chipset has a hardware bug in interrupt remapping.

Yikes - that's good to know!

Proxmox is great for this as well, minus VMWare’s optimized “VNC” compression and ability to pass through hardware devices.

Proxmox has SPICE which is what I use for VDI instances on my server - it's not as good as native but much better than VNC (in my eyes it performs similarly to RDP and is not as fast as Citrix or Horizon). I don't think it supports Mac guests though, only Windows and Linux.

I've never used it, but I guarantee that Proxmox can pass through PCIe devices with VFIO, and USB requires no hardware support at all.

vmkernel hardware passthrough has exactly the same requirements (IOMMU support).

My statement was vague, apologies. I meant client-side pass through of devices. (such as plugging a USB device into your client, and the remote VM believing it is local)

SPICE on Proxmox may support this but I find it incredibly clumsy to configure.

This is a great tutorial, thanks.

Slight aside, with the incredible performance possible with Rosetta on Apple silicone, what are the chances of using it as part of a virtual machine emulating x86-64 with similar performance? I understand that Rosetta “recompiles” to ARM which would probably be harder with a VM but I think it also does real-time conversion to. I have no idea about he details but could QEMU use Rosetta under the hood for emulation?

Literally second paragraph of the article mention QEMU-based app that allows you to virtualise x64 machines on M1 - UTM:

https://mac.getutm.app/

The thing is, it only supports windows xp + windows 7, and not 8/10/11 (those are supported as arm64). Apparently the amount of work is massive (so are performance hits for the Windows XP/7 when you run it).

It's one thing to emulate apps, another entirely to emulate whole advanced OS.

Yes, I read the article and was in fact aware of UTM, also noted:

> the performance penalty is significant

I was asking the genuine question as to if it would be theoretically possible to use Rosetta as part of the emulation for a VM to potentially help it to perform better?

For example could a tool be installed within a guest OS that effectively send binaries to Rosetta for translation before execution?

It just seems to me that if Apple have done such an incredible job with Rosetta wouldn’t it be brilliant if it was possible to use that within an emulated VM on Apple silicone.

No rosetta doesn't do system level emulation. System vs app level are distinctly different problems.

Att the app level rosetta can operate at link time (yes it does AoT, but that's essentially just caching), where it knows things like what things are text regions, what the entry points are, it can assume that the code is "correct" (if your code is buggy natively, then rosetta won't stop it going wrong), etc.

System level emulation means rosetta (or whatever) loses that transparency, so can't precompile, and can't make any assumptions about code behaving properly. To get an idea of what the performance impact is you should check the performance of JIT compiled code under rosetta. Rosetta ensures that the JIT will work, but the perf hit is staggering. System level emulation basically means treating everything as being JIT code.

I'm sure a system level emulator (Qemu?) could do better than for example just slapping a system emulator mode into rosetta, but that would be because rosetta is optimized for app level emulation.

Rosetta is deliberately just enough to run macOS/Intel apps on macOS/ARM. It's theoretically possible for Apple to enhance Rosetta to emulate a whole VM instead of a process but they don't feel they need that so they're not going to.

If Apple can do it, will VMWare / Parallels eventually be able to achieve similar levels of performance or is there something physically stopping it from emulating a whole OS's required processor features?

No, Rosetta is private and not exposed to third party applications, and not really appropriate for supporting a VM because it is designed for apps running in userspace.

Are there any architectural lessons that could be learnt from Rosetta and applied to QEMU?

So for example the “precompiling” of a binary, would that be possible with a VM?

It’s not impossible, but it would be significantly more difficult and limited in scope.

What about the hardware support for total store order that Rosetta uses to provide the x86 concurrency model? Is that something third-party implementations could conceivably use?

As long as that is available, it seems like third-party virtualization software could get similar performance to Rosetta, and potentially experiment with attempting to support system-level CPU emulation.

TSO is only enabled for processes using Rosetta, so if you want to use it you’ll either need to hijack an existing process or expose it via a kext that patches in the appropriate flags. TSO is also only one of the puzzle pieces for why Rosetta is fast: you’ll also have to figure out how to avoid a software MMU if you want similar performance. This is difficult for whole-system emulation.

Not very good, even in Docker.

Just noticed today that I have a python script that takes 4.5 minutes to process on m1 but only 2.5-3 minutes on an intel Mac.

What do you mean with Docker? Docker isn't mentioned in the article. Probably your container would run faster when your container image was native arm64 architecture and there wouldn't be emulation.

I switched to a "physicalization" model a long time ago.

It's hard to scale in a bigger company, but it just makes more sense to me to segment the user experience layer from the random compute layer.

I run a dozen servers for a little hobby project that has a few customers. One unexpected cool way this has benefited me is I eliminated the need to lug a laptop anywhere. I have a little tools environment setup in iSH on my iPhone and can quickly troubleshoot most things or perform various tasks on the command line. A few others that haven't been scripted yet I can run by SSHing in.

This isn't really all that new, running macOS on non-Apple hardware has been done, including virtualisation. It also is pretty much the default way to do it unless for whatever reason you need to run it locally (i.e. if you don't have a server or if you need local hardware access).

At first I thought this as a 10 year old article, but it appears it really was a re-discovery of the same thing you do with KVM, Xen, Hyper-V, ESXi, VirtualBox etc. for years now.

Intel NUCs are now Apple's new biggest dongle!

Please forgive my ignorance - what are the benefits of this approach, as opposed to Remote Desktop into a Windows machine?

The ability to have multiple VMs with many different operating systems.

But you could easily do that with ESXi free running on your home desktop, running multiple VM's (lets say Windows, linux, BSD, etc etc) and then RDP from a mobile laptop to any of those VM's.

Not needing to run Windows 24/7 to support non-Windows work - one practical benefit is that you might not need to install as many security patches or they might install faster. Another reason might be to avoid running Windows entirely ;-)

Using Clover is incorrect. It is outdated, full of hacks and nowadays based on OpenCore anyway. However, the right way to go about putting macOS on custom hardware is using OpenCore directly. Depending on your hardware, you may actually get native level of experience or even sometimes better.

I had an intel MacBook that was a good fit for doing reverse engineering work on malware. I suppose this method would be kind of a compromise for those that still want to use Mac system, even know it is incompatible with most of their RE tools/virtual machines.

Anyone used Ghirda on a M1 Mac?

Plenty of anecdotal evidence to suggest Ghidra works fine on Apple Silicon (I also haven’t personally run up against anything unexpected)

Yup, it works.

I’ve been using Ghidra on Apple silicon even before M1 came out, it works quite well. I’d suggest building it yourself so you can get native decompilers, though.

What Apple Silicon before M1? The Ax Processors?

A12Z on the DTK.

> At least now I can comfortably work on Apple Silicon without losing my VMware library.

Adding an intel NUC to a MacBook Air probably increases the weight – and reduces the time you can run your VMs on battery.

Vey cool. This has been my deal breaker. I need to play with x32 and x64 assembly all the time.

You know what would be awesome though? If qubes os with its Xen stuff allowed this sort of remote access.

> I need to play with x32 and x64 assembly all the time.

This should work fine in Rosetta?

I meant VM guests not host apps.

Wait, MS isn't letting you buy windows/arm for AS?

Correct. Their license currently prohibits it.

It’s much cheaper to install Linux and libvirt on an old PC and then use virt-manager as the client on the Mac. This has no annual license fee.

Is there anything like this, but for docker (compose)?

Not sure I understand the question, but you can point your docker socket configuration at an external machine of any architecture. That means you can run compose locally and it build/runs remotely.

Go check out docker-machine, but a fork: https://docs.gitlab.com/runner/executors/docker_machine.html...

Because Docker stopped maintaining it.

I don't know why they recommend the $100 Fusion Pro? You can do the exact same with VMware Remote Console which is free :)

Good to know. Not sure if this is still relevant but talks about how to use VMware Remote Console with free ESXi: https://derflounder.wordpress.com/2015/04/18/using-vmwares-s...

You can get VMware Remote Console from the App Store: https://apps.apple.com/ca/app/vmware-remote-console/id123024... or at https://www.vmware.com/go/download-vmrc

Why is Apple virtualizing Intel, but is Intel not virtualizing Apple hardware?

Intel is incumbent, so most software (short of ios/android apps) are already written/complied for x86. Because of this, there's no real need for intel/amd to support ARM.

1. Smaller companies like Corellium have already tried making that mistake, and Apple sends their lawyer death squads to their doorstep every time they try pushing a new feature.

2. There aren't really any killer apps for ARM yet. ARM itself is a fine ISA, but the majority of consumer desktop software and server software is still x86 first (particularly on platforms that Intel sells for). Until that changes, I doubt Intel will really care all that much. When it does change, I'm inclined to believe that the rest of the industry will be eyeing even more minimal ISAs like RISC-V. That's mostly just speculation though.

Well, first off, Apple isn't virtualizing Intel, they're emulating it. "Virtualization" implies that your CPU has special hardware support for running a kernel as if it were a regular OS process; which almost never crosses architecture boundaries. Apple does have special support in M1 for certain x86 features like total store order, but most of the work is done in software to translate binaries to ARM64 before they are executed. They just omitted emulating anything beyond Intel macOS user-mode, because that's relatively easy and lightweight[0].

Intel doesn't virtualize Apple hardware because it doesn't make business sense and is questionably legal (see also: Corellium). Apple silicon is just-different-enough from a regular ARM64 design that anything that can actually boot macOS almost certainly exists purely to break the macOS license agreement.

[0] Specifically, having all your code neatly packaged into executables and libraries means that you can AOT-compile nearly everything and not have to actually interpret or JIT x86 code. There's also no need to emulate, say, x86 context switches or page table walking; you just stop emulation, run the syscall on the ARM side, and then restart the emulation once that's done.

Of course, they still have a JIT, just in case you decide to run a binary that also contains a JIT.

> They just omitted emulating anything beyond Intel macOS user-mode, because that's relatively easy and lightweight

Is it that? I would’ve assumed it’s because the patents hadn’t expired yet.

I'm pretty sure some of the AVX and SSE instructions are patented still, and those are accessible in user-mode and supported by Rosetta 2.

Apple almost certainly was able to either brow-beat Intel into a license, or design around the claims of said patents[0]. FWIW Apple has UI strings for when Rosetta 2 is disabled, so they might have had to consider cases where their license expires or something.

[0] Notably I have no idea if patent claims on a hardware instruction & it's associated implementation would automatically prohibit emulation of that instruction, or if emulation would be a separate patentable.

> Rosetta translates all x86_64 instructions, but it doesn’t support the execution of some newer instruction sets and processor features, such as AVX, AVX2, and AVX512 vector instructions.

https://developer.apple.com/documentation/apple-silicon/abou...

> anything that can actually boot macOS almost certainly exists purely to break the macOS license agreement.

Apple provides a hypervisor framework that emulates a standard ARM64 system and macOS ARM64 kernels that run on it. This is how macOS on Apple Silicon virtualization works. It's not really Apple Silicon, it's boring standard ARM on Apple Silicon. Soon after Apple started shipping these kernels, before it was even announced, someone tried it on QEMU on another platform and found it worked just fine in single user mode.

The only catch is the macOS UI requires Metal, and paravirtualized graphics are implemented using serialized Metal passthrough. In other words, if you want to boot ARM64 macOS to a desktop, you have to implement Metal in your VM host.

Aah, TIL.

Do we have paravirtualized Metal support in Linux guests yet? That would probably neatly brush away any legal claims against third-party hosts for implementing it on non-Apple hardware.

Mac simply isn't a significant part of intel's market, and running Mac apps (the things you'd want to emulate) essentially requires a macOS license, which means owning a Mac.

> it may never be feasible to do so on Apple Silicon at a useful speed

What a uninspiring and uninteresting start of the article.

Does this method allow you to pass-through USB ports?

How is it different from running an XPRA server on the remote machine?

Reply on news.ycombinator.com