Vftool runs Linux virtual machines in macOS Big Sur | Hacker News Reader

Vftool runs Linux virtual machines in macOS Big Sur

github.com

195 pointssyrusakbary86 comments

> The bridging requires the binary to have the com.apple.vm.networking entitlement, and Apple docs helpfully give this note:

>> Note This entitlement is restricted to developers of virtualization software. To request this entitlement, contact your Apple representative.

I have not done development in Xcode for some time, but the first thing I wanted to do for an internal project was hamstrung by restricted entitlements. Luckily, it was just an experiment, but it makes it feel like this is coming up a lot. Is there still a way around this (i.e. to run your own code on macOS) ?

This isn’t the first time I’ve seen a project based around Virtualization.framework; that’s both exciting and disconcerting. Issues like entitlement deals (that I will never be able to obtain) make the wide open path forward look a lot like a narrow track, the direction of which only Apple controls.

I feel like unless you are one of the bigger players, you never get a reply. I work for a company that develop apps, and we got into a situation where we had a need to be able to disable NFC temporarily, because Apple Pay was triggering when it shouldn't (on a ticket scanning gate). Hidden deep in an obscure doc, I found a notice about a special entitlement you could request by emailing the Apple Pay support team. Emailed them two times over the last two years, never even got a reply

Apple customers surrender their phones, and experiences, to Apple's custody. That walled garden is discussed every day here.

But I wonder how much 'normal' Apple consumers realize how shoddy some of that custody really is.

I bet your customers blamed you for that bug, not Apple?

Of course they gonna blame the creator of the app, not Apple. The customer has no idea about the development and what entitlements are available or not. Why should they blame Apple, I don't understand your point.

As the creator, notify your users about it and there you can blame Apple as long as you want. But if the users are not notified, they don't really know that the creators are waiting for Apple.

As an user I literally run into this usecase: Germany, local bus company, I've run out of cash, so I just bought online ticket via Deutsche Bahn app (it's a QR code). When you enter the bus, you need to scan this QR code, but for some stupid/smart reason Apple Pay triggers and hides this screen, and of course when you try to debug Apple Pay screen is gone, QR code is back. Trying to explain to the bus driver that this was some stupid Apple Pay interaction was quite an experience. Lesson learned: make sure to always have some spare coins with you.

Maybe the QR code scanner and the NFC POS have been installed in a poor way that make the two options collide. Since QR code could be provided from general purpose apps (browser, photo gallery) there is no software solution to this.

Hey, I may know some people who’d be very interested in these failure modes on the team. Which bus company? Any chance you know what terminal they use? Would you mind getting in touch off HN to discuss further?

Same problem in France, public train company SNCF. All year long I had to ask the (human) controllers to open the glass barrier supposed to be opened with a QR scanner in the gates.

Same flawed workflow. Show QR to scanner => not work => check phone on apple pay mode with credit cards => repeat => ask the (human) controllers.

You can ping me @ lucas@concpt.io

I have encountered the same problem with the tunnelbana in Stockholm. Normally I use a card, but on a few occasions I have left it at home or forgotten to top it up, and bought a ticket with the app.

When it wasn't scanning and I took my phone off the scanner, I briefly glimpsed the Apple Pay screen before it animated away, so I realised I was "holding it wrong", and was able to get it to scan by holding it at a different angle to keep the NFC sensor at the top of the phone away from reader, but I can imagine a less technically minded user than myself might have serious problems.

I don't know anything about the scanner or NFC standards used by this system, sorry.

> I have encountered the same problem with the tunnelbana in Stockholm.

It took me a second to realize you weren't talking about the NFC chip, which is codenamed Stockholm ;)

I encourage developers not to discuss issues with Apple via these kind of back door channels. It only encourages them to keep things locked down and inaccessible for everyone.

If Apple wants feedback from its developers, it should try actually listening to all of them, not just the ones that make enough noise on social media.

(I'm assuming the parent comment is from someone at Apple. If not, my point still stands in general.)

Isn’t that half of reason why HN exists? How many times have there been issues with FB, Google, etc where the poster ask for help on HN? HN may be the only connection to SV insiders that many users have.

This isn’t to say HN is a good replacement for a proper support channel, but sometimes you need a little help. These are big companies, so these small issues (for Apple) don’t always make it to the right team. The parent poster here isn’t even the dev of the app in question. How are they supposed to know where to report that?

I think the point being made was that using this ad-hoc channel perpetuates it and prevents an actual way to contact people at Apple from forming.

Ha. Unless you cannot Pay cash anymore within the bus, e.g. Chemnitz https://www.cvag.de/de/Tickets_Tarife/Bargeldlos_10842.html

Did you have your ticket in the Apple wallet or the DB App? I have found that DB Scanners don’t like the wallet version.

Yes! This was the exact same use case we had!

A lot of transit gates do this because you can use Apple Pay or a contactless credit card directly on the gate without going through the intermediate stage of a ticket.

This is very handy in Chicago if you are making a single leg journey. Due to Apple Pay’s privacy architecture it can’t recognize transfers and always charges full fare.

This has nothing to do with Apple's architecture but is probably a limitation of the transit agency's backend systems.

The card number of a given card on a given device does not change between taps for Apple Pay. Otherwise, tapping for refunds would not work, among other things.

> Due to Apple Pay’s privacy architecture it can’t recognize transfers and always charges full fare.

I don't think this is the case—in London, I'm pretty sure you can pay with Apple Pay and have your fares "capped" (so you never pay more than the price of a full-day ticket). You can also set up an account with your card number (the one on your physical card) and view any recent journeys you've made.

https://www.ventrachicago.com/how-to/mobile-wallet-apps/ This claims PAYG works with transfers now, although I thought it always did for me (but I don't have an iDevice).

In my opinion, Apple's over-excitable field detection (which considers every NFC field a potential for an Apple Pay transaction, not just payment and transit terminals) is mostly a nuisance, at least for Face ID phones.

@djxfade: I might know some people on the team who’d be very interested to hear about it. what’s the best way to get in touch?

Unrelated question as an outside observer: how is it possible to get interactions like these as opposed to "file a feedback"?

Hi, you can contact me on thomas(at)apility(dot).no

I recently learned about the "technical support incident" help you can get from Apple, you get two such incidents included with your annual developer program - maybe worth a go, I believe they respond to you quite quickly from what I've heard: https://developer.apple.com/support/technical/

> Is there still a way around this (i.e. to run your own code on macOS) ?

In Big Sur I’ve noticed there’s now a “Developer Tools” entitlement in Privacy. I’m not sure exactly what it allows executables to do, but as it starts off with Terminal.app as a suggested entry, I’d assume it at least allows the given app to spawn unsigned binaries.

I think this lets apps ignore things like quarantine and data vaults.

Besides a cunning and evil plan to gradually restrict user freedom (/s) I can also imagine this is meant to control an OS feature that can be used to nestle some pretty audacious privacy-violating software, away from the rest of the OS restrictions.

Perhaps Apple will figure out some software control to sidestep this administration overhead, or just make it an xtra license to milk more money. After all Microsoft has been doing this since the beginning...

Yes. Freedom in computing is a very sharp double-edged sword. On one hand, it gives you control of your device and your computing world and lets you create anything you want. On the other hand, it lets malware and spyware take total control of your machine and steal all your personal data or extort you with ransomware.

Security is always somewhat like that. It really wouldn't be that hard for a determined burglar to break into my house. I could make it a lot harder, but doing so would come at the expense of a great deal of my own convenience and even freedom. I don't want bars on my windows and I don't want to have to authenticate against a panel or app/keyfob every single time I enter or leave my home.

My neighborhood has police and I live in a very civilized place. The Internet is an open war zone overrun with criminals and malicious privacy-invading corporations (and governments, and not just your own government) bent on subjecting you to a total panopticon. Computers must be secure in a much more hostile environment than my house or my car. A computer today must be more like a hardened compound in a failed state than a house in a typical developed world city or suburb.

I've been saying "the Internet is a failed state" for a while. Apple with its locked down walled garden is like one of those private mercenary security armies that provide security to the rich in failed states.

In this example, I see Apple as one of those armies that only allows you to take groceries from their blessed vendors into the walls (some rumor about razor blades in fruit from the other shops?). They charge those grocery shops for this privilege.

I don't disagree with the comment's observations, though I don't see a path to get to a "friendly" internet without sacrificing other important freedoms.

Problem is the razor blades are not a rumor. The food really is full of poison and razor blades. The Internet is a war zone. Tech-savvy people can be fairly good at spotting this stuff and knowing where to get things that are safe, but tech savvy people are not the majority of the market.

I understand the concern, but this seems to derail the whole discussion of this lightweight tool supporting virtualization on m1/Big Sur.

That was not my intention, but I can see how that would be the case. As I have seen several mentions of wrappers for Virtualization.framework, the common thread that jumped out to me was the continued march toward dependence on Apple’s provisions. I do hope that the rest of the thread continues to tend toward the merits of this particular tool.

yeah, I can see the topic being worth a blog post and a dedicated HN discussion, since there are pros and cons of locking things up like this.

For that specific entitlement, it's overridden by vmnet.framework if you call it from root.

However, Virtualization.framework runs out of process, and Apple forgot to check for root if the entitlement isn't present. That should get solved soon.

> >> Note This entitlement is restricted to developers of virtualization software. To request this entitlement, contact your Apple representative.

I can understand the desire, for convenience's sake of testing stuff quickly, to directly virtualize a guest linux OS on a developer's workstation laptop or desktop running MacOS.

But that's an absolute non-starter for me. Contact your sales representative for an entitlement? Forget that, I'll keep on using Linux kernel based virtualization with either xen or kvm, and the equivalent of xen domU PV or HVM on a normal x86-64 server platform, controlled remotely from my laptop, thanks.

The whole language in that sounds like they hired some MBAs away from Oracle.

Or you could wait until I port Linux to these Macs and then use KVM on that :-) (shameless plug)

Seriously though, the machines are open, you can disable all the security to various degrees. The entitlement stuff is for deploying to vanilla Macs with all the codesigning requirements enabled (e.g. via the App Store), for security. It doesn't stop you from bypassing it on your own machines.

If you disable SIP, iOS apps stop working.

https://forums.macrumors.com/threads/if-you-disable-sip-all-...

Your line was my line for a long time. But I feel like this is a significant step. For the first time, by taking control of your own device you’re loosing functionality.

That’s because iOS devices are locked down for security. iOS apps are given a promise of a trusted computing base which must be kept. Otherwise your company, government or criminal gives you a MacOS laptop with SIP disabled and can now trivially circumvent the privacy protections built into all iOS devices, including for data sync’d over the cloud with your other secure devices.

You mean like you can with any iOS device produced prior to 2018 via checkm8?

I think that is exactly their point. That is why disabling SIP is not a practical way to work around the restrictions applied to the user. There need to be more levels of control than just disabling SIP, so users don't have to sacrifice functionality and security to run the apps they want.

That would not quite be my point.

SIP is already fairly flexible, in that it allows you to selectively disable only certain protections (see csrutil's man page). The things people complain about being unable to do without disabling SIP—injecting into other processes, modifying system files, etc—are in fact the exact things SIP exists to prevent.

A lot of Mac users I've talked to, including technical ones who IMO should absolutely know better—appear to be under the impression that disabling any part of SIP will perform voodoo magic that makes them an instant hacker target. This just isn't true. Running `csrutil disable && csrutil enable --without debug` allows mac apps with root privledges to inject code into other processes, nothing more and nothing less. To the extent that allowing code injection is a security risk, this is a security risk. If you want to inject code, turn it off and have fun.

iOS apps are the first time to my knowledge that disabling SIP has actually broken something unrelated. (But do let me know if there's a --without-disabling-fairplay option, I don't have an M1 Mac and there's a dearth of information on this.)

As far as having a trusted computing base, well, I think anyone who believes that's actually possible on a mainstream end-user device is fooling themselves. Jailbreaks already exist, as does Corellium. If you're distributing a private app to employees on company computers, set up an MDM profile which prevents messing with SIP, and schedule regular, in-person device check ups to look for signs of user tampering. In any other situation, assume your software is going to be run in an untrusted environment at some point, because it will be.

I think this entitlement controls only the ability if you are not running as root. That makes perfect sense, because you don't want random user processes to be able to sniff your network traffic...

> A Boolean that indicates whether the app manages virtual network interfaces without escalating privileges to the root user.

https://developer.apple.com/documentation/bundleresources/en...

> I think this entitlement controls only the ability if you are not running as root.

Indeed, but why can't it be granted by root?

Well, if you don't care about security, you can always put the setuid bit on the binary (I suppose that would work?).

I agree that the situation is not ideal, but at least you can run it on your own computer if you actually want to...

In the past there were some bad actors doing shady stuff on iOS with this type of entitlement. I wonder if that influenced this type of restriction: https://guce.techcrunch.com/copyConsent?sessionId=3_cc-sessi...

Is it restricted for developer ID/notarized software, or just the App Store? The latter doesn’t surprise (or bother) me particularly, but if it’s the former it would be pretty surprising and annoying. In the past I’ve even used weird internal entitlements like the custom lisp sandbox directives just fine with signed binaries, though I haven’t tried to notarize them.

Imagine if you wanted to run a Linux program with KVM support, and you had to ask Linus to enable a flag on your Linus ID Account.

Eventually even Linux will diverge from its idealistic roots as our generation has left the world of living.

You can self-sign binaries to run on your own machine with the entitlement. You only need the entitlement from Apple if you want to distribute applications with the entitlement to other people.

https://twitter.com/Docker/status/1337163987641802752

> Releasing this week:

> - Docker Desktop 3.0

> - Docker M1 Mac Technical Preview

...anticipation builds...

It's worth jumping directly to some issue comments,

https://github.com/evansm7/vftool/issues/2#issuecomment-7354...

as this doesn't seem to work right away for popular options, e.g. I tried the Alpine aarch64 net boot and nothing happens.

I was then able to get Docker working with the Ubuntu VM (on m1)

Gosh, I hope the SSH performance is not representative of what the final products from VMware and Parallels will be. I've got about 750msec lag per keypress-- it's rather painful!

That's odd, I just booted it back up to check and ssh'd in, and there's no noticeable lag.

Hi! Has the VM with vftool been stable for you? I have tried it for Docker but either I got kernel panics in the VM or performance was poor with multiple virtual cores.

and in terms of speed, measured with scimark4, the VM is between rosetta2 and native.

It would be great if Apple would add support for running OCI containers. Then most people would not need VMs.

How could you do that without running a Linux kernel?

Implement Linux syscalls like Windows did.

But Windows already gave up on that (WSL2 is VM-based, with a Microsoft-built Linux kernel); why would Apple go down that road?

Microsoft gave up on the syscall-emulation approach mostly because of filesystem performance issues - WSL1 uses the host NTFS filesystem for everything whereas WSL2 uses a backing ext4 disk image, which means that as long as one stays in the "Linux world" the performance is way better.

That strategy did not work out well for WSL, so in WSL2 they chose to virtualize Linux instead: https://docs.microsoft.com/en-us/windows/wsl/compare-version...

It's not hard to imagine that Linux binary compatibility might work out much better on macOS than it did on Windows, due to macOS already being a Unix. FreeBSD has been doing it forever, though I'm not sure what the current state is with regards to performance and completeness.

While true, the closer Darwin is to Linux, it simultaneously eases implementation while disincentivizing doing it in the first place. Rarely you want to run a prebuilt standalone Linux binary on macOS; you just compile it to run on Darwin. In cases where you actually want Linux, you probably don't want even the slightest edge cases to be present. Besides, VMs are quite fast if you have hardware support.

Running docker provides that incentive. No matter how fast VMs are, they are still quite a jump from just running local processes, e.g. they consume resources while idle.

Also,.having to rebuild your binary just to run it under docker/linux means you cannot benefit from your compiler cache you just used minutes ago to run your unit tests locally (e.g. if you use tool chains like go or rust that do use compilation caches)

https://github.com/linux-noah/noah

Sadly not really under active development these days :(

Noah is archived. It has been forked 78 times. How do I find out who has taken over to continue maintain and actively updating a fork?

Or IOW is there an equivalent actively maintained?

In theory, one could build Darwin-based OCI images: the nix package manager does, in fact, build such images on macOS but there’s no platform to actually run them on. However, I’d really like this ability: the whole Linux VM overhead on macOS is annoying for development environments.

But are most container images are not for Linux?

Virtualization.framework appears to provide Virtio disk, network, etc. I wonder how Apple did it: I doubt they could copy GPL code from qemu. Maybe they copied something from one of the *BSD implementations? If they have implemented it from scratch, I wonder how buggy it is? Anyone has any information on this?

I think Virtualization.framework lays on top of Hypervisor.framework, which Apple has included since Yosemite (released in 2014). So while I can't refute your suspicions, they would have had the time to implement it from scratch.

I’ve been trying to understand the difference. Even have an open unanswered question on stack overflow.

What’s the diff between Hypervisor and virtualisation frameworks? Docs are ambiguous.

I wonder how this compares to xhyve, which I use on Intel Mac. It's nice to see something like it for ARM.

It appears xhyve is built on Hypervisor.framework while vftool is built on Virtualization.framework. Is that the main difference? What does that mean?

https://developer.apple.com/documentation/virtualization?lan...

https://developer.apple.com/documentation/hypervisor?languag...

Looks like Virtualization.framework is newer (Big Sur only) and uses Objective C classes rather than C functions.

I know from the xhyve README that it contains code responsible for booting a Linux kernel, which suggests Hypervisor.framework does not take care of that for you. The Virtualization.framework API on the other hand takes a linux kernel + ramdisk as its inputs.

So it sounds like the framework vftool is built on is more high-level than that of xhyve, and like vftool is Linux-only.

Yeah, Hypervisor.framework is an analogue of KVM (i.e. just manual VM contexts, the caller needs to handle exits to provide virtual hardware). Virtualization.framework is closer to something like QEMU in library form.

I’m using xhyve as well, but I’m having trouble getting it to boot newer kernels. 4.4 era works, (16.04) but 18.04 and 20.04 have various failures. I note that docker is built on xhyve and doesn’t seem to have this issue. So it’s not fundamental, but the fix hasn’t made it back to thx mostly abandoned xhyve.

Ubuntu 18.04 (Linux 4.15) in xhyve works fine for me. I tried to get 20.04 set up a while ago but I gave up because its installer kept flaking out on me, something about the disk if I remember correctly.

Thanks to another commenter I found https://github.com/evansm7/vftool/issues/2#issuecomment-7354... which is exactly what I needed for Ubuntu 20.04: the idea to use cloud images which bypasses the silly installer.

If you just want to run a recent Ubuntu VM, you can use https://multipass.run/ from Canonical. (It uses hyperkit rather than xhyve as the hypervisor.)

I finally wound up with multi pass, but on the Mac it’s limited to ubuntu images, and I occasionally need to spin up arbitrary distros for testing things.

Docker is using hyperkit [1], an xhyve fork that's still actively developed.

[1] https://github.com/moby/hyperkit

Do any of these solutions work for a thin mac on mac virtualization? Ideally opensource and not veertu. :)

Seems interesting but far too immature for me to go anywhere near.

wow, I am surprised Apple allows this

Allow what? This is a public API: https://developer.apple.com/documentation/virtualization?lan...

You will then be pleasantly surprised that the underlying Hypervisor.framework is used by Veertu in their Anka product. You can download and use it for free to virtualize macOS:

https://veertu.com/download-anka-develop/

Reply on news.ycombinator.com