Asahi Linux for Apple M1 progress report, August 2021
asahilinux.org
asahilinux.org
My conclusion: You can get into just about anything, and for the most part the difficulty level is fairly uniform. But there's simply a vast sea of domain-specific spec knowledge out there. It doesn't mean that it's too hard for you or you can't learn it. Anything that is done at scale will fundamentally be approachable by most developers. Just be prepared you'll need to put in the time to acquire a familiarity with the local standards/specs and ways of doing things. Knowledge turns seemingly hard things into easy things, and if it's been done before chances are it's documented somewhere.
The truly hard stuff is innovating/doing things that haven't been done before. Novel development happens rarely.
And with the bugs. Especially with bootloaders because you're in a preboot environment.
What I learned starting these daunting task (especially the ps3 which was closed specs), is that it s still done by humans and following familiar patterns. Most of the time you bang your head against unclear doc or go through successive layers of abstraction (to do a ps3 rendering engine, better know your basic openGL on PC, which is only possible if you know matrix and vector geometry) but EVERYTHING is possible to reach at a workable level (expert level I feel comes with money incentive, team mates and repetition). I spent 2 years on japanese, 3 hours a day, and could integrate the meaning of Haikus at the end.
I think the only true talent you must have is insane stubborness. To go through, a doc at a time. Usually after the first insane challenge (for me: learning english at a near native level, reading literature or discussing politics) of your life, you understand it s all pretty much time investment.
Here's [1] the first few commits to OpenCore. Much more approachable and inspiring.
[1] https://github.com/acidanthera/OpenCorePkg/commits/master?af...
This in turn implies apart from technical skill, charting a good course from layer to layer can make a big difference, vs meandering without quite knowing where you are going.
How interesting: by moving some of the driver to firmware, Apple has effectively made it easier for OSs like Linux to have good support for the hardware, because the OS<->hardware protocol operates at a higher level of abstraction and may work across multiple chips.
The trade-off is that the protocol is not stable and so Linux has to decide which versions it will support, and make sure the OS driver always matches the version of the hardware firmware:
> As a further twist, the DCP interface is not stable and changes every macOS version! [...] Don’t fret: this doesn’t mean you won’t be able to upgrade macOS. This firmware is per-OS, not per-system, and thus Linux can use a different firmware bundle from any sibling macOS installations.
NVMe exposed as an MMIO device directly, not as a PCIe one.
If I am remembering right, this was the original idea behind UEFI drivers too — you could in theory write drivers that the firmware would load and they would present a simpler/class-compatible interface to whatever operating system was loaded after that. I think Apple did pretty much this on Intel Macs for a number of things.
Not really; pretty much every UEFI driver shuts itself down when you call ExitBootServices, which is necessary before you can do virtual addressing (as required by every OS)
(My memory is fuzzy but I think the one exception is UNDI/the network driver, but you never want to use UNDI to drive your network card except out of desparation)
Wow, that's a tall stretch of a conclusion, bordering on ignorance, when the reality is that the M1 only supports macOS! Apple fans shouldn't be blind to the fact that developers are trying to reverse engineer the M1 to run Linux on it. (And they have a long way to go before they succeed). Apple has not even published any literature that can help system developers create and run alternative OSes on it. And this is by design - the M1 (and all Arm processor based macs) are designed to be a a closed box like their other iDevices.
(Honestly, except for an academic interest, I believe open source developers are wasting their time trying to support such a useless "closed" piece of hardware, and instead should be boycotting it - or we'll end up losing control over our desktop computers too as other manufacturers move towards this model too).
> On most mobile SoCs, the display controller is just a piece of hardware with simple registers. While this is true on the M1 as well, Apple decided to give it a twist. They added a coprocessor to the display engine (called DCP), which runs its own firmware (initialized by the system bootloader), and moved most of the display driver into the coprocessor. But instead of doing it at a natural driver boundary… they took half of their macOS C++ driver, moved it into the DCP, and created a remote procedure call interface so that each half can call methods on C++ objects on the other CPU! Talk about overcomplicating things… Reverse engineering this is a huge challenge ....
By reverse engineering closed systems, you are adding value to it but not changing its nature - you are not really making it "open" (the next iteration will just "close" it again). Meanwhile, Apple just uses these reverse engineering effort cleverly for PR for the M1 through social media marketing. In effect, you are just helping a trillion dollar company design better closed systems while helping them make more money (by attracting naive Linux users to spend money on their hardware).
There are open systems that are struggling because they don't have talented developers working on their projects. Wish these developers would instead work on these projects - Sailfish OS, PineOS, Librem - Purism etc. For example, there are a lot of Intel / Arm tablets that either run only Windows or Android. It would be nice to run one of these Linux based mobile OS distros on it, and create a genuine alternate to the Android / ios duopoly on Tablets, right?
M1 is a beast and needs no further PR to attract buyers. You can ignore its advantages and yell out "our system sucks but at least it's open!", or, you can work on bringing the advantages of the product of a trillion dollar company's RND to the open world.
And regardless of what you think, the M1 does need constant marketing as its limitations becomes obvious. Apple is betting their future on such closed systems and desperately needs it to be a success. That is why Apple marketing is desperate to spread PR fluffs about it to its Apple fans that ignorantly parrot them on the social media on threads like these:
- Apple has effectively made it easier for OSs like Linux to have good support for the hardware
- Ok, but they’ve done the opposite- helping linux projects get linux on M1 macs.. ( https://news.ycombinator.com/item?id=27943196 )
... and similar statements like these that try to convey that Apple is actually helping developers port Linux to their platform.
Obviously, any posts (like mine) that point out the obvious lie in these statements are downvoted / vote-brigaded because it is in Apple's interest to do so.
No, quit the Dunning-Kruger, you're being downvoted because most people here disagree with you. Myself included: I don't particularly like Apple, I'm not paid by Apple, I'm a life-long proponent of FOSS and open systems, and I downvoted you because your posts are uninformed and either you haven't thought this through, or you don't have enough context to know what you're talking about.
Insinuations of shilling/astroturfing are against the guidelines, FYI.
That said, I have no hesitation in pointing out that that calling my posts "uninformed", without actually providing any actual context, is just a lazy and aggressive way of avoiding a discussion.
Or you can engage with me and point out where my reasoning is flawed. Here's my stated position and reasoning for what I have said so far, and you can chime in with you thoughts:
First, I started by pointing out that the beliefs of some Apple fans that Apple has made it easy and / or is actually helping developers to port Linux to the M1 is pure ignorance - Apple is doing no such things, but it is in their interest to let the public believe that. What is uninformed about this, in your opinion?
Second, I voiced the opinion that "open source developers are wasting their time trying to support such a useless "closed" piece of hardware, and instead should be boycotting it". The reasoning behind it is obvious - supporting a closed black box system, like the M1, and the proliferation of such devices will mean we will lose more of our computing freedom / right-to-repair on the computers we own.
Is it "uninformed" to point out that the M1 is a closed system that limits us, compared to Intel / AMD alternatives available? Or that the efforts of the developers will be a waste when Apple can, at any time, thwart or derail such reverse engineering attempts on the M1 by updating its firmware?
Third, porting Linux to the M1 will make it attractive, and help its sale. But it is a short-sighted and regressive move that will not help the open-source movement, and will backfire on us as it will help to make such closed box systems popular. The better approach for the open source community is to actually publicly boycott and disparage the M1, so that Apple is forced to open up the M1. Porting Linux to it then would not only be easier, but also makes more sense. What is "uninformed" about this way of thinking?
Fourth, while I do appreciate the strides made by Apple in developing their own custom ARM processors, I do genuinely believe that the technical gains (more computing power) are competitively temporary - both AMD and Intel systems have at some point always shown such leads. And it is not far fetched to say that this technical lead is even negated due to it's closed box nature. Do you think I am wrong in concluding that AMD / Intel will not catch up, or even overtake, Apple ARM processors?
Nothing, that's not the part I (personally) had a problem with.
> Is it "uninformed" to point out that the M1 is a closed system that limits us, compared to Intel / AMD alternatives available? Or that the efforts of the developers will be a waste when Apple can, at any time, thwart or derail such reverse engineering attempts on the M1 by updating its firmware?
You just described reverse engineering. That's what it always looks like. It's not in Apple's best interest to thwart those efforts, and if they do, it becomes a game of whack-a-mole.
What I believe is uninformed about your posts, or at least your arguments, is the lack of understanding of both how we got to this point, and how progress can happen.
We are not at a point in the world where open hardware is winning. Not just CPUs, just open hardware in general. This won't change by ignoring the progress in closed systems. Reverse engineering for compatibility makes the progress more public and more accessible to the open systems. There are fundamental issues that need to be addressed before we can start on relying on open-by-default progress in hardware.
To answer your question about what is "uninformed" about it: Maybe a better world would be pointlessly idealistic. You won't address the issues above with a boycott, and it doesn't matter because a boycott won't happen anyway (if you can't convince a die-hard supporter of open source like me, you won't convince the general public). So it's all a bit of a waste of time; I like progress to not just be theoretical. Reverse engineering for compatibility is a practical way of progressing.
> Do you think I am wrong in concluding that AMD / Intel will not catch up, or even overtake, Apple ARM processors?
I think it's a good bet, but I don't think any of us can say so with certainty and definitely not "when" it will happen. Intel has been spiraling lately, who knows if they will catch up at all?
You can't just get a small batch of royalty-free ARM cores and make your own phone. Did you know there's still work going on to reverse engineer the Raspberry Pi systems because Broadcom doesn't release full docs?
That argument would totally hold for iOS devices: you might find a bug and create a jailbreak on a particular iOS version, but that "hole" closes by the time the next major version comes out. Apple really doesn't want to allow other operating systems (or even root access).
But reverse-engineering macOS hardware is very different. A solid Linux port like Asahi fundamentally changes the nature of Apple M1 computers: they can be great Linux machines (and given m1n1's versatility, maybe even Windows machines!), not just macOS machines. Sure, Apple might "close" things again in the future, but that future is a long way off. They have given no indication of hostility to Asahi, so it's most likely that Apple computers will run Linux without any trouble until they release their next brand-new architecture. If history is any indication, that transition is likely more than 15 years in the future (68k was 1984-1994, PowerPC was 1994-2006, x86 was 2006-2021).
> There are open systems that are struggling because they don't have talented developers working on their projects. Wish these developers would instead work on these projects - Sailfish OS, PineOS, Librem - Purism etc. For example, there are a lot of Intel / Arm tablets that either run only Windows or Android. It would be nice to run one of these Linux based mobile OS distros on it, and create a genuine alternate to the Android / ios duopoly on Tablets, right?
I agree, that would be great in many ways. But honestly, all other things being equal, given a choice between running Linux on a Qualcomm ARM64, a Apple M1, or an Intel/AMD x64 chip, which one is the most desirable? Which will likely be the most desirable after the M1x or M2 comes out?
At the most fundamental level, it's desirability that drives demand, which in turn decides which systems will be the most well-resourced.
Your ideal taken to its conclusion is that we wait for RISC-V or OpenMIPS or something to catch up to ARM before we write or use any software. That's simply untenable. What software would we even have to port to such things if we couldn't develop open software on closed hardware? What use would RISC-V be outside of embedded projects without QNX, Linux, BSD, RiscOS, or something else developed originally on closed hardware to run on it?
Think about where we might be if that never happened.. no clones? No open hardware standards? I mean, I guess if Compaq didn’t do it, somebody else would have (maybe?), but yes, thank god for the brilliant engineers of the world who have the focus, diligence, and skill, to be able to do these sorts of things.
Today's computing world is no longer like the past. Open source is an accepted reality in both the corporate world and among the general population.
You are right that in the past, reverse engineering helped clone hardware and make it more open. It was necessary then. Perhaps I am ignorant here, and I would like insights on this, but I don't understand the practicality of trying to reverse-engineer Apple's custom GPU when it is not even freely available or can even be used in any custom hardware configuration. Especially when alternative GPUs with open source drivers already exist (and need work). And Apple can anytime thwart such efforts very easily with a firmware update.
My main concern is that the BigTech are again closing ranks together to fight against the open movement - today, we are at the cusp of losing our computing freedom again with the aggressive push towards cloud computing, and locked down systems like the Apple M1. Apple is in the forefront here as it has build its business with closed-box systems and resisting against right-to-repair legislation. Fighting and avoiding open systems is in their DNA.
That is why I believe porting Linux to the M1 is not only unnecessary but also a waste of resource and talent. Supporting existing and new open systems (if necessary by reverse engineering it) is more practical in the long run.
Some other clone companies directly copied the IBM BIOS and got sued, so it was Phoenix that provided the first 3rd party PC-compatible provably clean-room BIOS implementation.
Yes, the hardware is peculiar in many ways and Apple didn't bother to document it (it would have costed them some resources for no clear benefit to their business model) but they also didn't bother to completely lock it down, like they did with the iPhone and iPad. This is what allows other OSes to be ported to it if they are able/willing to reverse engineer and develop support for it.
This reverse engineering work was done so far by just a handful of people who would also like to have Linux run on it, partly sponsored by a few hundreds that share this desire, or just doing it voluntarily as a challenge/interesting project.
The m1n1 bootloader/hypervisor built by the team allows running macOS and logging all the hardware access.
Once the reverse engineering is done for a given component, all the other OSes can implement support for it.
The fact that a lot of the functionality is implemented in firmware blobs makes it easier to support at the OS level, essentially each OS driver only needs to give high level instructions to the firmware, without having to implement all the minutiae done in the firmware.
> No, Apple still controls the boot process and, for example, the firmware that runs on the Secure Enclave Processor. However, no modern device is “fully open” - no usable computer exists today with completely open software and hardware (as much as some companies want to market themselves as such).
I think they're aiming at purism here, but might have forgotten about the MNT Reform, even though it is currently specced at the lower end of "usable".
In the end, you're always drawing arbitrary lines in the sand. If you really want to go all the way to actually reach a tangible goal, I'm only aware of one project that actually offers trust guarantees backed by hard engineering arguments: Precursor
https://www.crowdsupply.com/sutajio-kosagi/precursor
(TL;DR on the trick is that, by using an FPGA, you make it nearly impossible to backdoor, because it would take a huge amount of compute power to engineer a backdoor into the FPGA silicon that can analyze arbitrary randomized FPGA designs to backdoor them).
For more practical computing, I find M1s have a very solid secureboot design. We can retain that security even after putting m1n1/Asahi on there; doing so requires physical presence assertion and admin credentials and locks you into a single m1n1 version without repeating the process, so we can use that to root our own chain of trust. Similarly we can continue to use Apple's SEP (in the same way one would use a TPM, HSM, or a YubiKey for that matter; sure, it's a proprietary black box, but it also can't hurt you except where you let it) just like macOS does, for things like encryption and storing SSH secrets. And all the coprocessors have IOMMUs, and the whole thing is very logically put together (unlike the giant mess that are Intel CPUs; e.g. we know they have a JTAG backdoor in production chips after authenticating with keys only they have, never mind things like ME) and designed such that the main OS does not have to trust all the auxiliary firmwares.
I'd love to have a more open system competing in the same space, but I have no problem trusting Apple's machines much more than I do the x86 PCs I use these days anyway, and it's hard to find things that compete in the same performance/power space that are more open than either of those, unfortunately.
Safety is not binary.
If you have an open system that does not do robust secureboot (i.e. secureboot that can withstand a physical access attack), then it doesn't matter that it's open; a physical attacker can still break into your machine and backdoor it. If it does do robust secureboot, it is pretty much guaranteed to rely on a proprietary ROM, which will likely have backdoors or exploitable bugs (see e.g. Nvidia Tegra machines; yes you can burn in your own secureboot keys on systems that come unlocked like the Jetsons, but their ROM is a mess and all their security is now moot as you can exploit it). Apple have a lot of experience with this, having long been one of the prime targets, and have put a lot of effort into hardening their boot chain, so I trust them to do a better job than pretty much any other silicon/firmware vendor (and I say this as someone who has looked at the security of many vendors).
People like conflate openness with trustability, but they do not go together. Most open systems I see being developed explicitly choose not to enable or support this kind of security, which makes them less trustable systems than an Apple machine: I don't care that all the boot process is open and I can inspect it, if anyone else who steals my laptop or messes with it while I'm away can steal my data. You need to rely on maintaining physical control. With an Apple machine, you don't: only Apple can break its security, nobody else, so your data is safer as a result. The machine is designed so I do not have to trust Apple's components to be reasonably confident that my machine is safe once it boots into my own OS, thanks to the IOMMUs and general design of the ARM architecture (e.g. a transparent malicious hypervisor isn't really possible). And none of this has anything to do with remote access backdoors (there is nothing like ME on these machines), so the entire story is about what you can do with physical access.
That is not to say there wouldn't be advantages to fully open systems; for example, it would allow for a simpler and more user-friendly installation process than what we have to do with Apple's framework, and it would make the DCP thing much less of a mess. There are definitely practical, real advantages to having a more open system. It's just that all too often I see the trust/security argument being thrown around and... that just isn't true in most cases, when you look at the details.
I'm no Stallmanesque ideologue; I'm attracted to the M1 because of its low power usage and I've used Macs for years and for my next mobile system I may end up picking up an Air. But as a desktop workstation POWER9 systems like this really are more open and credible alternatives, and I felt it was well worth the price.
The C++ stuff isn't that bad, in the end you can just pretend it's messages with fixed layout structures. But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful.
In this case the answer might be just that they were too time constrained to design something better. But I was just curious if the RPMsg framework has too many issues that would make it unsuitable.
In general, this kind of message passing thing is easy to reinvent and everyone does it differently. You'll find tons of mailbox drivers in Linux already, all different.
1. Probably years ago: refactor the relevant components (display driver, etc.) to run as Mach components with Mach IPC, in the form of:
- One XCode project, with
- an “IPC client” build target
- an “IPC server” build target
- a “shared datatypes library” with well-defined IPC serialization semantics, static-linked into both the IPC client and server
- a single test suite that tests everything
3. Basically do nothing in the IPC client build target — it should now be oblivious to whether its Mach messages are going to an on-CPU kernel daemon or to an RTOS-coprocessor-hosted daemon, as the Mach abstraction layer is taking care of the message routing. (Kinda like the location-obliviousness you get in Erlang when sending messages to PIDs.)
This seems likely to me because there was (and still is!) both an Intel and an Apple Silicon macOS release; and they would want to share as much driver code between those releases as possible. So I think it’s very likely that they’ve written drivers structured to be “split across” Apple Silicon, while running “locally” on their Intel devices, in such a way that the differences between these approaches is effectively transparent to the kernel.
To achieve #3 — and especially to achieve it while having only one shared test suite — the firmware would have to be speaking the same wire protocol to the IPC client that the in-kernel IPC daemon speaks.
And, given that the in-kernel IPC daemons were designed to presume a reliable C++ interface, access to shared-memory copies of low-level Frameworks like IOKit, etc.; the firmware would need to provide this same environment to get the pre-existing code to run.
They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.
> They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.
Presumably, they didn’t always have the coprocessor as part of the design, especially at the prototype phase. Imagine what type of workstations were built to prototype the Apple Silicon release of macOS — they probably didn’t have any Apple-specific coprocessors at first (and then they were likely gradually added for testing by sticking them on a PCIe card or something.)
As for the non-DCP driver, the odd thing is that it's specifically for the same chip generation (H13, i.e. A14 and M1), not an older one or any weird prototype thing.
No weird "workstations" were built to prototype the Apple Silicon release of macOS; Apple Silicon has existed for many years now, coprocessors like these included, in iPhones and iPads. The M1 is an iPad chip, the A14X, that they beefed up just enough to be able to stick it in laptops, and then marketing rebranded it as M1. DCP specifically is relatively recent, though, I think it only showed up in a relatively recent Ax silicon generation.
This was more my point.
I find it unlikely that Apple were doing most of the testing of macOS-on-ARM (which probably occurred for years prior to the M1 announcement, and prior to the A14X being created) directly using the iOS device architecture. Doing that wouldn’t have allowed them to develop AS support for regular PCIe devices attached through Thunderbolt, for example, since there’s nothing like a PCIe lane in that architecture.
Instead, to test things like that, I suspect Apple would have needed some kind of testbench that allowed them to run macOS on an ARM CPU, while attaching arbitrary existing peripherals into said ARM CPU’s address space, without having to fab a new one-off board for it every time they tweaked the proposed architecture.
I would guess that their approach, then, would have been very similar to the approach used in prototype bring-up in the game-console industry:
- Use a regular Intel machine as a host / hypervisor (in Apple’s case, probably one of the internal mATX-testbench-layout Intel Mac Pros)
- Put whatever recent-generation ARM CPU they made, onto a PCIe card
- Build a bespoke hypervisor to drive that CPU, which presents to the CPU a virtualized chipset matching the current proposed architecture (e.g. a DCP or not, a Thunderbolt controller, etc.)
- Have the hypervisor configure the host’s IOMMU to present both virtual peripherals (for bring-up), and arbitrary host peripherals, to the CPU
It’s not like Apple are unfamiliar with “SoCs used to accelerate a host-run emulator”; decades ago, they put an Apple II on an accelerator card and drove it through a host hypervisor just like this :)
Of course they would be doing it like that, since the macOS kernel was already ported to ARM for iOS. They even handed out developer kits that literally used an iPad chip.
https://en.wikipedia.org/wiki/Developer_Transition_Kit_(2020...
macOS on ARM is very clearly a descendant of the way iOS works. It's just macOS userspace on top of an ARM XNU kernel, which was already a thing. The way the boot process works, etc. is clearly iOS plus their new fancy Boot Policy stuff for supporting multiple OSes and custom kernels.
> Doing that wouldn’t have allowed them to develop AS support for regular PCIe devices attached through Thunderbolt, for example, since there’s nothing like a PCIe lane in that architecture.
iPhones and iPads use PCIe! How do you think WiFi/storage are attached? PCIe isn't only a desktop thing, it's in most phones these days, the Nintendo Switch, Raspberry Pi 4, etc. Most modern ARM embedded systems use PCIe for something. You'd be hard pressed to find a high-end ARM SoC without PCIe lanes.
They didn't have Thunderbolt, but since Apple rolled their own bespoke controller too, there is absolutely nothing to be gained by bringing that up on Intel first. Instead they probably did the same thing everyone does: bring it up on FPGAs. Possibly as PCIe add-ons for existing iPad chips, possibly hosting a whole (reduced) SoC; both approaches would probably be mixed at different design stages.
I'm sure they also had at least one unreleased silicon spin/design before the M1, during this project. You never quite get these things right the first time. No idea if that would've made it close to tape-out or not, but I'm sure there was something quite a bit different from the M1 in design at some point.
> Use a regular Intel machine as a host
> Put whatever recent-generation ARM CPU they made, onto a PCIe card
You can buy one of those, it's called a T2 Mac. Indeed some Apple Silicon technologies came from there (e.g. how SEP external storage is managed), but not in the way you describe; most of the T2<->Intel interface was thrown away for Apple Silicon, and now the OS sees things the same way as BridgeOS did natively on the T2 on those Macs.
> there is absolutely nothing to be gained by bringing that up on Intel first
If you're an OSdev fan, then "self-hosting" (i.e. being able to do your kernel development for your new system on the system itself, making your testbench your new workstation) is usually considered a valuable property, to be achieved as soon as is feasible.
Of course, the M1 effort probably had a lot of involvement from the iOS kernel teams; and the iOS kernel folks are a lot less like OS-enthusiast hackers and a lot more like game-console platform-toolchain developers, thoroughly used to a development paradigm for putting out new iOS devices that focuses on non-self-hosted bring-up via tethered serial debugging / whatever the Lightning-protocol equivalent of ADB is. So they probably don't really care about self-hosting. (Are they even there now? Can you iterate on XNU kernel drivers effectively on an M1 Mac?)
You can build a completely open source XNU kernel core for M1 and boot it, and iterate kernel drivers, yes. All of that is fully supported, they have the KDKs, everything. It's been self hosted since release (well, things were a bit rocky the first month or two in the public version as far as custom kernels).
Correction: they sometimes have the KDKs ;)
Not sure how different it was, but there was a t8028 at some point that never got released.
For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc).
After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based.
When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena.
In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-management schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within.
(Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).
https://www.patreon.com/marcan/ https://github.com/sponsors/marcan
I’m considering buying the frame.work laptop, daily driving pop!_os and then virtualizing OSX on it for the few OSX programs I use (this supports framework and not apple).
It looks like it may be fairly easy and possible with a 10-20% performance hit?
https://github.com/foxlet/macOS-Simple-KVM
From what I can tell you can pass through a single GPU with a bit of work, even a iGPU? Is that correct?
Now, this too doesn't quite work out of the box because VMWare’s products have a completely artificial check and wherein they'll refuse to boot macOS on non-Apple hardware by default. To get around this, VMWare needs to be patched, but macOS runs entirely unmodified! And because it's ultimately the same VMWare codebase (quite literally the same binary in the case of ESXI), it runs just as well as it would on a Mac.
Native Hackintosh setups, when done properly, also work much better than many non-Hackintosh users assume. The initial setup process can be quite time consuming and fiddly, but if and when you get everything working, you can not touch it for years and it will be be just like an actual Mac. Even across updates! You absolutely don't need to "enjoy diagnosing errors rather than working," except at the start.
Now, the job of a boot-loader is to set up a friendly environment for the OS, so I'm not sure where the line is. But OpenCore goes pretty far!
* macOS lacks drivers for GPUs commonly emulated by VM software, and as such runs absolutely horribly without graphics acceleration because it assumes the presence of a competent GPU at all times - software rasterization seems to be intended to be used only as an absolute last-resort fallback. As such, passthrough of a supported GPU (generally AMD, though Intel iGPUs might work) is basically a requirement.
* It's technically against the macOS EULA. Apple hasn't gone after anybody for it to date, but it doesn't mean they won't. This is particularly relevant for devs building for Apple platforms.
No GPU? WindowServer crashes and you'll never see the GUI.
Though there isn't much of a difference for Linux; you can't actually share the Recovery image (since it's per-OS, even within a single Recovery volume), and since you need to repartition to create standard Linux partitions anyway (since linux-apfs isn't quite at the point where you'd want to use it as your root filesystem as far as I know...) it doesn't make much of a difference if you create a 2.5G stub container too, and makes it much easier to blow away the entire Linux install without affecting macOS. Perhaps in the future, when linux-apfs is solid enough, it would be interesting to support this in the installer in order to have a fully space-sharing dual-boot macOS/linux system, without partitioning.
Also, U-Boot would have to grow APFS support in order to have a truly all-APFS system :) (we plan to use U-Boot as a layer for UEFI services and such, so you don't have to burn kernels into the m1n1 image, which would suck for upgrades).
If it's possible to do this in APFS (not sure), the installer could do something similar for APFS to what WUBI does for NTFS: fallocate(2) a contiguous-disk-block single-extent file inside the APFS volume, write the rootfs into it, and then configure the bootloader entry's kernel params with the APFS container+volume and the path to the file within the container.
In this setup, linux-apfs would then only need to work well enough to expose that file's single extent's raw-block-device offset during early boot; then the initramfs script could unmount the linux-apfs volume and mount the ext4 rootfs directly from the block device at that offset.
Yes, it's almost the same as resizing the disk smaller — but it's at least a less-"permanent" change, in the sense that deleting the rootfs file from macOS would give the space back, without needing to know to re-grow your macOS APFS container.
I mean, "moving it elsewhere" is why you'd call into linux-apfs on every boot — to find out where the rootfs backing file's extent is living today. It might move around between boots, but it won't move while macOS isn't even running.
And this approach presumes that there exists an explicit file attribute for keeping a file single-extent / contiguous-on-disk (as far as the host-NVMe interface is concerned, at least.) Such an attribute does exist on NTFS, which is why WUBI continued to work there. I've never heard about such an attribute existing on APFS, which is why I said I'm not sure whether APFS "supports" this.
Usually there is such an attribute in most-any filesystem, though — it exists to be used with swap files, so that kernels can directly mmap(2) the disk-block range underlying the swap file rather than having every read/write go through the filesystem layer.
I know that macOS uses an APFS volume for swap (the "VM" volume), which is kind of interesting. Maybe in APFS, there isn't a per-file attribute for contiguous allocation, but rather a per-volume one?
In that case, it might be possible for the Linux rootfs to be an APFS volume, in the same way that the macOS VM volume is an APFS volume.
Ah, yeah, I've never heard of anything like that. I didn't realize NTFS had this. I have no idea how APFS handles the VM volume, though.
Apple engineers are also probably reading all this and are very impressed.
I generally hit the arch wikis with specific models for the best information.
This github repos also does a good job laying out current support for the 2016 models. https://github.com/Dunedan/mbp-2016-linux
Frankly, I haven't tried on a newer model then that - I don't buy apple hardware anymore.
For $50 more, you can get: https://www.solid-run.com/arm-servers-networking-platforms/h... on which you just add your own DRAM sticks too.
And since then: 12nm -> 10nm -> 7nm -> 5nm
For "real workstation" class hardware (i.e. working PCIe/XHCI/AHCI on standard ACPI+UEFI systems with mostly FOSS firmware) nothing beats SolidRun boards (MACCHIATObin/HoneyComb LX2K). Yeah yeah the Cortex-A72 cores are really showing their age, oh well, on the other hand the SoCs are not embedded cursed crap :)
Is this a kernel replacement, as in I could run Manjaro Gnome, and just load the Asahi Kernel and it all Just Works?
Or will Asahi Linux need to be a full distro of its own, in order to be useful?
That said, all components will be upstreamed into their respective projects such that other distributions should eventually be able to add support for Apple's line of desktop computers with Apple Silicon.
[1] https://asahilinux.org/about/#is-this-a-linux-distribution
m1n1 is our bootloader, which once installed will present a standard (ish) U-Boot/UEFI boot environment. From there you will be able to boot any distro or USB installer, in principle, once the changes trickle upstream.
Our installer will install m1n1 and then give you the choice of also installing a distro image. We'll be providing our own based on Arch Linux ARM, but we're open to other distros providing images so we can add them to the list. We already have interest from Fedora, for example.
So we expect that people who want to run Linux (or in fact openbsd or anything else) on these Macs will first use our bootloader installer, then either pick one of the built-in images or use their own USB installation media from whatever distro.
So for GPU we would need 3 pieces? Alyssa's work on reverse engineering the protocol and writing a driver to submit the correct commands to the GPU. A working DCP driver to handle actually displaying rendered frames. And finally a driver to be able to submit the commands to the GPU. And work on the last of those has not started yet, right? It's hard to piece together how everything fits for me.
I can’t wait to use Linux as daily driver on my M1
Apple provides restore images for this purpose (downloadable without any authentication). This is how the installer works, so we don't have to redistribute any firmware ourselves. Redistribution is the usual thorny firmware problem; since we don't have to do that, we're in a pretty good position.
Whether Apple's EULA applies to macOS obtained in this way, without actually going through a normal boot for that instance and thus the click-through EULA, is legally unclear. That said, here's a quick analysis:
2A says you can only install and run a single copy of the software. That might seem to imply that you wouldn't be able to dual-boot macOS and Asahi Linux, but this clause is pretty dumb. By design, all of these machines with one instance of macOS have at least two copies of part of the software: the recovery image - is that a violation? What about the fact that the bootability process creates a copy of all system firmware? You end up with 4+ copies of some components. What about installing macOS multiple times? Multiple versions of macOS? Do APFS CoW clones count as copies? If so that's another two copies of the recovery image. If they don't, can we just CoW clone stuff from the currently installed macOS version within the same container instead of doing a from-scratch install? Once Apple's updater breaks the CoW link, does that mean they are causing you to violate the EULA at that point? These are all things Apple obviously intended for people to do with these machines, regardless of what the lawyers say, so I don't think we have to worry too much about it. Legal texts are often completely at odds with the technical realities involved, and in the end nobody cares.
2M explicitly allows you to replace the Open Source components of macOS with your own version. This is exactly what m1n1 is: it is a replacement for xnu, the open source macOS kernel. So this clause explicitly puts at least our original full-fat macOS install process in the clear. The clause also says their warranty does not cover damage caused by such modifications, but does not say your warranty is void by merely doing so (so they'd have to prove any damage was caused by it).
2N says no reverse engineering, but you don't care about that, we do. I'm pretty confident I'm not going to get a C&D from Apple's lawyers for working on Asahi Linux, and if I do, I know how to defend myself and what organizations can help with that. It would be terribly bad PR for them to do that.
4A says you can transfer your Apple software, but 4B says you can't do that if you've modified it under 2M. That might mean that, on paper, you can't give away or sell a Mac that has Asahi Linux installed, without first uninstalling the bootloader and having the next owner reinstall it. Again though, this is a silly technicality. What Apple means is they don't want you patching macOS and distributing it separately. They also say the macOS components are a bundle and cannot be distributed separately, but our installer doesn't distribute anything, it pulls specific components from Apple themselves, which they are distributing.
4C says that you may only use copies made available for restorative purposes for those purposes and may not be resold or transferred. That might put our usage of ipsw restore images in the installer out of compliance, though the "restorative purposes" thing is a bit unclear. Does it count as "restoring" Apple's boot components into a blank OS partition? Obviously we are not reselling or transferring anything, so that part is not an issue.
This is an EULA, and to what extent these clauses are legally enforceable is of course unclear. We aren't doing the big copyright no-no of redistribution, and Apple themselves provide the software for free download, and it is a fact that every Mac user is licensed to use it. So in practice it would be quite questionable to try to go after people for technicalities here.
The better question is whether Apple's lawyers will care, and I'm pretty sure the answer is no. The EULA is there to give them legal ammo against companies trying to do things like provide virtual macOS as a service or installing it on non-Apple machines; they don't really care what end-users do (they don't even care about hackintoshes as long as you aren't selling them, which is a blatant EULA violation). The general idea of running Linux with Apple's firmware is completely kosher per 2M (if done on a single macOS install, e.g. just replacing xnu with m1n1 without a separate OS instance), so there is no legal basis for Apple to block our project entirely. It would be extremely silly if they went after our users to block our specific install process just to annoy them or get them to replace macOS with Linux instead of dual-booting, and it would also put in jeopardy things that regular macOS developers do, like having multiple OS installs.
also m1 is pretty quick
Anyway, the ISA does not necessarily imply higher or lower energy consumption.
The hypervisor feature is indeed single core, but that's just because I haven't bothered implementing SMP (would need a bunch of extra locking, but it's doable, just not really necessary for us).
It's a very real fact that if attackers can't find your bugs, they are less likely to exploit them. Apple are well versed in writing secure code, but they slip up sometimes. It seems their calculus at this time is that the time gained by not allowing attackers easy visibility into their code is worth it, in case bugs are found. I don't fully agree with that choice, but I can see why they made it; it is not meaningless.
"Hello! I’m Hector Martin and like to go by the nickname “marcan”. I currently live in Tokyo, Japan as an IT/security consultant by day and a hacker by night. For some definition of day and night, anyway."
Hacker for hire basically. This project was started via crowdfunding enough monthly sponsors (overnight from his previous reputation and interest in this project) and is somewhat half day job and half night hacking out of personal interest as a result.
I admit that excludes more and more modern hardware, but apple is the most expensive and profitable.
Unless you want to risk losing your files or your entire machine. Maybe it would void your warranty if you do this. Who knows.
I've never had the installer do anything crazy, nor does it support any resize/deletion operations yet, so it's pretty unlikely that it'll wipe your data (it's much more likely to just fail to complete the installation properly).
More likely is that if you boot Linux, a driver bug will cause data corruption. We've had some reports of filesystem issues, though as I said the kernel tree situation is haphazard right now, so we'll see if they still happen once we put together a tree with properly vetted patches :)
(and many more other issues)
Storage-wise, you can't brick the machine by wiping NVMe. You can always recover via DFU mode, which you can also use from another Linux machine by using idevicerestore (no need for macOS). This is very much unlike your typical Android phone, which is indeed a flimsy mess that will brick itself at the slightest touch of the wrong partition in eMMC.
Hardware-wise, there's the usual "you can technically do ugly things with I/O pins", but in practice that's quite unlikely to cause damage. The lowest level, dangerous things are handled by SMC, not the main OS. We barely have to deal with any of that. And this is a thing on PCs too, just read your chipset's user manual and you'll find out the registers to flip random I/O pins.
Firmware-wise, I do know of one way of causing damage to these machines that cannot be recovered by the user: wiping NOR Flash. This isn't because it'll make DFU unusable, but rather because it contains things like system identity information and calibration data that only Apple can put back properly. But again, this isn't much different on PCs; you can brick a PC by wiping the NOR Flash from your OS too. In fact there have been BIOSes so broken that you can brick them by doing a `rm -rf /`, which descends into efivars and the UEFI implementation crashes when it finds no EFI variables.
In order to avoid the NOR Flash bricking risk, I do not intend to instantiate the NOR device in our device tree at all by default. I do not believe we need it for regular operation. Flash memories have specific unlock sequences for writes/erases, so there is practically zero risk that a crazy random bug (i.e. scribbling over random I/O memory) could trigger such an operation, and without the NOR driver bound, the code that actually knows how to cause damage will never run.
For what it's worth, I have quite some experience not bricking people's machines, even when dealing with unofficial exploits/hacks/jailbreaks. To my knowledge, not a single user out of the probably >10m people running The Homebrew Channel / BootMii on their Wii, which I helped develop, has bricked their system due to a problem traceable to our software. Nintendo has a much worse track record :-)
Ha. This reminds me of a Motorola Linux phone I bricked as a kid by trying to write a package manager in C shell (lol) which inevitably led to rm -rf / being executed which wiped a unique per-device security partition required for the boot process.
Why did Apple make the same choice of putting critical data onto an OS-writable flash chip? Chromebooks do it right – everything critical lives in the Security Chip's own on-chip flash, and the list of commands it would take from the host system is very limited by default (most things are only available to the debug cable).
As far as I know, fully wiping it is recoverable, but involves putting the Mac into device firmware update mode, and then recovering from another machine.
However, I can't claim to have tried it, so there may well be additional safeguards. I'm just not particularly interested in risking it for our users either way :)
Plug the bricked machine into another Mac using a TB3 cable in a particular slot (they tell you which) and follow the instructions. Worked for me.
Well that is another notable risk isn't it? If something goes wrong on the filesystem level (since you admitted it) then the worst case is that the OS and your important files become corrupted some how and it is back to recovery once again, if not then they will use Apple Configurator which requires another Mac.
Regardless, I would most certainly use another machine to test this, hence why I said 'use at your own risk', especially when the installation script is pre-alpha.
What is wrong with such a disclaimer?
Regardless I think most of the problem was with defaulting to fear mongering, particularly at the warranty, rather than gaining an understanding of what the risks are before making recommending action to avoid said risks.
Fine for those who have a spare M1 machine but the added risk comes when it is their only machine with important work and files on it.
Do you want to be liable when someone tries this script out and somehow ends up losing their important work because of those recently admitted 'filesystem issues' on the only machine they have? That is my point and why I said: "I would most certainly use another machine to test this"
I don't see lots of general users racing to install this right now, since they probably know that it is still incomplete with lots of bugs and is still unusable. Unless you want to install it right now on your work machine with important files perhaps? If not then wait.
Is that functionality in idevicerestore ready to use instead of using Apple Configurator?
There's some bug with the laptops, but it should be easy to fix once someone takes a good look at it. I might spend some time on that; I definitely want idevicerestore to be in good shape before we start encouraging more wide usage of Asahi. (https://github.com/libimobiledevice/idevicerestore/issues/41...)
Please don't do this! Monterey is an OS I will never install given that it will include the Apple-provided spyware. This shouldn't be the earliest supported version! I suggest instead finalizing on a current/later release of Mac OS 11 as updates for it will slow down once Monterey is released. A lot of people won't be updating to Monterey.
I’m quite sure this won’t be true, HN threads full of uninformed complaints notwithstanding.