The C++ stuff isn't that bad, in the end you can just pretend it's messages with fixed layout structures. But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful.
For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc).
After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based.
When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena.
In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-management schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within.
(Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).
In this case the answer might be just that they were too time constrained to design something better. But I was just curious if the RPMsg framework has too many issues that would make it unsuitable.
In general, this kind of message passing thing is easy to reinvent and everyone does it differently. You'll find tons of mailbox drivers in Linux already, all different.
1. Probably years ago: refactor the relevant components (display driver, etc.) to run as Mach components with Mach IPC, in the form of:
- One XCode project, with
- an “IPC client” build target
- an “IPC server” build target
- a “shared datatypes library” with well-defined IPC serialization semantics, static-linked into both the IPC client and server
- a single test suite that tests everything
3. Basically do nothing in the IPC client build target — it should now be oblivious to whether its Mach messages are going to an on-CPU kernel daemon or to an RTOS-coprocessor-hosted daemon, as the Mach abstraction layer is taking care of the message routing. (Kinda like the location-obliviousness you get in Erlang when sending messages to PIDs.)
This seems likely to me because there was (and still is!) both an Intel and an Apple Silicon macOS release; and they would want to share as much driver code between those releases as possible. So I think it’s very likely that they’ve written drivers structured to be “split across” Apple Silicon, while running “locally” on their Intel devices, in such a way that the differences between these approaches is effectively transparent to the kernel.
To achieve #3 — and especially to achieve it while having only one shared test suite — the firmware would have to be speaking the same wire protocol to the IPC client that the in-kernel IPC daemon speaks.
And, given that the in-kernel IPC daemons were designed to presume a reliable C++ interface, access to shared-memory copies of low-level Frameworks like IOKit, etc.; the firmware would need to provide this same environment to get the pre-existing code to run.
They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.
> They do have a non-DCP driver they're still shipping, that runs all the same code entirely within the macOS kext. I'm not sure exactly what it's for.
Presumably, they didn’t always have the coprocessor as part of the design, especially at the prototype phase. Imagine what type of workstations were built to prototype the Apple Silicon release of macOS — they probably didn’t have any Apple-specific coprocessors at first (and then they were likely gradually added for testing by sticking them on a PCIe card or something.)
As for the non-DCP driver, the odd thing is that it's specifically for the same chip generation (H13, i.e. A14 and M1), not an older one or any weird prototype thing.
No weird "workstations" were built to prototype the Apple Silicon release of macOS; Apple Silicon has existed for many years now, coprocessors like these included, in iPhones and iPads. The M1 is an iPad chip, the A14X, that they beefed up just enough to be able to stick it in laptops, and then marketing rebranded it as M1. DCP specifically is relatively recent, though, I think it only showed up in a relatively recent Ax silicon generation.
This was more my point.
I find it unlikely that Apple were doing most of the testing of macOS-on-ARM (which probably occurred for years prior to the M1 announcement, and prior to the A14X being created) directly using the iOS device architecture. Doing that wouldn’t have allowed them to develop AS support for regular PCIe devices attached through Thunderbolt, for example, since there’s nothing like a PCIe lane in that architecture.
Instead, to test things like that, I suspect Apple would have needed some kind of testbench that allowed them to run macOS on an ARM CPU, while attaching arbitrary existing peripherals into said ARM CPU’s address space, without having to fab a new one-off board for it every time they tweaked the proposed architecture.
I would guess that their approach, then, would have been very similar to the approach used in prototype bring-up in the game-console industry:
- Use a regular Intel machine as a host / hypervisor (in Apple’s case, probably one of the internal mATX-testbench-layout Intel Mac Pros)
- Put whatever recent-generation ARM CPU they made, onto a PCIe card
- Build a bespoke hypervisor to drive that CPU, which presents to the CPU a virtualized chipset matching the current proposed architecture (e.g. a DCP or not, a Thunderbolt controller, etc.)
- Have the hypervisor configure the host’s IOMMU to present both virtual peripherals (for bring-up), and arbitrary host peripherals, to the CPU
It’s not like Apple are unfamiliar with “SoCs used to accelerate a host-run emulator”; decades ago, they put an Apple II on an accelerator card and drove it through a host hypervisor just like this :)
Of course they would be doing it like that, since the macOS kernel was already ported to ARM for iOS. They even handed out developer kits that literally used an iPad chip.
https://en.wikipedia.org/wiki/Developer_Transition_Kit_(2020...
macOS on ARM is very clearly a descendant of the way iOS works. It's just macOS userspace on top of an ARM XNU kernel, which was already a thing. The way the boot process works, etc. is clearly iOS plus their new fancy Boot Policy stuff for supporting multiple OSes and custom kernels.
> Doing that wouldn’t have allowed them to develop AS support for regular PCIe devices attached through Thunderbolt, for example, since there’s nothing like a PCIe lane in that architecture.
iPhones and iPads use PCIe! How do you think WiFi/storage are attached? PCIe isn't only a desktop thing, it's in most phones these days, the Nintendo Switch, Raspberry Pi 4, etc. Most modern ARM embedded systems use PCIe for something. You'd be hard pressed to find a high-end ARM SoC without PCIe lanes.
They didn't have Thunderbolt, but since Apple rolled their own bespoke controller too, there is absolutely nothing to be gained by bringing that up on Intel first. Instead they probably did the same thing everyone does: bring it up on FPGAs. Possibly as PCIe add-ons for existing iPad chips, possibly hosting a whole (reduced) SoC; both approaches would probably be mixed at different design stages.
I'm sure they also had at least one unreleased silicon spin/design before the M1, during this project. You never quite get these things right the first time. No idea if that would've made it close to tape-out or not, but I'm sure there was something quite a bit different from the M1 in design at some point.
> Use a regular Intel machine as a host
> Put whatever recent-generation ARM CPU they made, onto a PCIe card
You can buy one of those, it's called a T2 Mac. Indeed some Apple Silicon technologies came from there (e.g. how SEP external storage is managed), but not in the way you describe; most of the T2<->Intel interface was thrown away for Apple Silicon, and now the OS sees things the same way as BridgeOS did natively on the T2 on those Macs.
> there is absolutely nothing to be gained by bringing that up on Intel first
If you're an OSdev fan, then "self-hosting" (i.e. being able to do your kernel development for your new system on the system itself, making your testbench your new workstation) is usually considered a valuable property, to be achieved as soon as is feasible.
Of course, the M1 effort probably had a lot of involvement from the iOS kernel teams; and the iOS kernel folks are a lot less like OS-enthusiast hackers and a lot more like game-console platform-toolchain developers, thoroughly used to a development paradigm for putting out new iOS devices that focuses on non-self-hosted bring-up via tethered serial debugging / whatever the Lightning-protocol equivalent of ADB is. So they probably don't really care about self-hosting. (Are they even there now? Can you iterate on XNU kernel drivers effectively on an M1 Mac?)
You can build a completely open source XNU kernel core for M1 and boot it, and iterate kernel drivers, yes. All of that is fully supported, they have the KDKs, everything. It's been self hosted since release (well, things were a bit rocky the first month or two in the public version as far as custom kernels).
Correction: they sometimes have the KDKs ;)
Not sure how different it was, but there was a t8028 at some point that never got released.