I wonder if there exists a parallel dimension where linux is microkernel design and folks are pushing for monolothic citing the driver friendliness and performance
If I'm sure of one thing, it's that as soon as we decided to build everything as Microkernels, we'd have the same squeaky wheels touting the massive benefits of Monotlithic OS design. We're hilariously cyclical in our preferences.
"Think of all the runtime efficiencies of the shared memory, and how much easier it would be to develop on the shared codebase!"
Also, your phone's baseband processor almost certainly runs a microkernel, and likely so does your car.
> The Secure Enclave runs an Apple-customized version of the L4 microkernel. This microkernel is signed by Apple, verified as part of the iOS secure boot chain, and updated through a personalized software update process.
https://www.blackhat.com/docs/us-16/materials/us-16-Mandt-De...
https://source.android.com/devices/architecture/kernel/modul...
https://source.android.com/devices/architecture/hidl/
So everyone running Android Oreo or newer on their phones, not only has a microkernel on their basebase radio, they also have a Linux tamed into a microkernel on their main ARM SOC.
This means most (updated) Android 8 devices in the field are not Project Treble and are running old-style module-less kernels and don't have A/B system partitions.
HALs only seem to be there to abstract over the APIs of different hardware implementations and access to them. Not necessarily their drivers.
I would love to be wrong about this, so I would really appreciate evidence to the contrary here.
[1] https://source.android.com/devices/architecture/kernel/modul...
There are two HALs in Android.
The old HAL, previous to Treble, which was hardly used by OEMs.
And the new HAL, which is enforced by Treble.
On the new HAL, drivers are implemented as Android services using Android IPC via interfaces defined in HIDL, or by using the new shared memory APIs introduced in Oreo as well.
https://source.android.com/devices/architecture/hidl/service...
https://source.android.com/devices/architecture/hidl/binder-...
https://source.android.com/devices/architecture/hidl/memoryb...
https://source.android.com/devices/architecture/configstore/
As I understand it HALs and HIDL are used to provide a standardized way to implement new device features in a compatible way. So a vendor introduces a odor-sensor and can define an HIDL interface for the userspace to call the device driver provided in a kernel module.
I do not see a requirement to implement drivers in userspace, nor provisions for it, like interrupt handling, i2c/spi access or similar.
"Binderized HALs. HALs expressed in HAL interface definition language (HIDL). These HALs replace both conventional and legacy HALs used in earlier versions of Android. In a Binderized HAL, the Android framework and HALs communicate with each other using binder inter-process communication (IPC) calls. All devices launching with Android 8.0 or later must support binderized HALs only.
Passthrough HALs. A HIDL-wrapped conventional or legacy HAL. These HALs wrap existing HALs and can serve the HAL in binderized and same-process (passthrough) modes. Devices upgrading to Android 8.0 can use passthrough HALs. "
https://source.android.com/devices/architecture/hal-types
Here are the user space driver APIs for i2c/spi access on Android Things.
https://developer.android.com/reference/com/google/android/t...
It if was, there would be documentation on how to write a userspace device driver, but there isn't.
Interprocess RPC is only for Oreo and newer.
Going forward, I expect the vendors to modify their kernel drivers to export the treble API over binder IPC directly while in kernel space.
So I wouldn't be surprised if it ends up that interprocess RPC only ebds up getting used in systems before Oreo.
I'm guessing that the Treble modules expose the traditional /dev and /sys interfaces like before, and these new HALs talk to devices through those, right? Is that not still a runtime-linked monolith, with fairly-thin userspace services between the kernel and the framework?
The bulk of driver code logic (e.g. interfacing with a touchscreen controller) is probably still running in the kernel - show me if I'm wrong. I would think this would be how you define a microkernel. One exception may be the GPU -- those drivers are often very thin in the kernel because of GPL, with a fat userspace library that they can legally keep closed-source.
https://lwn.net/Articles/758745/
> Could allow for APK installable kernel drivers, allowing modules to be upgraded (even live)
https://twitter.com/MishaalRahman/status/1029064974805688320
It's quite crazy to count all the layers
cpu microcode
cpu isa
hypervisor
os kernel
userspace api
browser runtime
js vm
application
your mouse
...
webassembly vm
(insert variable height custom stack)
application
...
> Microkernels are intended to create a minimal set of low-level operations that would be used to build an operating system. While it's popular to slap a monolithic kernel on top of your microkernel, this is not how microkernel based OSes are supposed to be; a real microkernel OS should have lots of separate pieces that used the microkernel services to work with each other. Using a microkernel as not much more than an overgrown MMU and task switching abstraction layer for someone's monolithic kernel is a cheap hack driven by the needs of academic research, not how they are supposed to be.
> By contrast, hypervisors virtualize and emulate hardware at various levels of abstraction. This involves providing some of the same things that microkernels do (eg memory isolation, scheduling), but people interact with hypervisors in very different ways than they interact with microkernels. Even with 'cooperative' hypervisors, where the guest OSes must be guest-aware and make explicit calls to the hypervisor, the guests are far more independent, self-contained, and isolated than they would be in a microkernel. With typical 'hardware emulating' hypervisors this is even more extremely so because much or all of the interaction with the hypervisor is indirect, done by manipulating emulated hardware and then having the hypervisor reverse engineer your manipulations. As a consequence, something like guest to guest communication delays are likely to be several orders of magnitude worse than IPC between processes in a microkernel.
People never seem to want to admit that sometimes, technology just dies, or at least becomes obscure, and so will invent bizarre "connections" between what we used to be working on and what we have now.
https://microkerneldude.wordpress.com/2008/04/03/microkernel...
NOVA is an open-source micro-hypervisor available in GenodeOS that similarly combines hypervisor functionality with a microkernel-like design:
The Xen hypervisor also got many of its design elements from the Nemesis OS, which had a lightweight kernel. There's plenty of overlap possible. A number of research and production systems are also both hypervisors and microkernel-like, too. So, it's more an existing concept than a bizarre, hypothetical connection. For OKL4, also deployed in over a billion phones they claim.
The hardware-like interface to the hypervisor is both inefficient and not very abstract. For instance, a guest kernel attempts to perform a series of manipulations of the page table, each of which traps to the hypervisor, or else it performs an equivalent upcall to manipulate the page tables.
The hardware emulation code ads a lot of potential bugs to the hypervisor. I hope our current hypervisors are an evolutionary transitional form and in the future we'll interact with hypervisors solely through upcalls.
When we choose a direction, we see all the difficulties that path provides and kind of forget about all of the things it gives us.
And when looking at the other option, we see all the things that would overcome our difficulties while assuming that we get to keep everything good from the current path.
Everything is a trade off.
QNX and many embedded OS, some of which driving high integrity software, are all microkernel based.
Including the one most likely handling the real time communication of this mobile radio.
And I'll throw out there that many times the the term microkernel in a lot of embedded OSs has been contorted by marketing speak into something unrecognizable. Basically if you have multiple threads in the kernel, structure the kernel code into modules (but perhaps don't even allow dynamic loading of modules), and can communicate through async queues, then it's called a microkernel. For instance VXWorks until recently didn't have memory protection between tasks, but still called itself a microkernel.
Strictly speaking, "hard realtime" means strictly bounded latency. Full determinism is overkill.
As was posted elsewhere, they seem to be fast enough for games on the Nintendo Switch: https://news.ycombinator.com/item?id=17768537
>><rant>
>>Why do the file systems have to be so tightly integrated in the "ring0" core? This is one subsystem that screams for standard callouts and "ring1" level.
>></rant off>
> Because only naive people think you can do it efficiently any other way.
> Face it, microkernels and message passing on that level died a long time ago, and that's a GOOD THING.
> Most of the serious processing happens outside the filesystem (ie the VFS layer keeps track of name caches, stat caches, content caches etc), and all of those data structures are totally filesystem-independent (in a well-designed system) and are used heavily by things like memory management. Think mmap - the content caches are exposed to user space etc. But that's not the only thing - the name cache is used extensively to allow people to see where their data comes from (think "pwd", but on steroids), and none of this is anything that the low-level filesystem should ever care about.
> At the same time, all those (ring0 - core) filesystem data structures HAVE TO BE MADE AVAILABLE to the low-level filesystem for any kind of efficient processing. If you think we're going to copy file contents around, you're just crazy. In other words, the filesystem has to be able to directly access the name cache, and the content caches. Which in turn means that it has to be ring0 (core) too.
> If you don't care about performance, you can add call-outs and copy-in and copy-out etc crap. I'm telling you that you would be crazy to do it, but judging from some of the people in academic OS research, you wouldn't be alone in your own delusional world of crap.
> Sorry to burst your bubble.
* https://blog.darknedgy.net/technology/2016/01/01/0/ (https://news.ycombinator.com/item?id=10824382)
But hey, it's lunch and why not.
1) Microkernels are defined by their small size
> In summary, the microkernel provides mechanisms corresponding to hardware features. It doesn’t provide services, just fundamental mechanisms. In particular, it does not duplicate OS services. This misunderstanding was one of the causes for the failure of first-generation microkernels. The OS community understands this.
I'd argue that exokernels fit this model much more effectively. I love XOK's kernel level support for what we'd now call DPDK with BPF (they had user space multiplexed packet queues with an in kernel vm doing filtering). And it's capability based file system cache was an amazing way to address something that no microkernel has tried to, and I've never seen anyone try to replicate anything like that yet. In fact exokernels became so good at multiplexing hardware that Xen came out of the exokernel research.
2) Microkernels are unperformant
He does what most microkernel fans do, which is regurgitate a bunch of papers from the early 90s about how if you use a 486 or early RISC (where trapping to kernel space is only as costly as a branch mispredict) then it's not that expensive. Well guess what, it's not 1992 anymore, trapping to kernel space is on the order of 1000 cycles these days instead of 5, and even outside of microkernels, that bridge has gotten so expensive that you see people not even using the kernel anymore, but instead running drivers inside the user process (see DPDK).
And yes there's a lot of hand waving about FUSE, but nobody claims that FUSE is performant solution just that the slow model that can't crash the kernel is useful in a lot fo cases.
The QNX stuff is a little disingenuous as it's essentially marketing documentation. Yes, if you push everything through a pipe on unix, you won't get the best speed. Let's compare QNX to the Unices performant solutions (ie. mmap and shared memory). Yes they made some microbenchmarks faster, but most of that is viewed as control plane rather than data plane stuff on Unix.
I'll extend a peace offering here and say that a lot of the microkernel is unperformant was a reflection of gen 1 and 2 ukernels like mach that had to do a permission check on every port access. More modern system (specifically thinking about the L4 variants here) have a capability based model that needs far fewer checks at runtime. However what I said about how traps into the kernel have gotten more expensive still hold true.
3) Microkernels are a diversion, because a userland server failure will be just as catastrophic as a kernel failure anyway
Monolithic kernels typically have the same layering internally that Minix calls out. It's about as easy to add layers to linux block device structures as it is on minix, including all of the checksumming, etc.
And neither linux nor minix protects you from logical bugs in filesystems screwing up anything and everything. We like to think of the kernel in a ukernel as being totally separate and pure from the FS and block device drivers, but it had to be loaded from somewhere...
IMO, the monolithic/microkernel dichotomy here is a false one. Neither really address the issue per se, we need to move to type safe machine checkable languages. A recent example of microkernels screwing this stuff up was the original exploit chain that opened the nintendo switch.
4) Microkernels turn well understood memory-protection problems into poorly understood communication problems
He just spends the whole time here handwaving away how difficult distributed systems are because the internet works a lot of the time. Huge red flag for me.
5) Microkernels are a bourgeois plot to undermine free software
I used to write off what he's arguing against here, but I'm really coming around to the idea. Google's Zircon in Fuschia seems specifically designed to remove GPLv2 requirements from systems like ChromeOS and Android. Seems like a net loss for software freedom.
6) Hah, those stupid fucks are running Linux on top of their microkernel! What happened to microkernels being so great, fags?
(Ignoring the slur for the purposes of this discussion) he has a decent argument here, but I'd like to see the argument compared to containers where spinning up new OS spaces is even cheaper than a whole VM for a personality. I think a blend of the techniques a la the NtPicoProcess stuff that makes WSL here is going to be the winner here more than current ukernel design where that code runs in a different address space.
7) If microkernels are so great, why is nobody using them?
This is where he hits that disingenuousness I bring up in my parent post.
TRON isn't a microkernel, hell it doesn't even require an MMU.
OKL4 loves bringing up how basebands they're running on, but how many codebases is that?
Same with MINIX.
Done for now, haver to get back to work
Yeah, no. A kernel has access to all of the data on the machine. A compromised userland is much more strictly limited.
Certainly if a compromised disk driver exposes a lot more data, not so much if you compromise your video driver. Compromising either in a monolithic kernel provides the same level of access. Strong isolation boundaries strictly limit risk.
> 6) Hah, those stupid fucks are running Linux on top of their microkernel! What happened to microkernels being so great, fags?
Driver support is always an issue. I don't see the problem.
> Certainly if a compromised disk driver exposes a lot more data, not so much if you compromise your video driver. Compromising either in a monolithic kernel provides the same level of access. Strong isolation boundaries strictly limit risk.
About the only part of the video drivers that runs in kernel space even on monolithic kernels these days are the drivers for the GPU's MMU. You screw that up and you're right back to where you started, corrupting random memory without regard for protection boundaries regardless of where that code lives. Everything else runs in user space, but not in an isolated process like on a ukernel, but directly in user process code, more like an exokernel.
> Driver support is always an issue. I don't see the problem.
My point underneath was comparing to other paravirtualized kernels was a best case comparison for ukernels, and zones/jails/containers has added a new more performant option for them that doesn't look as good for ukernels.
https://www.youtube.com/watch?v=PYZDl4RNEVE
Maybe they had some stuff in kernel mode in there. They still show you can start with a microkernel as the baseline then optimize from there retaining the architecture with its benefits in average case or most cases.
When I was in college, the first assignment was to run a provided benchmark program that jumped all over a gradually growing buffer and graphed average latency vs. buffer size and we had to identify the sizes of thle various caches from the generated graphs. I was tripple-booting Linux Windows, and QNX at the time and did the same assignment on each OS (identical hardware). QNX had more sudden transitions between latency levels due to delaying transition later due to smaller cache footprint.
If your userspace NIC driver and TCP/IP stack are implementee as a library and you are running your high performance application on a dual NIC box, your application can be the driver for one of the NICs and the other NIC can handle everything else. It's hard to beat zero copy/zero context switch I/O. Of course, putting the application in the kernel would yet be faster, at least if in both cases the kernek's cache footpint is tiny.
Assuming you didn't have those mistakes, the difference was likely due to memory allocator choices. Scattering the pages of memory is very different from laying them all out one after another. Scattered pages will cause benchmark results to be less predictable, with evictions seeming to happen at random. QNX probably made one great big allocation, in both the virtual and physical address spaces.
I noticed a later filesystem simulation benchmark for the same class ran faster when I moved my mouse. After a few minutes of investigation, I submitted a patch to prefer /dev/urandom to /dev/random on hosts that had /dev/random.
That debate would be short and the answer would be "nope, we shouldn't because that would be stupid".
edit added
> Guys, there is a _reason_ why microkernels suck. This is an example of how things are _not_ "independent". The filesystems depend on the VM, and the VM depends on the filesystem. You can't just split them up as if they were two separate things (or rather: you _can_ split them up, but they still very much need to know about each other in very intimate ways).