Hacker News Reader

Comment by scorpioxy | Hacker News Reader

From my understanding, the microkernel approach has been deemed impractical for real-world use. I wonder if there is anyone out there using it?

View on HN

Your understanding is, I'm sorry if this comes across as hostile, absolute nonsense. You should actually consult the literature instead of reading Linus Torvalds rants.

Back in 1990, the Amoeba system revealed 2x improvement in throughput and 5x improvement in latency for its RPC over the SunRPC of the time. [1] Relative measurements for the V-System and Sprite were similar. QNX, a microkernel-based system with high commercial success and a long history (used by over 40 automotive manufacturers) has highly fast IPC by integrating message passing with the CPU scheduler. After years of research, Jochann Liedtke introduced L4 in 1996 [2] using only seven generalized calls with a 20x improvement in speed over prior art such as Mach. Mach, by the way, is the glaring exception to microkernel performance because of complicated message packing and port rights checking. Yet the Hurd developers are doing well in optimizing it, and to this day Mach is used to malign microkernels by people with no background on the subject. OKL4, in turn, has shipped in ~1.5 billion devices by 2012, powering the baseband processor behind nearly every mobile phone. [3] MINIX 3 further only shows a ~5-10% performance drop relative to monolithic Unixes, this for 2006. [4]

They never learn.

[1] http://www.scs.stanford.edu/nyu/03sp/sched/amoeba.pdf

[2] https://homes.cs.washington.edu/~bershad/590s/papers/towards...

[3] http://www.creativemac.com/article/OK-Labs-Software-Surpasse...

[4] http://www.minix3.org/doc/ACSAC-2006.pdf

as an aside, how is Mach doing these days? I suppose the most visible use is OS X and I assume Apple's engineers have evolved it greatly to keep up with the times compared with the earlier versions of OS X?

Apple's XNU is de facto monolithic in nature. It simply reuses OSF Mach primitives to build the BSD Unix layer next to it. Mach IPC is used primarily for service discovery these days, I think, what with launchd being responsible for being the bootstrap server, which maintains name-port bindings for Mach services to call into. This is a side effect of Mach port namespaces being internal to tasks, and hence not having an intrinsic arbiter. In contrast though, the Hurd decentralizes this activity by just assigning a bootstrap port for the client making the RPC call to rendezvous with the server or library in question. These days, OS X seldom uses Mach IPC directly, but rather via XPC.

Mach had a surprising resurgence in interest over NextBSD implementing a ton of compatibility layers and mangling Apple sources in a weird, misguided attempt to make FreeBSD the new OS X. We'll see how it turns out.

By way of example, you can't create a Mach task that isn't a BSD process as well: https://github.com/opensource-apple/xnu/blob/10.10/osfmk/ker...

The Apple Newton used a microkernel inspired by Mach, with a custom MMU that reduced the cost of context switching.

It was . . . okay. The number of cycles it actually took to do a lightweight message pass was dismaying, though. The newt would have benefited from a less partitioned design for the underlying page and storage management.

I used another microkernel OS on a set-top box. Now that was a misery, but one I attribute more to the uncaring attitude of the company that wrote the STB's firmware than any inherent problem with their OS (a decade later, critical and customer-affecting race conditions are still present in their code). To be honest, their OS didn't help.

I don't think microkernels are bad. Making bad choices about performance that affect performance, battery life and other things that customers care about is bad, and you might have to break some abstractions in your microkernel to address those.

> Now that was a misery, but one I attribute more to the uncaring attitude of the company that wrote the STB's firmware than any inherent problem with their OS (a decade later, critical and customer-affecting race conditions are still present in their code). To be honest, their OS didn't help.

Hmmm STMicro-based? :)

68K, actually, and MIPS. Different platforms, same damned OS bugs. Let's hear it for portability :-)

Didn't the Microsoft NT project start off as a micro kernel and then changed course to a hybrid model ?

https://en.wikipedia.org/wiki/Hybrid_kernel

NT was never a microkernel. It has an intenral microkernel-like design, though; it's divided into cleanly separated modules, and modules may only communicate using a special IPC mechanism. But like OS X, the classic pieces (file systems, graphics, device drivers) run in kernel mode.

Originally, graphics ran in user mode, but Windows 2000 moved it back into kernel mode for performance reasons. Vista got a new device driver model and a new graphics driver model (WDDM), which has a way for graphics to run partly in user mode. Vista and later also support user-mode device drivers.

Windows NT 3.1 was based on Microsoft OS/2 NT 3.0, it was originally Microsoft's version of OS/2 with the Windows 3>X GUI added to it instead of IBM's Presentation Manager.

The NT Kernel and OS was designed to run DOS, POSIX, OS/2 1.X CLI apps (But not GUI) and 16 bit Windows and 32 bit Windows apps. Later on OS/2 and POSIX support got dropped.

Microsoft OS/2 NT 3.0 was a rewrite of OS/2 for 32 bit systems, before that OS/2 was on 16 bit systems with 1.X and IBM and Microsoft fought over 2.0 standards, and Microsoft stopped support for OS/2 and focused on Windows instead. Then renamed their OS/2 to NT for New Technologies.

The Interesting thing about Windows NT was that Microsoft had plans to port it to different processors and eventually dropped those plans and stuck with X86 processors instead.

https://en.wikipedia.org/wiki/Windows_NT

It was designed as a modified microkernel.

> The Interesting thing about Windows NT was that Microsoft had plans to port it to different processors

The didn't plan it, they did it. I was looking after Windows NT on Alpha processors at one job. I never personally saw the NT on PowerPC or MIPS, though they did exist.

You got it wrong (your first sentence), WinNT3.1 is written from scratch by Dave Cutler and his group from 1989 to 1994. Originally there would have been a OS/2 compatible subsystem, and some ideas stem from OS/2 but NT has a lot of more similarities with VMS, the previous OS project of Mr Cutler at DEC.

> The NT Kernel and OS was designed to run DOS, POSIX, ...

What you're referring to were what the NT team called "personalities", or more formally "environment subsystems".

A subsystem was an API on top of the native NT API. Win32 was one such subsystem; users were exposed to the Win32 API, and weren't supposed to talk directly to the intentionally undocumented NT API (although some developers did reverse-engineer it eventually). POSIX and OS/2 compatibility was implemented as subsystems, but these were rarely used by anyone, and eventually removed in XP. DOS was not a subsystem, but rather its own thing.

Note that this has nothing to do with microkernels. In a microkernel architecture, things like file systems and device drivers live run as normal processes separate from the kernel. In NT, everything was in the kernel. These subsystems didn't act as part of a microkernel design.

Oh, and NT was not a rewrite of OS/2. It was written from scratch. The project was started back when Microsoft and IBM had a good relationship, and NT was originally planned to as OS/2 3.0.

I think a lot of the common idioms passed around about microkernels stem from the old Torvalds v. Tannenbaum argument [1]. It wasn't that Torvalds was right and Tannebaum was wrong. It was simply that Tannenbaum had reliability and theoretical correctness in mind, while Torvalds was concerned with performance. The less context switching you do, the more performant your computer will be. We can take this to the extreme and look at OSes like TempleOS [2] where EVERYTHING runs in kernel mode. Most people prefer a bit more reliability, and so modern desktop kernels are generally hybrid kernels (with the notable exception of Linux which is, of course monolithic). I don't think there's any real technical reason a microkernel couldn't work on today's hardware which is many orders of magnitude faster than what we had in 1992, but a hybrid kernel is always going to be faster.

[1] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...

[2] http://www.templeos.org/

I always found it interesting how much Intel's protection ring architecture dictated the direction of operating systems back in the 1980s. (I'm sure Intel didn't even invent it; similar concepts were probably already in place in mainframes?)

x86 has highly specific support for protection rings and switching between them, as well as things like page faulting and interrupt management, leading to the classic kernel/user split with a kernel as a privileged actor underneath a user mode.

But having just one exclusive, reserved "kernel mode" is starting to look old, which is why there's now so much talk about virtualization and exokernels and so on. The microkernel design certainly seems very elegant, but it looks to me like Intel's architecture was always a stumbling block. You have to wonder about what hardware support you could invent that would make microkernels a better fit.

Take a look at the Mill's security presentation.

It's an architecture where there is no real border between a monolithical kernel and a microkernel. It's just differences on access control policies, to the point that those names lose their meaning.

As always, it's a very interesting architecture. I hope they produce it someday.

Protection rings are a lot older than x86. They were first used on the Honeywell 6180 for Multics back in the 60s, and used in pretty much every sufficiently large processor. The Vax had two-ring layout, as did the 68k, so the seeds for OS development in the 80s were already pretty soundly planted.

The page faulting was also on older systems, because putting those things in hardware is a lot faster than doing those things in software, plus controlling memory access really should be a privileged activity. Interrupt handling is in a similar situation, and even there, you still need some process handling the interrupt vector table. It's possible to make most of an interrupt handler a user-level process through page table and interrupt return address hacking, but for the moment, it's unfortunately rare.

As for your desire for a replacement for one exclusive, reserved kernel mode, there have been a few OSes that have tried to break that pattern. OS/2 used Ring 2 of the x86 for drivers, but unfortunately that bit wasn't added to Windows when they were forking NT. Being able to put semi-trusted drivers in a separate area, and perhaps even a user session manager too, could allow for some interesting security experiments that don't rely on (para)virtualization.

Hardware-wise, it would be useful to have hardware contexts, like sparcs have, so that the group of registers a process has can be swapped in and out a lot easier. Context switching is expensive, and building processors that realize that the modern user tends to have more than one task running would be a pretty good performance win.

The vax had 4 modes: kernel, executive, supervisor, user. I just read[0] that kernel and executive had implicit SETPRV privilege (and thus access to those modes would be tightly controlled because SETPRV allows bypassing all access controls); and I think I heard a while back that normal programs ran in user mode but the shell-equivalent ran in supervisor mode (and [1] sort of implies this).

[0] http://h30266.www3.hp.com/odl/vax/opsys/vmsos73/vmsos73/5841...

[1] "If you created the name with a DCL command, the access mode defaults to supervisor mode. If you created the name with a program, the access mode typically defaults to user mode." http://h71000.www7.hp.com/doc/731final/4477/4477pro_007.html

You're right. The biggest news for operating systems is the fact that the hardware virtualization features on modern processors allow breaking free from the old model.

A modern intel/amd processor lets you virtualize the CPU, MMU, and I/O. Network cards and HBAs can be partitioned into virtual NICs.

Two primary functions of a classic OS are process isolation (i.e. virtual memory) and hardware sharing. But now that these two primary functions have been pushed down into hardware, it has changed the way we think about operating systems considerably.

There isn't much left for a classic OS to do other than provide a common set of APIs for programs to talk through. Yet these days we can statically link even the largest libraries into our exokernels. And hypervisors are capable of using techniques like same-page merging to reduce the memory burden of running many large (exo)kernels at once.

I don't expect the the classic one-OS-running-many-processes model to go away overnight, or possibly ever. But the exokernel model is very compelling for large-scale high performance software services and it will continue to catch on.

GEMSOS and STOP OS built highly-secure systems (for the time) on Intel by using the protection rings and segments. Both only put the security kernel in kernel mode, user apps in user mode, and OS services in middle rings.

Here's architecture for GEMSOS. See design/assurance sections.

http://aesec.com/eval/NCSC-FER-94-008.pdf

Look up SCOMP Final Evaluation Report if you want to see how STOP OS used four rings and had an IOMMU despite that being "invented" recently. ;) The XTS-400 is the Intel version, uses same architecture minus custom hardware, and is still doing its job at hundreds of installations.

Definitely a major performance hit on both GEMSOS and STOP but they were 80's era stuff. Modern separation kernels do most stuff with just user/kernel mode separation with tiny kernels (4-12kloc). LynxSecure claims helps them keep CPU 97% idle with 100,000+ context switches a second. I'd expect old architectures to run even faster with modern techniques.

> Torvalds was concerned with performance

Performance is not a single metric. There is throughput, latency and then you can screw it all up and make it much harder by demanding guarantees on either of those.

Performance without guarantees is worth very little in quite a few situations.

Good point. The other argument is also true; Security and correctness is not a single metric.

With modern (buggy) hardware and DMA access, when your driver and/or hardware fails all bets are off. Some hardware may be possible to reboot (much as you'd reinitialize a kernel module in Linux), but sometimes your best course of action is a complete reboot.

As for security, you also need to take a long hard look at the the operating systems your operating system relies on, such as the ones powering your disks, nic, pci-controller etc. There are some potential tricky security interactions with them.

SMM and other such ring "-1" type "services" in modern CPUs make your point quite clear to anyone who digs deep enough.

When trying to secure a system, we have reached the point where you have to sometimes as "is this CPU opcode safe?" Sometimes it just feels like modern hardware complexity is reaching some kind of critical mass threshold for "stupid shit"

That point was reached back in 90's when first security evaluations of Intel architecture were done, found tons of black boxes like SMM, and said to ditch it for security or virtualization. Invisible Things did a good job demonstrating an old risk but people should've ditched it long ago.

If you want verifiable hardware, look up the VAMP processor as it has everything from design descriptions to formal proofs of correctness. Not sure about its availability. SPARC and RISC-V are very open with open-source implementations available with Linux and compiler support. So, there's a solution if people ever want to put the work in.

QNX [1] is probably being the most successful and most mainstream example of a microkernel in real-world use. It's extremely popular for embedded use and/or apps with real-time demands, most famously in nuclear power plants. I've never used QNX myself, but I believe people also run full desktop environments on top of it.

[1] https://en.wikipedia.org/wiki/QNX

I've used the desktop version. It's amazing: it looks, and feels, and operates, just like a Unix, except if you want it it's got this ultra-fast real-time core.

When I say 'just like a Unix', I mean: you log in and start up some terminal windows and it's sh and you can compile and run X11 software with configure scripts and gcc and you can print stuff with lpr and it all just works. It even runs Java --- my copy was completely self-hosting via Eclipse!

Years ago there was a single-floppy QNX demo disk: you booted from this and you got a basic desktop with dialup modem support and a web browser. SINGLE FLOPPY.

Hey, look! I was all set to write a paragraph about how sad it was that you couldn't get the bootable QNX CD any more, but look what I found!

http://www.qnx.com/download/feature.html?programid=19602

It doesn't even need registration and a login any more! Holy crap, I have to see if this still works...

Edit: looks like it won't install without a license key. I'll see if I can get one...

Edit: apparently I have an account with QNX dating back from 2012, with three hobbyist license keys, one of which makes the installation CD happy. I don't know whether these are still available, though.

Edit: so it installed into a VM in about two minutes flat, and I now have a very old copy of Firefox running. It can't see the network, but that's nothing to do with QNX and everything to do with my inability to set up kvm. I need to find a real machine to run this on.

Edit: it will only install from CD, not from USB (unetbootin doesn't help). And I've lost the power cable for my CD burner. So until I find it, I won't be able to proceed here. Sorry. Still good to know this still exists, though...

They have a license for evaluation & noncommercial use. http://www.qnx.com/legal/licensing/non_commercial.html

> Years ago there was a single-floppy QNX demo disk: you booted from this and you got a basic desktop with dialup modem support and a web browser. SINGLE FLOPPY.

Dan Hildebrand's original announcement on comp.os.linux.development.system back in 1997: http://marc.info/?l=freebsd-chat&m=103030933111004

Archived homepage of the QNX Demo Disk: http://web.archive.org/web/20011019174050/www.qnx.com/demodi...

including "How we did it":

http://web.archive.org/web/20011106140711/http://www.qnx.com...

and "What people are saying" (to give folks today an idea of the excitement around a 1.44MB GUI OS with networking and Japanese support back in the 90s):

http://web.archive.org/web/20011106141359/http://www.qnx.com...

Download links and screenshots:

http://marc.info/?l=freebsd-chat&m=103030933111004

And a very nice paper by its wonderful last author

https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.p...

That sounds awesome. How does it compare to RTLinux or Linux with the PREEMPT_RT patches?

I really enjoyed using QNX (I had a job porting Unix programs to QNX---it was not that hard of a job actually). What really blew me away as the inherent transparent networking. I could, from the command line, run a program on my computer, referencing a file on a second computer, pipe the output to a program on a third computer which sends the output to a file on a fourth computer. Really mind blowing stuff.

And fast. I knew the owners of a company that sold X Windows commercially, and their fastest version ran under QNX, using the native QNX message passing facilities. The fact that the QNX kernel on a Pentium was only 8K in size was also mind blowing.

I used an eval of the desktop version of QNX for a short time. Got it from a CD that came along with a computer magazine.

Even in that short time, I got a feel that the OS was quite fast, compared to early Linuxes that I used around the same time, and Windows. Remember reading in the mag that it had that Photon GUI, IIRC, that others have mentioned in comments here. I tried out at least some apps, bash, etc., and they were all snappy.

I used QNX extensively for a few projects at work, as it has historically been used quite a bit in the "automotive infotainment" world. I was really impressed by the overall clean architecture and documentation of the system, and I much prefer implementing device drivers in the QNX microkernel environment over implementing them in Linux, where interfaces are under-documented and always changing, and it's quite easy to lock up the entire system during driver development.

An interesting data point, though: QNX had a very clean and modularized 'microkernel-like' network stack called 'io-net'. But due to throughput issues in some situations, they switched to a new architecture a few years ago called 'io-pkt'. This is essentially the kernel networking code from BSD transplanted to a process in QNX; one advantage of this is that there's a large stock of drivers available to port and a lot of people are familiar with the BSD networking model, but some of the lesser-used corners of it weren't fully debugged when I was doing network protocol hacking, and in general it made me sad.

Anyway, you can definitely run full desktop environments on it, and you can especially run a full tablet environment if you buy a recent Blackberry tablet, as RIM now owns QNX and uses it as their latest Blackberry OS. This was, unfortunately, a step backwards in their openness and embrace of Open Source.

To swipe a line from Chesterton: micro-kernels have not been tried and found wanting, they have been found different and wanted trying.

It took decades to get UNIX's reliable enough to trust for day to day operation with labor estimates in hundreds of millions to billions of dollars. Three or so people got that done in a few years with MINIX 3 using micro-kernel architecture and self-healing services. QNX and INTEGRITY RTOS's have been doing that stuff for years in the field with less resources. And QNX-based Playbook was more reliable, responsive, and fast than iPad in its first incarnation. Before that, KeyKOS outperformed IBM's monolithic stuff on their mainframes with better security and full persistence of system state.

So, they worked better and keep working better. People just use monolithic kernels for some reason. Just like they took forever to get off of C for reliable, business applications. "Worse is better."

Your phone likely has a L4 microkernel running the radio. There are millions and millions of microkernels out there running all kinds of things, going where Linux fear to tread.

I'm surprised that something so incredibly prevalent is so rarely discussed.

Much of the tech community views this as "settled" thanks to the Linux versus Tannenbaum debate.

Microkernels aren't going away any time soon. :-)

Oh I know - I meant more "running L4 kernels", with the seL4 currently looking quite interesting.

At what point do you conclude that you're:

* running everything in hypervisor X, thus hardware support doesn't matter * running only servers, thus desktops don't matter * and thusly, may stand more to gain from an environment that never catered to the above two?

In his defence of them, Tannenbaum mentions a lot of companies using microkernels for high-reliability applications:

https://www.cs.vu.nl/~ast/reliable-os/

For consumer devices, the Blackberry Playbook was one of the better examples given it was built on QNX microkernel for reliability and nobody was complaining about its performance (i.e. doing two games at once). Lots of phones have microkernels in them to isolate baseband from problems in main OS. Did you know you were using a virtualized OS message passing to other components? I didn't either until I saw the vendor's press release. They might be efficient after all. ;)

Here's one used a lot in safety-critical and security-critical apps with nice security architecture:

http://www.ghs.com/products/rtos/integrity.html

The beautiful and efficient MorphOS uses a microkernel:

http://www.morphos-team.net/intro

One that was made for capability-security model with FOSS code:

https://web.archive.org/web/20070428020436/http://www.eros-o...

Note: See their papers and KeyKOS especially, as it was first successful one on IBM mainframes of old.

GenodeOS and MINIX 3 are the best efforts to look at in FOSS today as they're actively maintained. Not at Linux feature set or stability yet given almost no staff haha. Already on bare metal and hosting apps/desktops, though. Just don't think stuff like Hurd is representative of microkernels in general. It's just a failed GNU project. ;)

I doubt the engineers at Blackberry would agree with you when they chose the QNX microkernel as the basis of their next generation OS. Also I never heard any complaints about AmigaOS’s or Symbian's microkernel design as being impractical.

AmigaOS never implemented any memory protection, did it? So it hugely favours a microkernel design, since there's no expensive context switch to translate a call into kernel mode; everything is user mode.

Whilst you are correct in that AmigaOS didn't implement memory protection, it's possible to argue that wasn't by design.

The original design for AmigaOS was known as CAOS. The CAOS design included resource tracking, which I believe is a useful feature when it comes to implementing memory protection.

If I remember correctly what ended up happening was the microkernel for CAOS was developed in-house (Exec), but contractors were being paid to develop the rest of the OS, and they disagreed with the design decisions of CAOS and wanted to make it more Unix-like. Once the Amiga team realised the contractors had wasted time developing something different from what they intended, they sought out an alternative solution to replace the user-space side of the OS. As a result they contracted a different team to rework an OS called TripOS to use the Exec microkernel (that was originally intended for CAOS), which then became the AmigaOS that Amiga users know.

You can read a bit more here:

http://www.amigahistory.plus.com/caos.html

Quark microkernel in MorphOS https://en.wikipedia.org/wiki/Quark_%28kernel%29

MorphOS - quite fast and usable :) http://www.morphos-team.net/intro

So, out with the old and in with the new.

Does MorphOS have resource tracking and memory protection now? It didn't last time I was following it (about 3 years ago). To be fair, none of the next gen Amiga operating systems (OS4, MorphOS, AROS) had implemented that last time I checked.

The second paragraph of the Wikipedia article mentions user/kernel-mode division and kernel-protected objects. Past that I don't know. Probably none for legacy systems and similar to Darwin (coarse-grained) for new ones. Could be more fine-grained, though.

Both Qualcomm and Apple are known to run variants of L4, a microkernel, on their mobile hardware: https://en.wikipedia.org/wiki/L4_microkernel_family#Commerci...

And Samsung uses INTEGRITY Multivisor in KNOX.

http://www.ghs.com/products/rtos/integrity_virtualization.ht...

Multivisor is kind of a combination of the INTEGRITY RTOS and virtualization stuff.

The leader in this kind of stuff was OKL4 like in your reference, though. General Dynamics supplies their stuff now due to acquisition.

Do you mean for GNU Hurd specifically, or just microkernels in general? Because there are several microkernels in production use if you look beyond desktop OSes.

As other said, QNX is used, in a lot of places. The microkernel issues raised in the 90s have been addressed partially at least (I'm just a noob) in the L4 mkernel family. Minix v3 too IIUC.

Not all microkernels. There are many newer ones which have much better performance characteristics of their predecessors. Mach, however, has rather slow IPC, which would be a problem for Hurd.

Mac OS X is based on the Mach microkernel. I would think that counts as real-world use.

XNU (OS X's kernel) doesn't really use Mach as a microkernel. It supports user-mode device drivers, so you might call it a hybrid microkernel, but practically speaking, it's a monolith just like Linux; everything of practical relevance (file system, graphics, most disk and networking I/O) runs in kernel mode. Some parts (high-level USB, Bluetooth, power management, etc.) run as user-space processes, but they talk to kernel-mode device drivers.

(XNU was inherited from NeXT, which was based on Mach 2.5, which was actually not a microkernel at all.)

Reply on news.ycombinator.com