A reimplementation of NetBSD using a Microkernel [video]
youtube.com
youtube.com
All you really need in a practical microkernel is process management, memory management, timer management, and message passing. (It's possible to have even less in the kernel; L4 moved the copying of messages out of the kernel. Then you have to have shared memory between processes to pass messages, which means the kernel is safe but processes aren't.)
The amusing thing is that Linux, after several decades, now has support for all that. But it also has all the legacy stuff which doesn't use those features. That's why the Linux kernel is insanely huge. The big advantage of a microkernel is that, if you do it right, you don't change it much, if at all. It can even be in ROM. That's quite common with QNX embedded systems.
(If QNX, the company, weren't such a pain... They went from closed source to partially open source (not free, but you could look at some code) to closed source to open source (you could look at the kernel) to closed source. Most of the developers got fed up and quit using it. It's still used; Boston Dynamics' robots use it. If you need hard real time and the problem is too big for something like VxWorks, QNX is still the way to go.)
I would like to hear Mr. Tanenbaum's answer to the less provocative form of the sentiment: "What design decisions were made with MINIX3 that other RTOS with microkernels didn't consider?"
And Minix isn't a micro kernel in the same way that QnX is.
If something didn't work it was either hardware or our own code, the way it should be. It's not magic but it is very good at what it does. And it is a way of building things that just works, message passing is a very powerful technique for making reliable distributed systems.
To me this sounds a bit funny - isn't the whole point of microkernel architecture that you achieve the reliability by simplifying the kernel - it's a less complex piece of technology, so it can be more reliable, and the complexity is offloaded into user space processes.
Yes, it does not do what it should not do, because it does less.
It's just that with the microkernel more of the code will be in stand-alone programs. Every driver, file system, network layer and so on will be a program all by itself.
If you need hard real time and the problem is too big for something like VxWorks, QNX is still the way to go.
There's all sorts of much tinier RTOS like FreeRTOS, MicroC/OS and Contiki that are used out there for particularly critical and/or constrained environments.
They've downplayed that recently, alas, but I believe that if you hunt around on their website you can find a bootable CD. I think platform support has slipped a bit so you might have trouble making it boot.
Way back when, there was a QNX demo floppy, which was a bootable 1.44 MB floppy disk which contained a full GUI, web browser, dialup modem support, etc. It'd run on a 386 with 8MB of RAM.
http://toastytech.com/guis/qnxdemo.html
QNX is pretty awesome.
Edit: Here's a similar writeup of the bootable CD, using a more recent version of the GUI. http://toastytech.com/guis/qnx621.html
Well, regardless of how late, Minix is bringing it fuss-free to the rest of us now.
I am not complaining as I find Minix great, just making the point that nothing is really free.
QNX used to be a standalone company. They declined a buyout by Microsoft. Then one of their key people died, and they sold out to Harmon, the car audio company. So then they were heavily into automotive dashboard applications, which continues. Then RIM bought them, which gave the Blackberry a better underlying OS but didn't fix Blackberry's problem of an obsolete UI and business model.
Meanwhile, QNX still sells to real-time and industrial automation customers, but those customers feel kind of neglected. Incidentally, the Boston Dynamics robots all have QNX managing the balance and the hydraulic servovalves. You need reliable hard real time for that.
Blackberry isn't failing because of their OS, they're failing because they were way too late to market with robust smartphones and lost a huge amount of mindshare after Apple and Google ate their lunch.
I think they should stick to their strong suit and build on strengths in business world. Collaboration, integration with legacy stuff, bake in more security than iPhone/Android, and so on. Build services on top of that like IBM does with all their stuff. And so on.
EDIT: http://www.embeddedrelated.com/showthread/comp.arch.embedded... says:
> the most fundamental difference between VxWorks and QNX is as you have described, QNX lends itself to a message passing architecture while VxWorks lends itself to a shared memory architecture.
>
> My personal opinion is that a message passing architecture is easier to get to grips with and as such is potentially easier to understand and debug.
> However, the majority of software engineers with experience of an embedded RTOS will be very well informed about the Shared Memory architecture.
With respect to the message passing - both support messaging. VxWorks has several types of message queues - vxworks proprietary msgQLib API, POSIX api etc. QNX has much the same MsgSend/MsgRecv which is the microkernel API and POSIX. QNX has an add-on PubSub middleware that the OP of the usenet group may be thinking of.
Based on the MINIX 3 microkernel, we have constructed a system that to the user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and libraries, and passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running, and without user processes noticing it. The talk will discuss the history, goals, technology, and status of the project.
Research at the Vrije Universiteit has resulted in a reimplementation of NetBSD using a microkernel instead of the traditional monolithic kernel. To the user, the system looks a great deal like NetBSD (it passes over 80% of the KYUA tests). However, inside, the system is completely different. At the bottom is a small (about 13,000 lines of code) microkernel that handles interrupts, message passing, low-level scheduling, and hardware related details. Nearly all of the actual operating system, including memory management, the file system(s), paging, and all the device drivers run as user-mode processes protected by the MMU. As a consequence, failures or security issues in one component cannot spread to other ones. In some cases a failed component can be replaced automatically and on the fly, while the system is running.
The latest work has been adding live update, making it possible to upgrade to a new version of the operating system WITHOUT a reboot and without running processes even noticing. No other operating system can do this.
The system is built on MINIX 3, a derivative of the original MINIX system, which was intended for education. However, after the original author, Andrew Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy of Arts and Sciences and a 2.5 million euro grant from the European Research Council, the focus changed to building a highly reliable, secure, fault tolerant operating system, with an emphasis on embedded systems. The code is open source and can be downloaded from www.minix3.org. It runs on the x86 and ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited over 3 million times and the bootable image file has been downloaded over 600,000 times. The talk will discuss the history, goals, technology, and status of the project.
Here are some things to add:
- Nowadays x86 is built with LLVM by default, and ARM is using GCC.
- X11 is mentioned in the video, but the 3.3.0 release from last fall didn't ship with a working X server, although past releases have. There is a message on the mailing list from someone who writes that they've got it working on a subsequent snapshot release.
- You may have heard something in the past about 10 minute build times. That info is out of date as of the switch to NetBSD userspace for 3.3.0. MINIX itself (i.e., all the interesting parts) still only takes about 10-15 minutes to build, but there's no way to just slurp down the sources for its kernel/drivers/servers to build a MINIX "core" and then supplement it with the prebuilt binaries for userspace (at least, not without doing some significant work on your end to allow for that). The initial build for x86 on my modest machine takes about 3 hours, almost all of it spent building LLVM twice.
- For anyone looking for serious collaborators, MINIX is seriously lacking in infrastructure from a project/community standpoint. E.g., a fair bit of documentation is missing and much of what you will find on the wiki is out of date. Development processes are neither documented nor easily discoverable because there are effectively no development processes in place. Until about six months or so ago, MINIX was without a bugtracker. Organizationally/project-wise, the whole thing is pretty sparse.
- If you have watched previous talks on MINIX, e.g. FOSDEM 2010, you will be familiar with the open calls for those interested in working for pay, using the money from the two grants mentioned in this video. That money is now gone. During that time, MINIX was basically a research project run by grad students who were working ~full time on MINIX, with the funding from those grants. It's the same now as far as the student-run aspect goes, but with drastically fewer contributions. Not much of the paid man hours seem to have gone towards scaffolding out project infrastructure as I mentioned before, or the sorts of drudge work that volunteers are unlikely to take up.
- The code quality is... I dunno, fair? As I mentioned before, there are/were effectively no development processes in place; no code review, etc. So there's a fair bit of nastiness that got checked in directly, like copy and paste, especially among the arch-specific boot time stuff; the comments are fairly sparse; and you can find dead code and references to routines/fields that either no longer exist or have been renamed, even in the relatively short (~500 line) main.c.
For anyone looking in to maybe starting to work with MINIX, I'd suggest assessing whether or not you would be comfortable striking out and doing things on your own, and then being prepared to do so. With MINIX, you aren't going to find a thriving community that you can just add your piece to, so as to contribute to the effort. You might run into a certain level of that sort of old-guard, paralyzing stop energy, so in a way it's got a lot of the downsides of a greenfield project except with few, if any, of the upsides.
I would love to see a sustainable, non-vapourware, micro-kernel. I've _heard_ such great things about them :)
Neat idea but seems nowhere near done.
And as others have said, this was nicely handled by QNX way more than 15 years ago, I was running multiple users on a 80286 around 1986 or so. Really neat system.
I will concede that in some instances a microkernel may outperform a monolithic kernel in stability or performance or both. I am not the least bit excited about any progress made in microkernels, I feel that it can only result in much more closed systems that are easier to implement in ways that make them harder to modify. This is why I wish for Hurd to continue to fail.
[0]: http://www.oreilly.com/openbook/opensources/book/appa.html
Is it because you know that usage decisions of software are not based on technical merits? Or do you not want to be proved wrong? Or something else?
You're Western Digital in 2008 and you're making a TV set-top box called the WDTV-Live. I own one of these in real life universe. It runs linux, which is awesome, because that means that I can SSH into it. It runs an apache server in my home. It can download from usenet or torrents. I can control it via SSH instead of using the remote control.[0]
In this alternative universe, WDLX going to use Hurd instead of linux, because for this small device it will certainly have better performance on their underpowered MIPS chip. And they're not going to ship anything besides what they have to, becasue this is a small embedded computer.
What happens to that homebrew community when they ship a microkernel with proprietary servers for everything, and nothing else? It's going to be profoundly difficult to develop on this. You might already see this if you own a chromebook or a WDTV-- missing kernel modules means that you simply can't do anything without compiling your kernel. Couple this with secureboot and you're locked in.
I'm no expert on these things, most of this is based on brief research from years ago. If you think that I'm wrong, please tell me why, I'd love to be proven wrong. But for the time being, I believe widespread implementations of microkernels would be very anti-general-purpose computing.
[0]: http://wdlxtv.com/
Shipping an embedded appliance with a microkernel and proprietary servers again makes no sense, because it's akin to rewriting userspace from scratch on top of the base VMM, schedulers and disk I/O. Just for a TV set top?
Remember that the reason you can link against glibc is because it's LGPL and not GPL. The LGPL was created for a reason. There's also a reason why when the decision was made to release Java under the GPL, Sun explicitly added a linking exception. It's because that isn't something you just automatically get for free.
Something akin to to affero gpl?
So presumably in this hypothetical case you'd be able to upload and run whatever additional servers you needed on the WDTV. You might say "but they might make it impossible to login and do that", but they could have done the same under Linux just by not running sshd - however they didn't.
The closest you'll find to this alternate universe is Android.
In order to do what you did in your WD TV-Live you flashed the image with a new one. Otherwise you wouldn't be able too. So even in the case of a micro-kernel you would just flash the pre-installed with a new one (voiding warranty).
But to get to the point, do you have any idea how much effort would it take for a corporation, to write a reliable httpd server that has apache's capabilities, plugins, testing and support? Then write their own update system, dhcp client and so on? It would take huge amount of $$$ and time. And most of them would probably be buggy. So either way they would have gone with Free Software, if wanted to stay in the current price range.
This is the reason for AGPL/GPL3 -- not much of an argument against modular software/kernels.
It happens already, to keep hardware costs down. The whole point of linux is that you can pick and choose which userland services to ship... (SSH being a userland service)
> ship a microkernel with proprietary servers for everything, and nothing else?
Whats to stop them now? effort. It costs real money to create propietary programmes from scratch. One of the reasons they would have chosen linux in the first place is that half the work is done for them (decoding libraries, network stacks, hardware interfaces, communications daemons)
Xen allows full OSes to be guests and run on top of it. Microkernels only allow servers to run on top of it, and those servers have to be purpose-written and cannot meaningfully be ported.
Xen doesn't provide hardware abstraction and is fully invisible (except to the extent it advertises itself); microkernels are neither.
Paravirtualization (what you did before VT-x and similar) was an oddity, and blurs these lines a tiny bit, but the distinction is fairly clear otherwise.
L4 is an archetypal microkernel, and people often run full OSes or other ported software under it, including Linux.
> Xen doesn't provide hardware abstraction and is fully invisible (except to the extent it advertises itself); microkernels are neither.
Microkernels typically don't abstract any hardware other than CPU and memory; any other drivers would run under the microkernel.
And Xen is only "invisible" if you run full hardware virtualization and no paravirtualized drivers.
> Paravirtualization (what you did before VT-x and similar)
People still use "paravirtualization" today; see the "virtio" drivers typically used with KVM.
https://microkerneldude.wordpress.com/2008/04/03/microkernel...
Their highly-efficient microkernel has been doing so well that most people using it don't know it's in their smartphones. Do most virtualization solutions have a similar impact on user experience? ;)
Also Mach OS X and Windows are hybrid in design and not the monolithic traditional UNIX way.
https://2011.eurobsdcon.org/papers/gras/minix-bsd.pdf
"ABSTRACT MINIX 3 has imported a significant amount of userland BSD code. The trend began several years ago, but the pace has quickened markedly. We have already imported NetBSD’s buildsystem, NetBSD’s C library, the pkgsrc package management infrastructure, and various userland utilities from NetBSD and FreeBSD."
https://github.com/Stichting-MINIX-Research-Foundation/minix...
virtio seems to be working.
The latest work has been adding live update, making it possible to upgrade
to a new version of the operating system WITHOUT a reboot and without running
processes even noticing. No other operating system can
do this.
Can anyone correct me if I'm wrong – but can't Linux do this with Ksplice, and the more recent live kernel patching by Red Hat?> MUCH BETTER THAN KSPLICE
> KSPLICE can handle only small security patches
> KSPLICE patches the running process
> Over time, crud accumulates in the process
> If the update fails, there is no recovery
Note: And as far as OpenVMS, I believe it because those designers built it like their job depended on it not failing. A cluster of that OS shouldn't experience any significant downtime given about everything I've ever read on it.
They took it down in 2001 to replace it with something that took up 2U of rack space, about 2kw less power and ran windows 2000.
I turned up in 2012 to replace it again with something cloudy and it had 11 years of uptime (well done NT!) again[1] so YMMV.
The cloud based version has gone down about 10 times (thanks Azure!).
[1] not a great position but this was on an isolated network with locked down everything so less of a problem than a normally networked system.
http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm...
Notice how the Intel hardware all failed when things heated up a bit. The AlphaServers running VMS just kept chugging along. The eventual fail-over didn't loose a single transaction. Aggravates me that I can't easily obtain such reliable IT hardware/software anymore outside eBay. I mean, HP NonStop sure as hell doesn't have a hobbyist program with used servers for $130. ;)
Agree with ebay. I still look around for Sun Ultra kit now and then but the wife has other ideas because it's noisy and expensive to run.
" I still look around for Sun Ultra kit now and then but the wife has other ideas because it's noisy and expensive to run."
The battle that never goes away. Haha. This is why you need a basement or soundproofed room for that stuff. That's on my todo list for next house.
For me it is very interesting. I would like to see some day darwin running on a real micro kernel (e.g. an updated mach).
I also like the fact that Hurd makes some progress. when ready, I will definitely switch.
In any case hardware vendors must release more documentation on their hardware to revive the OS scene. Blobs do not do much good.
Sorry this really has nothing to do with the video, just that the tangental thought of linux on mach made me wonder and I was pleasantly surprised.