Roll Your Own UNIX Clone
jamesmolloy.co.uk
jamesmolloy.co.uk
It's absolutely doable.
If you want to make your life easier go for a micro kernel with message passing. That will partition the tasks to the point where a single determined person can go all the way from MBR to prompt.
Runner up in the experience category: make a game from scratch that has multiple threads of execution, but use a single threaded process. That's almost the same experience but completely in userland, which should make your debugging life a lot easier.
But if you want to learn all there is to know about memory management, interrupts and interfacing with hardware as well as process scheduling build a kernel.
Older dr. Dobbs issues contain tons of useful information.
Here is some food for thought: http://ww.com/task.cc.html and http://ww.com/task.h.html
Other sources of inspiration: the early minix kernels and the early work on what is now known as FreeBSD by Bill and Lynne Jolitz (also described in dr Dobbs iirc)
I don't know if I'd say that... For one, unless you are actually going to build vdevs and stuff any kernel you build is going to be a microkernel. In my experience, however, I would not say that message passing makes your life that much easier. In addition, it's probably a good idea to write an ELF compatible OS if you want a prompt. Writing a shell isn't that hard, but I wouldn't want to replicate the basic userland tools if I can just statically compile a busybox library. You should also plan to implement some sort of ramdisk if you want a useable environment (or virtual devs with drivers, but that would probably be harder). Actually, I'd recommend just building off QEMU, since that'll give you rudimentary device framework and keep you from having to debug on physical hardware (like I do).
I don't think I could have ever gotten my little OS to the self hosting stage without that.
It still took more time than I care to remember though :)
A virtual machine is a great tool as well, especially if it gives you access to the cpu contents of the running VM.
That will save you days if not weeks trying to find out why your switch from 'real' to 'protected' mode during the boot didn't work...
Kitten[1] is the LWK which I work with, although my focus is on the Palacios VMM. It's quite a simple OS that is by no means out of a good programmer's reach.
http://www.intel.com/products/processor/manuals/index.htm
[disclaimer: I'm an OS/kernel dev]
http://www.logix.cz/michal/doc/i386/
Getting into protected mode and exec'ing processes in their on private space has been done with ~4 pages of asm.
Also, while devs have to write portable code -- don't bother. It'll save you a bunch of time if you just write the IA32e specific asm code.
If anyone has any interest, I can also post my adaptation of an asmx86 vim syntax file I hacked up from something I found on Stack Overflow. It's not that great, but it does at least fix some of the register highlighting.
I've been working for almost 3 years now and this is an area that I would like to move into.
If you like, a much simpler version of the Linux kernel, Kitten OS[1], is used to run at Sandia and in sequence with the Palacios academic virtual machine[2], which is what I'm working on. It may be worth looking at, although I still highly recommend that you look into actual production code.
[1] http://www.amazon.com/Computer-Systems-Programmers-Randal-Br...
[2] http://www.amazon.com/Design-Operating-System-Prentice-Softw...
Minix 3 is a whole different ball game though. It's an interesting kernel with some interesting ideas, but I still recommend Linux as it's both more popular and reflects the majority of UNIX design decisions today. Even the OSs with praise for microkernel design tend to incorporate a number of monolithic features.
EDIT: Sorry! I had no idea there was a Third Edition of OSDI out http://www.pearsonhighered.com/educator/academic/product/0,,.... However, I should mention that I meant those Linux books to be read in conjunction with the kernel code -- the book is still valid today despite the additions to the kernel in the past couple years.
Great resource: http://www.osdev.org/
Once you've built a UNIX clone, you have polluted your mind as an OS designer. I did so (as a standard college homework assignment) and it took me years to shake the crud out of my head.
As for the tutorial, it is very well written, but x86-32-centric and therefore worthless. There are nontrivial differences between x86-32 and x86-64 from the standpoint of an OS author who wants to make full use of the latter's capabilities.
As for the x86/x64 differences, they're by no means insurmountable once you have the concepts down, which is exactly what this tutorial provides.
OS development is already difficult enough, why make it harder? For every ten people that read and follow this and decide to make the Next Big OS (TM) following Unix concepts, you might get one that sees something bigger.
Learning has a direct cost in creativity. Once you have mired your brain in cached thoughts (http://lesswrong.com/lw/k5/cached_thoughts/) of Unix brokenness, it is very difficult to dislodge them. They won't feel like cached thoughts, or like anything special for that matter - just "the way you write operating systems."
Not one of the systems you mentioned deviates in any fundamental way from the mistakes of the original Unix. Not one. One of the reasons for this is that each was designed by people who have been steeped in Unix internals.
In addition, what Unix mistakes do you see in Singularity, for instance? It's drastically different from Unix-like kernels in effectively every way.
What OSes would you recommend that budding OS developers study, if not these? Amoeba is one of the few I can think of off the top of my head that might fit what you're looking for.
Try this one:
It is just a 512-byte bootblock demo, and yet it does something which no braindead Unix clone can: orthogonal persistence.
Any insight on how this compares to other technologies out there ? Forth ?
Honestly, I'm sort of baffled we're still arguing about this. I can't stand Unix-like kernels, I just believe that this particular tutorial is excellent at teaching the basic concepts required to put together any OS. If you find a tutorial of this sort of quality for any other design, submit it and I'll be certain to upvote it.
The world needs more OS designers, and tutorials of this sort lower the barrier to entry.
Edit: Also, if you're on Freenode by any chance, shoot me a PM (my nick is my username here). I always enjoy talking with someone who's as passionate about OS technology as I am.
As I said, you can't expect someone to create something original unless they understand the mistakes of the past. I started off implementing toy kernels with no structure, moved on to Unix-like kernels, and eventually ended up in a realm entirely different. Just because someone starts out copying Unix doesn't mean they're going to be blind to other competing designs or completely unique ones.
Expecting someone to build something new without any basis is just foolish.
Base it on your mental model of how your CPU's internals ought to be used, as inferred from the latter's manuals - like 1980s microcomputer users did.
Creativity exists.
There are plenty of tutorials on ASM coding, and plenty on low-level C coding. There are plenty of books about OS internals. The books and articles I've found before tend to stay firmly in the theoretical, in the abstract. The aim of this series is to show how the theory can be implemented. Then, the reader has the knowledge to (possibly) implement their own algorithms and know how they link to the CPU's internals.
As to your second point - do you think that the memory manager in this OS is optimal? Do you think that the linked-lists everywhere are optimal? Of course not. Everything was chosen for its simplicity - I failed in some areas I know, and I'm rewriting them as I speak (they're stalled at the moment due to a lack of time); the heap for example is a mess in that series. The new one is much easier to understand.
The series does not create an optimal OS for IA32. So your point about transitioning to amd64 to "make full use of the latter's capabilities" is moot - you have to do more research to optimise for IA32 anyway, let alone amd64!
(And let's not forget that all amd64 CPUs are backwards compatible with IA32).
Ok, hes not building a real OS, but rather a toy OS (presumably for the learning experience), so these things are not so much of an issue. Its actually a good article, IMHO, and I wish it had been available a few years back when I tinkered with writing my own toy kernel. The "Roll your own unix clone" title given here is pretty misleading though.
Yes, I'm the article's author. Yes, I'm biased.
Any ideas?
[1]http://www.ibm.com/developerworks/linux/library/l-gas-nasm.h...
Write a relatively simple C program that does something useful that you understand thoroughly, then generate intermediate assembly code from your C compiler, with optimization turned off.
That way you get a problem that you already know how to solve in a .s listing that you can inspect and modify to your hearts content.
Then try to optimize it, make it run quicker by rearranging stuff.
You'll learn lots that way and the barrier to entry is low.
This is a neat book. I read the first six chapters a while ago and it seemed interesting.