Writing a simple operating system from scratch (2010) [pdf] | Hacker News Reader

Writing a simple operating system from scratch (2010) [pdf]

cs.bham.ac.uk

280 pointsjwdunne47 comments

If you're interested in this type of stuff I can personally recommend MIT's 6.828 course (Operating Systems Engineering) as an additional resource. Specifically, you'll want to take a look at xv6, a teaching OS implemented for the course (http://pdos.csail.mit.edu/6.828/2014/xv6.html). The OS is small enough to print as cross referenced code listing (http://pdos.csail.mit.edu/6.828/2014/xv6/xv6-rev8.pdf) and has a book that fully explains each piece (http://pdos.csail.mit.edu/6.828/2014/xv6/book-rev8.pdf). You are only 9,000 lines of code away from fully understanding basic operating systems concepts!

It's been a few years since I took the class but it was an absolute blast. The offering in 2011 has recorded lecture videos (http://pdos.csail.mit.edu/6.828/2011/schedule.html).

Thanks for posting. I hope this material is available for years to come.

Have you done any work in a related project after taking this course?

Yes I work professionally in the field.

Any advice for us OS wannabees? ;)

Sorry just saw this comment. Work your way through the 6.828 course because the presentation of concepts in that course is as fundamental and bite-size as it gets. Then when you get to real operating systems (which are much uglier than xv6), don't be intimidated by the complexity. For any difficult subject, there's always a point along the learning curve where things seem hopeless but if you push through you'll start to gain glimpses of understanding. These glimpses provide motivation which in turn lets you understand more... eventually process of learning becomes self sustaining.

No worries. Thanks for the pointers. The next time I find myself not doing a startup, I'm going to take that course!

The videos have been very horribly recorded though! Seem to have been recorded by a student.

Yes. It is astonishing how institutes with huge budgets put videos like these (it is quite common with such big names as MIT).

> Yes. It is astonishing how institutes with huge budgets put videos like these (it is quite common with such big names as MIT).

Yes if only MIT spent some money and made high quality videos available for free to millions of people...

https://www.edx.org/course-list/mitx/allsubjects/allcourses

Have you seen MIT OCW videos?

If it were altruism I'd applaud any effort. But their alailable lectures seem to be a bid to bring more students in to full enrollment. Poor video quality, as long as the awesomeness of the course itself can be seen, doesn't work against this goal.

If it were altruism they would soon enough be out of business and in the future content like this would no longer be made available.

By far one of the most satisfactory things (and also hair reducing things) I've had as a long term hobby is: O/S development. Been doing it seriously for just over a year now, it's slow because it is purely free time. Was working happily on the kernel but decided to go back and build a bootloader after I got to higher half kernel and was messing around with the GDT base address overflow hack. It's now turned into a multi-stage boot loader, because I appreciate the idea of your system have full control over your own partition, nothing else.

I enjoy going to http://wiki.osdev.org for a broad overview and then hitting the manuals for the details; sometimes that manual and note book session can take up to 3 days of my free time haha.

I especially love writing custom tools around the development process; I like to call it reducing reliance - I found that this really helps solidify concepts in my mind; although making the process even slower, I feel comfortable at every stage.

I highly recommend everyone to go through the book http://www.linuxfromscratch.org/ before, or as a complementary direction to, diving into any "operating system" resources mentioned on this page. (you could try this on a QEMU/VMWare/VirtualBox/KVM/whatever instance instead of risking your current hard-drive partition setup).

(alert: it's not really a book, it's a project: to compile and install linux without using any distro! the whole process could take somewhere from 1 day to 1 week, depending on the speed of your machine and your command-line skills.)

Because then you would discover (if you haven't already) that what all these people call "operating system" is essentially only a nucleus of the operating system (kernel) and that there is much more to an operating system than whatever is discussed in an OS class.

Also LFS is enlightening in ways I may not have expressed here. If you successfully go through the process, your first thought might be to roll your own linux distro!

I wonder how many dependencies would actually be needed for a self-hosting (i.e. able to re-compile itself) Linux-plus-shell system.

Some pointers: http://www.landley.net/ols/ols2007/tutorial.txt, found via https://encrypted.google.com/search?hl=en&q=linuxfromscratch...

Clarification: By "a self-hosting...system" I meant "the most minimal self-hosting system".

Better pointers: http://landley.net/aboriginal/about.html via more googling: https://encrypted.google.com/search?hl=en&q=minimal%20self%2...

It used to be, in the good ol' days, "about 4 floppies of kernel and drivers", and "12 floppies of baseutils" with your added "5 floppies for emacs" and "1 floppy for vim" .. but those days are long over.

Nowadays, the most useful 'self-building' OS/bin stack is debian with build-essentials.

Well not Linux, but all the BSDs are self hosting. Recommend NetBSD as you can cross build it from Linux it doesn't take too long.

> QEMU

Don't.

I developed a from-scratch operating system as part of my PhD thesis. We initially ran it on qemu due to its speed and simplicity, but the truth is that while qemu would run something developed on real hardware, it would also run a lot more. What it accepted was very, very loose and when you developed for qemu, you generally couldn't take it and run it on real hardware without days worth of debugging and trying to figure out where qemu did things just a little more loosely than actual hardware did.

We ended up using Bochs for that purpose, which is a lot slower but a lot more faithful (and unfortunately lacks a lot of secondary hardware that's important, but it will take you a long time in development before you hit that particular limitation).

Actually this need for a good debug tool of some kind is a key point. In the 80s I wrote an operating system for 6809, but I did have access to an ICE. I, and I'm sure many others, wanted to write something for that new 386 SX the second it came out. I thought Linus Torvalds had access to a 386 emulator at university, but I can't find a reference. I did find this, which describes how he debugged without one:

(from http://www.cs.cmu.edu/~awb/linux.history.html)

(he would have hated the machines I use these days, they take 6 minutes to boot)

"The worst part is starting off: after you have even a minimal system you can use printf etc, but moving to protected mode on a 386 isn't fun, especially if you at first don't know the architecture very well. It's distressingly easy to reboot the system at this stage: if the 386 notices something is wrong, it shuts down and reboots - you don't even get a chance to see what's wrong.

Printf() isn't very useful - a reboot also clears the screen, and anyway, you have to have access to video-mem, which might fail if your segments are incorrect etc. Don't even think about debuggers: no debugger I know of can follow a 386 into protected mode. A 386 emulator might do the job, or some heavy hardware, but that isn't usually feasible.

What I used was a simple killing-loop: I put in statements like

die: jmp die

at strategic places. If it locked up, you were ok, if it rebooted, you knew at least it happened before the die-loop. Alternatively, you might use the sound io ports for some sound-clues, but as I had no experience with PC hardware, I didn't even use that. I'm not saying this is the only way: I didn't start off to write a kernel, I just wanted to explore the 386 task-switching primitives etc, and that's how I started off (in about April-91).

After you have a minimal system up and can use the screen for output, it gets a bit easier, but that's when you have to enable interrupts. Bang, instant reboot, and back to the old way. All in all, it took about 2 months for me to get all the 386 things pretty well sorted out so that I no longer had to count on avoiding rebooting at once, and having the basic things set up (paging, timer-interrupt and a simple task-switcher to test out the segments etc)."

I partially disagree. Qemu, & in general simulators, can really speed up development but for the reason you cited, you need to periodically run your code on real hardware. The second suggestion is to minimize assembly language level code -- this will reduce the temptation to use esoteric h/w specific features that Qemu may not handle! In general micro-optimizations should be left for much later, if ever. Third, as soon as possible set things up to load your kernel over the network or a serial link (or JTAG). Particularly if you use a debugger, you will be running the same kernel over and over again to debug something. Fourth, if you plan to use a debugger like gdb, provide support for remote debugging. That is not a lot of code (basically support for communication, breakpoints, peek and poke).

For those that really want to build everything from scratch, including hardware, Niklaus Wirth has revised his Project Oberon book, including the Verilog files

http://www.inf.ethz.ch/personal/wirth/ProjectOberon/index.ht...

So you could have your own graphical workstation. :)

There's also the tried and true Lions' Commentary on UNIX 6th Edition with Source Code pdf floating around to understand what's going on in the workings of a timesharing OS.

Nickolai Zeldovich a current MIT professor wrote HiStar OS from scratch and it's source is still available online to view how a minimally trusted system could work http://www.scs.stanford.edu/histar/

Lions Commentary PDFs are pirated from the book, which is still in print (royalties to Prof. Lions' widow).

I'm inundated with OS resources at the moment. I hope that Coursera offers an OS course in the future that I can follow along.

Here are other resources that I'm tracking:

http://www.sal.ksu.edu/faculty/tim/ossg/index.html

http://joelgompert.com/OS/

http://www.csi.ucd.ie/staff/jcarthy/home/alp/alp-05.pdf

http://www.superfrink.net/athenaeum/

http://www.cs.utah.edu/flux/oskit/html/oskit-www.html

http://duartes.org/gustavo/blog/post/how-computers-boot-up/

http://duartes.org/gustavo/blog/post/kernel-boot-process/

http://www.ibm.com/developerworks/library/l-linuxboot/index....

http://www.osdever.net/bkerndev/Docs/title.htm

http://sourceforge.net/p/oszur11/code/ci/master/tree/

http://www.codeproject.com/Articles/737545/Writing-a-bit-dum...

http://freevideolectures.com/Course/2653/CSE-30341-Operating...

Although, osdev is what I'm following right now and have gotten as far as writing the bootloader. Next, I'm contemplating continuing there or following the Minix book.

Thank you for sharing this list of OS dev resources!

I would also recommend Andrew Tanenbaum's "Modern Operating Systems" or "Operating Systems: Design and Implementation" (the MINIX book), to anyone interested in OS dev. I read it when I was teenager (around 15), and I really really enjoyed it. Tanenbaums' books were one of the few truly addicting textbooks. As a side, "Structured Computer Organization" by Tanenbaum was also really great. It gives you a foundational understanding of how computers are architected that is indispensable if you're going to be doing OS dev.

Another: http://www.cse.unsw.edu.au/~cs9242/14/lectures/ UNSW's Advanced Operating Systems course for microkernels with focus on seL4

While you can find a lot of video lectures (complete courses) about OS dev, I'd really like to have some Database systems dev course from coursera or any other source.

This is so exciting! I never got to write my own boot sector in college, our lab exercises were kind of advanced and we had a lot of starting code.

That document inspired me to create a Github repo with only the code, split into micro lessons. Feel free to check it out, I'll update it at the same pace that I process the document and learn stuff myself https://github.com/cfenollosa/os-tutorial

I was sad that the disk driver section was unfinished. How does the disk driver for a spinning disk work? Does it take into account any physical properties of the needle?

I would assume that's the job of the disk's firmware, manage the physical disk and present a compliant interface to the OS (e.g. IDE, SCSI etc...)

I've heard certain types of schedulers can take the position of the disk head into account.

Been there, done that! Apart from this contributing to my grade, I still miss the point of the usability nowadays. I wouldn't say this gives you a real knowledge on how modern ooerating systems work. On the other side in the OS/C and C++ modules (the link is below) you can find good entry points for kernel programming, which has proven useful few times.

I would also look at the MS DOS source code that microsoft released earlier this year. Here's a link to MSDOS 2.0 source: http://www.computerhistory.org/atchm/microsoft-research-lice...

Lack of multitasking or virtual memory makes it not that interesting, IMHO.

Agreed, but I like the simplicity when starting out since there's just so much. I'm reading Peter Norton's Assembly programming book from '87 to keep things super simple.

(http://www.amazon.com/Peter-Nortons-Assembly-Language-Book/d...)

Do you have any recommendations?

This looks really interesting. Another course / book that I have used in the past which was great for getting to grips with the ideas of an entire system from scratch was http://www.nand2tetris.org/

Here is another example of an operating system written from scratch: http://www21.in.tum.de/~traytel/POTATOES/Home.htm

The page referenced says that POTATOES implements a broad range of operating system concepts and features, e.g. multitasking, paging, an extensive I/O subsystem as well as a custom file system, and runs on x86 hardware.

I found my operating system class very enlightening, although doing virtual memory was not fun... Implementing FAT 12 was an enjoyable challenge.

Though ultimately I have no plans to get into the kernel space for development.

Seems nice at first glance, but where the topics start to get a little less trivial (where, for example, the brokenthorn series on OSdev stop being relevant as well: the harddisk driver), the draft ends.

May also be useful:

http://www.cs.bham.ac.uk/~exr/lectures/opsys/13_14/lectures....

I wonder if there are any great resources for building your own OS to run in side VMWare Fusion or VirtualBox instead of running on hardware.

http://wiki.osdev.org is excellent. Most os tutorials use virtual box or qemu.

Posted upthread but it's important enough to repeat:

> qemu

Don't.

I developed a from-scratch operating system as part of my PhD thesis. We initially ran it on qemu due to its speed and simplicity, but the truth is that while qemu would run something developed for real hardware, it would also run a lot more. What it accepted was very, very loose and when you developed for qemu, you generally couldn't take it and run it on real hardware without days worth of debugging and trying to figure out where qemu did things just a little more loosely than actual hardware did.

Reply on news.ycombinator.com