How to create an OS from scratch – tutorial | Hacker News Reader

How to create an OS from scratch – tutorial

github.com

377 pointszabana58 comments

I've seen a number of "build an OS" tutorials pop up recently, and that's awesome! Building an OS will definitely help you understand how to write better user-space programs, and it's fun.

One thing I wish is that more of them would feature UEFI instead of BIOS, or at least come with a warning, as Intel plans to drop support for legacy BIOS in 2020.

Going through UEFI is actually easier anyway, IMHO. ... Maybe I should stop complaining and write that tutorial myself instead. Hmm.

Or use the multiboot format (https://www.gnu.org/software/grub/manual/multiboot/multiboot...), which lets you offload the dirty work to GRUB. Writing bootloaders is kind of pointless unless you want to learn about them specifically.

As opposed to writing an OS? I'm pretty sure the target audience for these tutorials are people interested in learning how these things work, not people who know and are writing The Next Big Thing

The common wisdom with OSdev hobbyists is "if you want to write an OS, don't write a bootloader".
Boot protocols are very uninspiring, arcane knowledge and understanding it does not bring any advantages (unless you want to write bootloaders). Especially dealing with legacy BIOS boot sequence, which is essentially a backwards compatibility emulation layer implemented in the firmware.
It's not like home computers of the 1980s where you'd look at the processor data sheet how the CPU actually boots (which address the program counter starts from, what's the initial machine state, etc).
By far the easiest way to get started in x86 bare metal programming is using the Multiboot header, which is supported by bootloaders (like Grub) and emulators (like QEMU). And you can use GDB for debugging.
Using UEFI is more involved to get started. The initial setup is more involved (you need UEFI headers and libs, etc), the build process is more complex (UEFI images are "Windows" PE executables) and attaching a debugger is not as easy as with a Multiboot-ELF image. UEFI "boot services" provide some nice things that may be helpful to get memory maps and CPU hierarchy so you don't need to go through the hassle of parsing EBDA and MP and ACPI tables.
The legacy BIOS boot protocol is "easy" to get started with (nasm -f bin mykernel.asm), but you have to deal with arcane stuff (like floppy drive BIOS routines, A20 lines), you don't get a debugger. This is by far the worst way to get started. It's not really "doing it from scratch", instead of using a proper bootloader you need to deal with a really bad bootloader implemented in your PC firmware.

Have you got any reference material on how to get started using the multiboot/qemu/gdb route? I've started (and failed) the BIOS-bootloader route a fair few times now, But I'm not really interested in the boot process itself in the first place.

You're allowed to limit how much crap from the 1980s you allow into your life. If you want to learn about the Good Stuff (schedulers, filesystems, TCP algorithms) as soon as possible, jumping a bit and leaving the completely arbitrary bootloader stuff to some pre-written system is completely reasonable.

Is there any downside to using the multiboot format for your OS anyway, bootstrapping off GRUB to get started, and then if you really want to learn how everything works, writing a replacement for GRUB? It looks like multiboot is just a protocol that lets you plug in to a bunch of industry-standard software; it can still be quite educational (and easier) to go rewrite both ends of that protocol independently.

No, there are no disadvantages to using the Multiboot protocol, but there are numerous advantages.
The biggest advantage is that your kernel images are a bootable, debuggable ELF binary.
The emphasis being on "debuggable", you can easily plug in GDB to QEMU and use a proper debugger with your bare metal projects. Using GDB is much better than using the built-in debugging facilities of QEMU or Bochs. You can get the source view, set breakpoints and do everything you normally do in a debugger. In QEMU/Bochs built-in debuggers you have no source view and can only inspect memory and registers - not variables with symbolic names.
Here's some examples and documentation on building a debuggable ELF image for bare metal project, which I contributed to someone else's project:
https://github.com/Overv/MineAssemble/commit/56e317a28d069f4...

One thing I wish is that more of them would feature UEFI instead of BIOS, or at least come with a warning, as Intel plans to drop support for legacy BIOS in 2020.
The more people keep using BIOS, the more likely it will stay around. IMHO UEFI is a bloated mess compared to BIOS, so I very much hope the latter will continue to live.

I have my own ideas on why UEFI sucks, mainly around utf-16 and such, but I wouldn't really consider it bloated compared with some BIOS implementations. What is your objection to it?

Could you say why the encoding format is an issue for you? Thanks.

If you are writing an OS, I highly recommend the Pure64 bootloader from Return Infinity. It's much more straightforward to use compared to GRUB which imposes a lot of conventions on how you structure things.
httpss://github.com/ReturnInfinity/Pure64

Working link: https://github.com/ReturnInfinity/Pure64

Do you know of any tutorials (like OP) that are based off of this?

Upvoted to encourage you to write that tutorial. Please feel free to harvest my profile email address for any mailing list you might have for it.

AWS and as far as I am aware Azure don't support UEFI so hard to see BIOS going away until that is done.

“Once you pass college, excessive theory is worse than no theory because it makes things seem more difficult than they really are.”
About every third minute I’ve changed my mind about this quote. I think that means it’s a good quote.

As someone from a computer science background who's recently been getting into electronics in a really big way, this quote really struck a chord with me.

Writing an OS is a fun exercise and it makes you appreciate all the effort that goes into making something like Linux usable and relatively robust and performant. It's also interesting to look at how Linux (or other open source OS) implements some of these features once you've tried implementing them on your own. You'll find that much of Linux is understandable once you know what problem the code in question is trying to solve.

Hi, author here. I’m humbled by the kind words. This is a project I started when I was in between jobs and unfortunately it’s kind of stalled because I had difficulties implementing the file system. Really want to get back to it some time.
Anyway, feel free to leave your comments, I appreciate them a lot.

Just shot you an email to ask you to do that! Thanks for this resource. Good work :)

Glad you liked it! Thanks for the kind words

Thank you for your work. It's really enjoyable and inspiring.

These kinds of posts are the reason I come to HN hoping to find. Amazing work! I'm going to come back and read some chapters over the weekend.

I'm currently traversing through Linux From Scratch, http://www.linuxfromscratch.org/lfs/. Not necessarily because I wanted to build a Linux OS but because it was the best guide I could find. But at a first glance this looks promising to me, so I'd definitely check it out and try out a few "chapters" when time comes.

I legit thought they wrote an OS in the scratch programming language for a second lol

Can anyone recommend good resources on writing a compiler from scratch ?

Stanford CS143 - Compilers [1] and [2] MOOC: This self-paced course will discuss the major ideas used today in the implementation of programming language compilers, including lexical analysis, parsing, syntax-directed translation, abstract syntax trees, types and type checking, intermediate languages, dataflow analysis, program optimization, code generation, and runtime systems. As a result, you will learn how a program written in a high-level language designed for humans is systematically translated into a program written in low-level assembly more suited to machines. Along the way we will also touch on how programming languages are designed, programming language semantics, and why there are so many different kinds of programming languages.
MIT 6.035 - Computer Language Engineering [3]: This course analyzes issues associated with the implementation of higher-level programming languages. Topics covered include: fundamental concepts, functions, and structures of compilers, the interaction of theory and practice, and using tools in building software. The course includes a multi-person project on compiler design and implementation.
[1] https://lagunita.stanford.edu/courses/Engineering/Compilers/...
[2] http://web.stanford.edu/class/cs143/
[3] https://ocw.mit.edu/courses/electrical-engineering-and-compu...

Wirth's book, "Compiler Construction"
https://www.inf.ethz.ch/personal/wirth/
On the way you can also grab "Project Oberon" from there, both 1992 and 2013 editions, to see how to write an OS from scratch, including an FPGA board, in a GC enabled systems programming language.
There are also the tiger books, in C, Java and ML variants.
https://www.cs.princeton.edu/~appel/modern/

Destroy All Software has (as usual) an excellent screencast on it: https://www.destroyallsoftware.com/screencasts/catalog/a-com...

I think the best first book is: James E. Hendrix, The Small-C Handbook, Reston 1984, ISBN 0-8359-7012-4
He explains every detail, goes through every routine, and you can see exactly what the compiler does. It's 'Small-C' so it doesn't have any fancy features, like floats or structs. But the basic machinery is there, and one can add features as desired. It does compile itself. The compiler outputs x86 assembly, which must be assembled and linked with standard programs.
The original Small-C was done by Ron Cain, and produced 8080 code. I like Mr. Hendrix' better because I have lots of x86 around (I specifically wanted it for my HP-200 LX palmtop, which I still have and works great!); and because the book is fantastic.
It's fully understandable in a small quantity of time. And, the back end is re-targetable, so you could, say, output to ARM or RISC-V or whatever.
I spoke with Mr. Hendrix some years ago, maybe 10 or so, and asked if he would consider putting the compiler into the public domain. He did.

Thanks for this suggestion.
In case anyone else is interested, there is a freely available ISO of the the book available from Dr Dobbs Journal. From the README in the ISO:
>"Welcome to Dr. Dobb's Small-C Resource CD-ROM, the definitive collection of Small-C related information and source code. This CD-ROM includes the full text to James Hendrix's book "A Small-C Compiler: Language, Usage, Theory, and Design," selected articles on Small-C from "Dr. Dobb's Journal" magazine, and Small-C implementations for a number of processor platforms, including the 8080, Z80, 6502, and others."
It's available here:
http://www.drdobbs.com/developer-network-small-c-compiler-bo...

There are already a lot of suggestions here, I've added to my always increasing reading list this one[1] because, being based on Scheme (Racket to be exact), I'm curious on how a Lisp dialect compiler works internally.
[1] https://github.com/IUCompilerCourse/Essentials-of-Compilatio...

Here's a great description of compiling from a largish subset of Scheme all the way to x86 assembly language: http://schemeworkshop.org/2006/11-ghuloum.pdf

The author is mentioned it that book in the introduction chapter and in the acknowledgments.
Thanks, I'll put in the pile of things to read.

https://gist.github.com/soupi/73ab7dfd4d65fc68b62868d56b720b...

https://compilers.iecc.com/crenshaw/

Crafting interpreters, also the dragon book.

I'll be the guy to mention Forth as it is the only system I know of that a single person can fully understand and build up a fully functional and REPL interactive system using just assembly and a microcontroller. Most Forthers are doing embedded work though and don't need everything that comes with an OS.

Does any one have other resources I can look into as well. I’m still a relatively new programmer but I really want to get a deeper understanding of how OSes work. I’m assuming I need to use either C/C++ or Rust? I only Python and am currently learning C++.

Stanford CS140e - An Experimental Course on Operating Systems [1]: students implement a simple, clean operating system (virtual memory, processes, file system) on a Raspberry Pi 3 in the Rust programming language and use the result to run a variety of devices
MIT 6.828 - Operating System Engineering [2]: This course studies fundamental design and implementation ideas in the engineering of operating systems. Lectures are based on a study of UNIX and research papers. Topics include virtual memory, threads, context switches, kernels, interrupts, system calls, interprocess communication, coordination, and the interaction between software and hardware. Individual laboratory assignments involve implementation of a small operating system in C, with some x86 assembly
[1] https://web.stanford.edu/class/cs140e/about/
[2] https://pdos.csail.mit.edu/6.828/2018/schedule.html

Thanks for these links. Very useful indeed. I've been looking for something to get me into C and Rust, so I can learn the lower level stuff, and get me away from higher level web technologies (which I play with all day, everyday.)

You are welcome.
Please note that the MIT 6.828 is really tough but will turn you into a bad-ass C hacker.
It relies on MIT 6.004 - Computation Structures [1] (has edx mooc [2,3,4]) and MIT 6.033 - Computer Systems Engineering [5] as pre-requisites given that they teach you the fundamentals of computer architecture (digital abstraction, gates, circuits, processors, assembler, memory, cashes, pipelining, virtual memory, interrupts/io, buses, what have you) and system design (basically, managing complexity); they are not programming courses, but I recommend them if you have the time and interest, they are an awesome and principled way of learning the core concepts that you really need to have down before dabbling with OSs.
[1] https://ocw.mit.edu/courses/electrical-engineering-and-compu...
[2] https://courses.edx.org/courses/course-v1:MITx+6.004.1x_3+3T...
[3] https://courses.edx.org/courses/course-v1:MITx+6.004.2x+3T20...
[4] https://courses.edx.org/courses/course-v1:MITx+6.004.3x+2T20...
[5] https://ocw.mit.edu/courses/electrical-engineering-and-compu...

Thank you so much for these resources.

XINU OS worked for me. You can work with a beaglebone or a raspberry pi. There is a nice book for explaining every piece of the OS (IO, scheduler, memory management, etc.) and the source code is all there for you to tweak, browse, read or do whatever.
HN discussion here: https://news.ycombinator.com/item?id=10643757

Is there an accompanying book for the Xinu on the Pi now then? I looked a while back and I only saw it for the Beaglebone Black.

I've been wanting to do this but with a Pi or any SBC board with Linux from Scratch, but not sure how I would even approach LFS on a Pi.

I'm in extremely early stages of writing up some documentation for how to build a Linux distro for a Raspberry Pi 3, gauging interest and figuring out pain points. Shoot me an email [my username]@gmail if you've got ideas or requests!

That sounds interesting. I think your guide would be extra helpful if it focused on building a distro that has as much of the system loaded into memory as possible to minimize amount of reads from the SD card, and which is configured with a read-only file system and never even tries to write to the SD card. That would be very useful. SD card corruption is a pain!

Maybe we can somehow refer to each other? I wrote a guide for a custom board I made running Linux, but it's very low level up to tge stage of getting the kernel running with a minimal user land.
I would be happy to refer to you at the end of mine: brainyv2.hak8or.com

What do you mean by building a distro? Have you looked at Yocto/poky/angstrom? It's kind of a distro-generator.

Might be worth doing the CLFS (Cross Linux From Scratch)[1] path since it'll be a LOT faster to compile things like the kernel and gcc on your deskop computer.
I did this and wrote a bunch of scripts to automate the process. CLFS uses busybox and DropBear to keep things really small (~42MB binary for the entire OS, including the Raspberry Pi blobs) - but you'll learn enough to be able to swap out components of your choosing.
[1] http://clfs.org/view/clfs-embedded/arm/

I'm not sure why you'd expect the approach for RPi needing to be significantly different from that of any other computer. Bootloader would be the biggest difference I foresee, but that is fairly small part of the whole LFS.

I've not done LFS, but have worked a bit with Linux on arm devices, and I would avoid choosing an arm device for this purpose because the state of upstream kernel support for many devices that arm manufacturers use is, well, shit. You'll almost certainly have to use some old, unmaintained manufacturer kernel to get all of the devices working (e.g. graphics).
Edit: It actually doesn't seem that bad for some(?) rpi devices: https://elinux.org/RPi_Upstreaming
Not sure if that list includes all devices found on even the newer ones too.

My problems with LFS are:
1. The documentation is always in flux. You may get lucky and it's accurate, or you may not, and spend hours looking for tarballs that no longer exist online, or worse find them and they are no longer compatible with LFS - They don't host the required source archives along with the LFS book, and a lot of links to parts you need end up going nowhere.
2. As a step-by-step it does do what it says on the tin, but as a learning tool, there's a lot missing. When I was less knowledgeable about Linux I found myself following the LFS book with little to no idea about what I was actually doing and why. It would say things like 'chroot in the build folder' without actually explaining why, and what chroot actually did.
3. The whole thing can fail at any point, leaving you with a broken build, that more often than not, you'd have to wipe and start over. As most of the steps involve compiling, that's a lot of lost hours.

I'm fine with older kernels / older devices. My goal is to see how it all comes together and learn more about the internals. I like SBC's because they're usually cheap and can be replaced quickly and easily enough.

Older devices is fine, but you can start to run into trouble with older kernels. For example, newer glibc versions are notorious for requiring newer kernels [0].
So then you're stuck with an old glibc for your kernel. Not all applications (especially newer ones) will successfully link against that, so then you're looking for replacements or older versions that do (if there are any). What about other libc implementations, like musl? There are differences between musl and glibc[1] that will drive you nuts trying to find things that will link in musl that were originally written to link against glibc.
0. https://mirrors.edge.kernel.org/pub/software/libs/glibc/hjl/...
1. https://wiki.musl-libc.org/functional-differences-from-glibc...

Does anyone else have a good list of similar tutorials? I'm highly interested in this type of implementation from scratch approach.

Reply on news.ycombinator.com