Bootstrap yourself to write an OS from scratch | Hacker News Reader

Bootstrap yourself to write an OS from scratch

github.com

379 pointshappy-go-lucky53 comments

I love this :D Very much in the spirit of my own Computer Graphics from scratch [0]

On an unrelated topic, I wonder whether this popping up in the front page is indirectly related to this comment [1] I posted here yesterday. Many, many times I read interesting comments on HN posts, forget about the topic, and a day or two later, another closely related post appears on the front page. I suppose these comments generate follow-up research that ends up in other interesting articles being discovered and shared. Is it just me? Have you observed this as well? Is HN actually becoming some sort of hacker collective consciousness?

[0] http://www.gabrielgambetta.com/computer-graphics-from-scratc...

[1] https://news.ycombinator.com/item?id=16460633

Ha! I was just thinking about this ‘HN collective consciousness’ the other day. Not sure if I’m just more aware of it now or if it is actually occurring more frequently.

Lets hope peeps are concious not just intelligent.

> Have you observed this as well?

Many, many times :)

I consume hacker news in what I believe to be a not so normal way. I load the rss feed into feedly and scroll through every single thing posted. This suits me as I don't care what's popular. I mostly like things that aren't too popular.

I notice what you have noticed often. There are waves of a certain theme that clearly show other people posting things after researching something that they first saw here.

So one use I have for HN is that I upvote submissions that are things that I want to come back to and read later. I know bookmarks are kind of made for this, but I've never really gotten into the bookmark thing. (For that matter, I also rarely use the speed dial on my phone..) That's probably because I change computers/browsers regularly, and having lost all my bookmarks years and years ago, I stopped using them.

It also bothers me to treat upvoting "comments" this way.. I want it to stay under the "upvoted submissions" link on my profile. So I googled "computer graphics from scratch site:news.ycombinator.com" and upvoted the first submission that linked to your page.

I guess what I'm getting at is that method somehow feels wrong to me. Anyone have suggestions for "read-it-later" type bookmarks?

I find if I don’t actively read something then it’ll never happen. I use (not well) safari tabs this way on my iPhone, as it allows me to create an infinite number of tabs that are backed up and accessible from other devices.

Use a blog-this bookmarklet. It inserts the highlighted text into a fresh post with the page title linked to the page.

I've been blogging for wel over a decade for the audience of 1. He is perfectly in tune with the author and immensely enjoys everything I post. Sometimes the audience grows by 100% for a short while but after clicking around the categories and the insane amount of hierachic tags he never visits more than 5 citations.

It is highly amusing when he suggests to make the website even better. As if that is possible.

When google droped it I explained that I make the website and they may index it or chose not to. Their ideas how to improve the site and their attempt to force me was like they live on a different planet. They use to do organic ranking? I thought it facinating how they thought they could do that without accounting for my interests.

Point being: you want one of these websites just to have people (consumers) hate on it.

Do try to keep the personal notes as if addessing an audience. When you read those 10 years from now you say out loud: woah this guy is clearly a genius. By that time your 20k "bookmarks" (from now on called blog posts) will have a search engine allowing you to swiftly deeplink buddies into your mind so that they can have conversations under it that become similar treasures the decade after that.

No link to my blog here :)

I use HN favorites to save things for later, so I have a separate list from the things I upvote.

>That's probably because I change computers/browsers regularly, and having lost all my bookmarks years and years ago, I stopped using them.

Something similar happened to me, so I started storing my bookmarks under revision control. I don't add to them too often, so committing & pushing is a process that reminds me to not bookmark things unless they're _really_ interesting.

https://github.com/skx/bookmarks.public

cool. I was looking to build something like this for a while. But never give serious thoughts. I am inspired again now. Along with links/bookmarks, I really want a support for words also. Because many times, it is just easy to save words and later query using google e.g. Radix Sort

The term is "group mind", not "collective consciousness" <g-d-r>

I've heard 'hivemind' more than 'group mind'.

This looks fantastic as well! I'm a computer graphicist as well, and I always enjoy seeing gentle yet thorough introductions to the topic.

Is there something like this for 2D instead of 3D? I see there's a little bit at the beginning of Part II here.

An absolutely phenomenal book I'd recommend on writing your own computer graphics from scratch is 'Tricks of the 3D Game Programming Gurus' [1] by Andre LaMothe. It was published in 2003, but as it's about how technology works and not using whatever the latest graphics API is - it's timeless. It goes through everything in immense, yet still very accessible, detail. You start from nothing and at the end have a fully lit, shadowed, spatially partitioned, animated, textured, 3D game - using a software renderer that you write from scratch.

[1] - https://www.amazon.com/Tricks-Programming-Gurus-Advanced-Gra...

I'd go with these two instead:

Michael Abrash’s Graphics Programming Black Book http://www.jagregory.com/abrash-black-book/

Computer Graphics: Principles and Practice: Principles and Practices https://www.amazon.co.uk/Computer-Graphics-Principles-Practi...

Baader-Meinhof phenomenon..

U r special and unique. Just like everyone else...

At a glance, this looks to be the most gloriously valuable thing I've seen in eons. Due to the software industry's resounding success at canceling out the resounding successes of the hardware industry, causing veritable supercomputers to groan under the weight of such complex tasks as displaying a character in a timely manner after I type it, I plan to spend my retirement years reinventing the whole stack from scratch, Terry A. Davis style. It's my way of seeking justice for all the times I had to go get a coffee while my OS renamed a folder or the browser struggled to return to usable state after I foolishly risked scrolling the page. I usually conclude that the only sane way to do this would be to build on top of a bare-bones linux kernel, but perhaps something in this book will persuade me to set even less realistic goals. At any rate, looks like a fantastic resource!

See here for other people with the same idea (and others): https://groups.google.com/forum/#!forum/idealos-design

bit sad in the era of 64 bit computing still every resource is at most about protected mode. osdev wiki has some info about long mode, but it's all fragmented. Been working now for ages to try to decipher amd and intel manuals about that :'). Still ,nice this page, always good to inspire people to write their own system softwares because i think it's to much 'magic' still where as it's doable even for 1 person to write an OS that is dedicated to for example a single task runnign in user mode. writing a multi-user / multi-purpose OS is maybe more work, but in those OSs a lot of problems exist today which is causing a need for people (imho) to step back and look at if this is really the way we want to go with computing... i don't see why a webserver or something with a single purpose like that needs to be on a multi-purpose OS. It just increases complexity and overhead and people have a world of trouble configuring an OS to do just 1 task good, have to stip our 90% of what comes with a basic install to reduce attack surface and get resource usage under control. I hope we will see more dedicated purpose OSes come up which handle 64 bit, and perhaps someday some resources for programming specifically that. I'm thinking of sharing some of my knowledge on the area sometime, but i dont know how much interest there is and seeing as i'm a bad writer and still pretty shit coder it will probarbly take me too much time now compared to how much use it would be (if i do it, compared to some veteran in the field!).

> bit sad in the era of 64 bit computing still every resource is at most about protected mode.

I've an interest in writing a hobbyist OS, and I completely agree. Also, (almost) everything focuses on BIOS, rather than UEFI. And again, (almost) everything focuses on calling things from C, with a minimal amount of assembly, rather than doing everything from assembly.

I'd like to see a resource which walks through an honest-to-goodness minimal UEFI application written in assembly. At the moment, I think that I'll need to write it myself …

On the doing everything from Assembly.

https://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/i...

I do a lot with assembly. most of the code that launches long mode is easier / essential to be in assembly. I personally opted for using GRUB, but now i have a good idea on what that does for me i want to go back to writing a bootloader for my OS too. (done some basic ones, but didn't really know what information i had to pass. multiboot info from grub contains this). I dont use UEFI, but i think there you will need to consult the EFI manuals and use the SDK for EFI to build a module. This module then takes over from the boot loader, so in essence you will need code in there that gathers a memory map/info, vbe and apm informations, ACPI pointer (rsdp) and some more informations.

efi is a firmware framework, so using an sdk here makes sense. otherwise you need to find out the offsets to the firmware functions yourself. BIOS uses interrupts for this but not sure how uefi does it sorry, i never tried!

I will say that once you land in kernel land C is really nice. it saves a lot of typing. But i started out using pure assembly as it made more sense typing these instructions based on the manuals, c can be a bit too obscure to translate too if you're starting out. (still have trouble :') sometimes ...) Also stuff like loading CR3 with new page table pointer etc. just isn't possible atall in C. (you can inline it with asm volatile (); but inline asm is tedious in itself...)

i'd reccomend using GRUB bootloader, then you can write a loader stub in there which moves the CPU into 64 bit mode and is in ELF format, so you can also compile the os itself in ELF format. I used to use binary blobs from nasm, but i had a world of trouble generating C code into binary blobs. The linking procedure to link a 32 bit loader into a 64 bit kernel is a bit tedious but once it's working you can have 1 script which only does some generation of .o files and links it all together instead of catting binary blobs together and using hardcoded offsets to load them into memory etc. (that was a fun adventure, but really caused me a lot of headache more than anything :D)

if you want to do your own bootloader, consider that you should really write an ELF parser. i dont like the idea of sticking with some file format that's made by other people, but if you want to do it in your own executable format then you need to figure out to write a compiler / assembler for that format which is a lot of work. hence the choice to make an elf parser will get you started easier. if you want to just use nasm -fbin to make everything binary blobs you need to figure out how you will load everything and create mappings in pages to contain data. hardcode a lot of things which the linker/loader usually does for you. i think this will be the next level :D

i can't imagine a better project for learning about how computers really works :D reccomened for anyone interested in computers!

It’s awesome in the most literal sense of the word that someone wrote a 300-page book on this topic and just put it online for anyone to read for free. I hope I can contribute at this level at some point.

Another good link for people looking for something like this: https://wiki.osdev.org/Main_Page

Also BrokenThorn entertainment has an EXCELLENT tutorial on writing an OS from scratch.

I started following a couple of OS-from-scratch guides a while back. There are a few that are very similar and will get you through the basics. I got things working up to the point of having a little OS that would boot up and take keyboard input and display it in a text terminal in various colors. This was all very straightforward and worked pretty well.

The next thing I wanted to work on was some type of process manager that would allow for multiple processes to run, each in a protected memory space. That's when it started to get pretty hairy and that little learning project bogged down. As I recall, my decision paralysis mostly stemmed from the fact that there were successive generations of ways to handle memory allocation and protection on the i386/x86_64 families, and the clear way that I should implement my little OS branched off into a lot of choices with ramifications that I was hesitating to commit to.

I'd love to go back and pick that project up where I left off. I'll muscle past that learning curve, but are there any good guides out there with more information on how modern 64-bit process management (including memory protection) is accomplished?

Seems analogous to the XV6 project: https://pdos.csail.mit.edu/6.828/2017/xv6.html

For many years, MIT had no operating systems course. In the fall of 2002, one was created to teach operating systems engineering.

That's surprising, since just about every CS program offers/requires an OS course. Does anyone know what took MIT so long?

Anyone know of any other resources about operating systems for beginners?

What do you mean? There's lots of C tutorials, but the Mac OS command line is bash.

Sorry about the confusion, I just read the preface:

> This book is will be especially beneficial to students who’ve just finished their first C/C++ course.

That's what I was looking for.

http://www.learn-c.org/ looks good, I've also got https://learncodethehardway.org/c/ but not everyone will like that style.

Thank you! I have a general grasp of C but these will help. I also have The C Programming Language by Brian Kernighan and Dennis Ritchie - to which I now have a exciting reason to get around to.

You may also find interesting the Beej's Guides. https://beej.us/guide/

+1, those guides are excellent!

I haven't read it but I've seen https://github.com/0xAX/linux-insides posted here a few times.

Also https://littleosbook.github.io/

Sweet, this will help (hopefully this will also help when I come around to taking a formal operating systems course). Sorry for the noob question but I should really migrate over to a Linux-based computer if I would like to work with operating systems extensively, right? If so, any recommendations in regard to computers? Right now for all my software needs I use Mac OS.

I haven't done much myself, but I think a lot of people working on hobby operating systems use emulators or virtual machines to make development easier. Qemu and VirtualBox are popular and both run on macOS.

If you want to hack on your main operating system, then yeah, you'll probably want to switch to Linux, but I wouldn't do that yet.

You're right, good call.

I think you should be able to get away with using macOS. Most tools you’ll need are for Unix systems anyway.

If you were on Windows, that would be a different story.

For an introduction to the theory, Nand2Tetris[0] starts you out with nothing more than an xor gate from which you construct every other logic gate, implement a CPU, then in part 2 you build an OS on top of that. Although that sounds intimidating, it's actually incredibly accessible. I went in knowing nothing about the topic, and was able to implement the CPU by the end of the course, which was a major confidence boost. Also I found the lessons learned from the course (basically, layer simple systems with clear interfaces together to get complex systems) extended well beyond just CPUs and OSes to software design in general.

That uses a very simple educational architecture designed for ease of learning, though. If you prefer a good introductory resource for real-world operating systems, I just finished Georgia Tech's Intro to OS graduate course[1] and found it very accessible as well, and now feel reasonably comfortable with the theory when reading about OS topics elsewhere.

[0] http://www.nand2tetris.org/

[1] https://www.udacity.com/course/introduction-to-operating-sys...

I haven't done the course, but I'm fairly sure that XOR isn't a universal gate, so you probably don't start out with "nothing more than an xor gate." Based on the name, I would guess you start out with only NAND, which is a universal gate.

Sorry, obviously yes, it starts with NAND. No idea where I got XOR from.

I did nand2tetris something like 7 years ago, it was a pretty fun introduction to digital logic. I remember hitting a point where I couldn't continue after designing the ALU due to either a paywall, something missing, or the Java software being broken. Not sure if it's been fixed, but I'm guessing running in an older JVM would have probably worked.

I did it last year through Coursera (it's what pushed me to apply for my current master's program) and got the certificate, so I was able to complete everything. I don't remember there being any special measures taken on my end to get it to work. Dunno what the issue was, if something was fixed in the intervening time, or what.

I would've done paging before task switching because TSS is such a broken way of doing things. Processes should live in their own separate address spaces.

Been following this project since it was in the rougher earlier days. Really glad to see it come together. Congrats!

How many of these large grand “tutorials” actually get completed?

I am still waiting to write my compiler in haskell [0] and to learn linux inside out [1]

[0] http://dev.stephendiehl.com/fun/

[1] https://github.com/0xAX/linux-insides

Wow, there is a lot of really good info in here, keep up the good work! Looking forward to part 3!

Is there a book like this but on building your own custom version of Android?

Yes. _Bootstrap yourself to write an OS from scratch_. As it says at the very top: "How to write an operating system from scratch by reading hardware datasheets. In the real world, it works like that. You won't be able to consult Google for a quick answer." To write an Android clone, you'll need to understand how cellphones work, which involves reading hardware datasheets (and possibly signing a few dozen or so NDAs to obtain the information).

Reply on news.ycombinator.com