Writing an operating system kernel from scratch | Hacker News Reader

Writing an operating system kernel from scratch

popovicu.com

350 pointsBogdanp69 comments

For those who want a "packaged" experience doing this same kind of work, there's "Operating System in 1000 Lines of Code" (https://operating-system-in-1000-lines.vercel.app/en/), I followed it a while ago in Zig (which meant translating C code snippets to Zig) and had a good time. I have the code and VODs here: https://github.com/kristoff-it/kristos/

What would be the level of Zig knowledge needed to do this exercise?

Or, asked differently, with no knowledge of Zig, what would be a realistic approach to trying this out? (Assuming an interest in learning Zig, with a background in C++)

I could easily understand stdlib code of zig after building some stuff for 2 days. Would be different for everyone but it is fairly easy

It's definitely possible to hit the ground running you have some knowledge of systems programming. I would say that the Zig version is much easier to understand than the C version provided by the book because the Zig one uses `packed structs` instead of bit operations in some places, everything else is roughly the same.

Submitted separately by the author:

https://news.ycombinator.com/item?id=45236479

Copying their comment to here:

I have redone the classical exercise of writing a tiny OS kernel with time sharing, which manages a couple of user threads. My goal was to experiment specifically on RISC-V + OpenSBI. Additionally, I wanted to explore Zig a little bit, so that was the language used instead of the traditional C, but it should be straightforward how to do the same experiment in either C or Rust.

It's definitely very rough around the edges, and it's more of an experiment and an intro for people who want to go through step 0 of learning OS kernel development and computer architecture. Nevertheless, I hope it is still a fun experimental thing to play with over the weekend!

The full walkthrough and the GitHub link are available at the link posted!

I will never not find this kind of project incredibly impressive. It’s interesting to think that Linux, after all, is really just the kernel — and yet getting that work done paved the way to getting an open source version of Unix installed on billions of machines. Great stuff!

It is equally valid to say that Stallman's starting to write a C compiler and Unix utilities (in 1984 whereas the Linux project started in late 1991) paved the way to getting an open source version of Unix installed on billions of machines.

I agree - there's a number of kernels that were "open source" and released at a similar time enough time to linux (e.g. 386BSD in '92) that I could see any of those winning the "community battle" and taking that space instead, but no real credible "development toolchain" equivalent until decades later.

Though I'm unsure how differing licenses might have affected this - I suspect that really early in it's development the "copyleft" nature of the GPL Linux didn't make as much of a difference, as from what I remember most commercial uses of Linux didn't come until it had already gained significant momentum.

The copyleft nature was essential to good driver support. It set it up such that for corporations making drivers the easiest path was to get the driver upstreamed. There was a bunch of hoops they could have gone through to avoid that (as many did, like Nvidia) but that became a sorta-default.

Copyleft encourages a collaborative relationship between entities because it makes trying to play it close to the chest with IP involve more legal effort (if it's possible at all).

Yes, I can see that stalling development as (at best) it turns into a pile of private forks rather than a cohesive project, but from what I remember that was already after Linux had "won" the "Open Source Kernel" race.

Commercial support for Linux was... Sparse... before the early 2000s.

> [Stallman/GNU] getting an open source version of Unix installed on billions of machines.

Agreed, funnily enough GNU tools/compilers also ended up getting installed on a lot of proprietary UNIXes because proprietary UNIX was mostly shit (in user space!). At least most of the ones I had the misfortune to have to work on.

I first came across GNU tools on NeXTSTEP, which wasn't too bad.

If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel whereas GNU's utilities and apps (e.g., Emacs) were immediately useful (i.e., without breaking any copyright law or violating any software license) to a large population, namely, anyone with an account on a proprietary Unix system, which explains why Stallman chose to start with the userland.

    > If Stallman had started with a kernel, there would be very few people who had the legal right to run any utilities or apps on the new kernel

That is really not true, one of the most important things when it comes to the GNU project and the whole Free Software movement is the ability to run _any_ program, be it non-free software or free software. This has been parroted for more than 40 years now ...

That's true in isolation, but is missing vital context. In the 1980s a user of proprietary Unix usually had no way to make the ls utility or a C compiler or a text editor run on an new Unix-like kernel. And even if he had the technical means to do it, he probably did not have the legal right, i.e., it would have been a violation of copyright law.

In the Unix world, there was no tradition of binary compatibility: you couldn't just take a binary compiled for NeXTSTEP and run it on Solaris or on Linux: you needed to recompile, which means you needed a C compiler and the source code for the program you want to run, and most users of Unix in the 1980s didn't have a practical way to get the source code (and if he did have access to the source code, using to to port the program to a new kernel was probably a copyright violation). Stallman could have tried to start a tradition of binary compatibility in the Unix world. That is one of the strategies he could have chosen for the GNU project: i.e., he could have picked a proprietary Unix, Solaris, say, and make sure the kernel of his GNU system could run all (or most) binaries that run on Solaris. But that strategy ran the risk that the owner of Solaris might have sued to stop this and the courts might have sided with owner, requiring the GNU project to shut down, pay the owner (Sun) a lot of money or start over with some other strategy.

(Replying to myself because it is too late to edit.)

I found a shorter reply to your comment, which I will now write:

>when it comes to the GNU project and the whole Free Software movement is the ability to run _any_ program, be it non-free software or free software.

Just because the GNU project allowed non-free software to run on GNU doesn't mean there existed in the 1980s a userland that would run on GNU if GNU had consisted of just a kernel (or just a kernel and a C library and a C compiler).

It's even more funny and amusing when you remember that in the initial release mail for Linux, Torvals said "just a hobby, won't be big and professional like gnu".

https://groups.google.com/g/comp.os.minix/c/dlNtH7RRrGA/m/Sw...

He's also referred to Linux as "the GNU Emacs of all terminal emulators".

> It's now the GNU Emacs of all terminal emulators.

> (Linus Torvalds, regarding the fact that Linux started off as a terminal emulator.)

http://neil.franklin.ch/Jokes_and_Fun/Linux_Quotes.html

That's the best reference I can find, but even if it's totally legit it doesn't make any sense to me.

Perhaps it's a reference to how Emacs does "everything" and some has also joked that Emacs is an OS with a text editor

Back then Emacs was considered what we call in modern software 'bloatware': resource hungry, big and full of feature creep.

BTW, in early versions of Slackware emacs was bundled in its own software category due to its huge size when selecting which packages to install on first boot.

Probably even had its own floppies too. Remember that install method?

Transcribed here for posterity:

> Hello everybody out there using minix - I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

> I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-)

> Linus (torv...@kruuna.helsinki.fi)

> PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

> I will never not find this kind of project incredibly impressive

I wouldn’t call it incredibly impressive. The path on how to write a minimal multi-tasking kernel has been beaten decades ago.

Writing a kernel that can boot and do a few things is ‘just’ a matter of being somewhat smart and have some perseverance. Doing it for RISC-V complicates things a bit compared to x86, but there, too, the information about initialising the hardware often is easily obtained (for example: https://wiki.osdev.org/RISC-V_Meaty_Skeleton_with_QEMU_virt_...; I wouldn’t know whether this project used that)

I think the author agrees, given (FTA) that they wrote “This is a redo of an exercise I did for my undergraduate course in operating systems”

It’s work, may be nice work, but I think everybody with a degree in software engineering could do this. Yes, the OS likely will have bugs, certainly will have rough edges, but getting something that can multi-process, with processes shielded from each other by a MMU isn’t hard anymore.

> I think everybody with a degree in software engineering could do this

Ideally this would be true, but it hasn't been my experience at all. At least with American graduates, I can't speak to other countries.

My CS undergrad in the UK had us write an ARM kernel with scheduling and IPC, though didn’t require we use the MMU.

Is there a book one can read to learn how to create one?

https://en.wikipedia.org/wiki/Operating_Systems:_Design_and_...

That is an excellent recommendation. For Operating Systems anything Andy Tanenbaum did is world class.

This made me look up what he has been up to, there is a 2023 edition of "Modern Operating Systems" which covers cloud virtualization and Android along with everything that came before, hm, tempting.

We really need a Tech Writer Hall of Fame. W. Richard Stevens, Andrew Tanenbaum, P.J. Plauger. Others?

Kernighan!

And how can we leave out the OG of tech writers: Donald Knuth. He got a bit distracted by developing TeX but he got a well deserved Turing award for the series.

I'd suggest https://os.phil-opp.com/ instead of a book from long ago. There is no reason to drag yourself through C anymore.

And if you do want the more historical content, https://www.projectoberon.net/

To you maybe. The subset of population that is even interested, smart, persevering to do this is extremely tiny.

I am 99.99% sure that less than 20% of Australian graduates could do this, and honestly I wouldn't be surprised to hear that the actual answer is <1%.

I was studying at Monash, which is considered a solid university here, and holy moly are the standards low. I had classmates in the second year of my machine learning postgrad asking me things like "What is machine learning?", and they all graduated and found jobs anyway.

What did they find jobs in? Australia has like one tech company.

Which is odd since their universities have built two of the most interesting CS projects I can think of (Mercury and L4). And WWWJDIC I suppose.

Folks end up at all sorts of places. Like I mentioned above, the banks hoover up a lot of graduates. There are a lot of smaller local companies doing web stuff. The consulting companies all have a presence here (KPMG, Accenture, Fujitsu Consulting, etc).

Atlassian Stripe, Zendesk,

Google and Amazon both have offices in Australia, and a bunch of banks / media companies / startups / SMEs employ machine learning specialists.

Others have pointed out a few "real" tech companies, but the economy is also surprisingly large. There are a ton of little data science jobs available on the market, though they're mostly on initiatives that will never plausibly work.

I'd agree with this. I did a double degree in Comp Sci/Comp Sys Eng at RMIT (1998-2002) and even from that era I would say that's largely true. Out of the people who did my course (and those I knew from other degrees like Comp Sys Eng/Business) very few are still doing deep technical programming for a career and/or hobby programming on the side on deep technical non-web things. The rest are mostly working for places like consulting companies, banks, big data, Telstra, etc in management roles like project manager, scrum master, solutions architect, change management. A lot of folks I think were just not that interested in stuff like writing an OS, how does virtual memory work, how does the hardware work, etc so they gravitated out of those software development roles into management roles. Nothing wrong with that, but I just think not everyone is interested in or capable of writing an OS!

Hmmm here at least in my uni in Argentina, we have an obligatory computer architecture class and we implement a protected-mode x86 (32 bits) kernel with interrupts, paging, etc. We obviously get some guidance but it's not that hard if you read the docs, nowadays you don't even need to go deep into the Intel manuals probably since osdev wiki has a lot of content.

Besides it's much easier nowadays, if something's wrong you can feed ChatGPT your GDT definition for example and find out if you misplaced a value, which used to be a PITA to debug.

As the parent comment says, I think the path to a booting usermode kernel has long been beaten, it's not trivial but it's not that hard either, I believe the impressive stuff is once you get out of "tutorial land" and implement a network stack, or a UI stack, etc.

Zig really is amenable to OS development. And so is RISC-V. I started this same exercise except with x86 first. I quickly got annoyed with all the legacy boilerplate required. RISC-v doesn’t have any of that. It’s so much faster to get up and running. Here’s my bad zig: https://github.com/Fingel/aeros-v

> with x86 first. I quickly got annoyed with all the legacy boilerplate required.

IMHO, if you use a reasonable bootloader, you don't have too much boilerplate. Multiboot loaders do leave you in real mode, and most people want to be in protected mode, so you have to set up some tables and do a jump, but that's not that much boilerplate. There's a bit more stuff if you disable legacy interrupt controllers (which you probably want to) but it seems to me being able to maybe run on a regular pc is worth it (caveats about console interfaces apply... my hobby OS needs bios boot and uses some vga things that I found aren't well supported... serial console is easier, but lots of computers don't have a serial port either)

You still have to set up the GDT, TSS, all that other legacy stuff that isn't actually used but it's still there. And if you set it up incorrectly, your punished for it with triple faults and the like, even if nobody even uses anything like the GDT or TSS these days.

that is a pain. oswiki is a good resource, and there are also a huge number of examples. but it still takes a day. context save and restore and interrupts is another speed bump. but its no different than any other development banging your head against the wall exercise, and its pretty rewarding to have smoothed all that over and go to town on bespoke scheduler design.

I dunno, I've done it myself, and I would say that it's qualitatively different than other "banging your head against the wall" software development. At least it's an order of magnitude more painful, due to lack of integrated tooling, questionable documentation, and a more delayed feedback loop.

Single-threaded kernel bringup is easier than most distributed systems though.

Funnily enough, when I was writing one in Zig for x86 I actually was the one who found a bug in (I think) 0.11/0.12 where packed structs broke when you assigned to their fields (and I also found that the `.interrupt` calling convention was a todo item in that version). GDT/TSS is always, always one of the most annoying parts of an x86 OS for me, because all of it is so tedious and unnecessary, and I wish Intel would just get rid of IA32 mode already, or add some kind of bit in CR0 or CR4 which caused all that extraneous setup of legacy structures to just cease to be. RISC-V is so, so much more pleasant to get started with. Limine helps regardless of which architecture your getting going with; it's pretty much my de facto bootloader at this point.

Setting up the GDT is exactly the point where I got annoyed enough to see what risc-v was all about.

Yep, agreed. Don't get me wrong, the Intel/AMD docs are amazing. But even getting the hardware to a usable state (where I define usable as "something that can enumerate PCI devices on the system, even without PCIe") is an absolute pain that really just shouldn't exist. I know that RISC-V ahs it's own tough spots but... IMO it's a lot faster to get up and running with because it doesn't have 10000 execution modes with tons of different requirements you need to do for each mode, and on and on and on and on. (Yes, I'm exaggerating a bit.)

I’d like to have a go at this. What are you using to run your risc-v kernel? Just Qemu or can you recommend any real hardware?

Just Qemu. I used this as a guide: https://operating-system-in-1000-lines.vercel.app/en/ It's C so translating to Zig is pretty fun.

Naturally, it is basically revisiting the safety of Object Pascal and Modula-2, and repacking it into a C syntax.

C was never special, beyond being widely available thanks to UNIX original licensing.

nice!

i love how accessible riscv isa is, the docs are amazing, there are tons of examples, and tons of emulators, the uncompressed machine code is quite readable.

i am writing a book for my daughter and am also working on a small os with forth and timesharing https://punkx.org/projekt0/book/part1/os.html and i dont think i could've done it on x86, learning both forth and riscv assembly on the way, its super fun.

honestly if you ever wanted to make a toy os from scratch there is no better time to start than now (besides maybe 1980), get a cheap riscv like rp2350 and just upload the relevant sections of the docs to claude and it will help you if you get stuck.

I just checked the prices for my country and I’m surprised how cheap this is. I’m 90% sure I’m buying one just to play around with.

and the datasheet is amazing https://datasheets.raspberrypi.com/rp2350/rp2350-datasheet.p...

there is an issue that language models have seen enormous amounts of rp2040 code and docs, and it they get properly confused a lot, so you should just pdf print relevant pages and use them in-context.

Always a fun and interesting thing to try.

Also go ahead and try hard things like your own cryptography. The advice “don’t roll your own crypto” just means don’t DEPLOY something that is not battle tested, because it is very easy to make BIG mistakes in that area. Always feel free to experiment and explore.

We need more operating systems, more choice.

> El acceso a la presente dirección IP ha sido bloqueado en cumplimiento de lo dispuesto en la Sentencia de 18 de diciembre de 2024, dictada por el Juzgado de lo Mercantil nº 6 de Barcelona en el marco del procedimiento ordinario (Materia mercantil art. 249.1.4)-1005/2024-H instado por la Liga Nacional de Fútbol Profesional y por Telefónica Audiovisual Digital, S.L.U. > https://www.laliga.com/noticias/nota-informativa-en-relacion...

Of FFS...

one thing is using http but another this one…

...what? a football organization in Spain has the power to restrict internet access to the entire country?

Football is really big in Spain dontcha know?

Gotta keep these billions flowing, and the shareholders happy..

Yes, that is exactly what is happening

I heard about this situation in Spain where Cloudflare (and maybe others) are blocked by mistake due to being associated with some sites streaming football.

How are people getting around this problem, I guess VPN? I mean, some of the wrongly blocked IPs are needed for work I imagine.

> initiated by the National Professional Football League and Telefónica Audiovisual Digital, S.L.U.

These people are criminals..

How does one get their hands on low-cost RISC hardware?

There's this $10 board on Aliexpress called Milk-V Duo S. It's been popping up on my recs every now and then. Looks interesting.

https://aliexpress.com/w/wholesale-Milk%2525252dV-Duo-S.html

Taken directly from its description:

>Milk-V Duo S is an upgraded model of Duo with an upgraded SG2000 master with a larger 512MB RAM and extended IO capability

>Onboard WI-FI6/BT5(Model Milk-V-Duo-S-512M-Basic/Milk-V-Duo-S-512M-eMMC does not have this function)

>USB 2.0 HOST port

>100 Mbps Ethernet Portwith PoE Support (via PoE HAT)

>Dual MIPI CSI with ISP

>The device also supports switch between RISC-V and ARM boot via a switch

I think there are dozens of boards by now, but what I found interesting and pledged to is this:

https://www.kickstarter.com/projects/starfive/visionfive-2-l...

I don't have a first version visionfive2 myself, but i heard good things about it and the eco-system sounds like it's growing. There are still rough edges. Lets hope they ship soon (tm)

Edit: What I currently have for my personal projects is a PolarFire SoC Discovery Kit. That's a quad-core RISC-V system with FPGA embedded. Maybe too expensive and not for everyone (130 bucks, fun fact: The dev board is cheaper than the chip itself).

https://www.microchip.com/en-us/development-tool/MPFS-DISCO-...

Also the microchip documentation, toolchain and so forth feel really crappy and old, but once you get used to it, it's not that bad and actually getting to run bare-metal risc-v code is easy. There are easy examples for linux and bare-metal

Maybe just run an emulator on your x86/apple, for now? It'll probably be faster than actual hardware, and it will for sure be faster to develop a kernel with.

https://www.qemu.org/docs/master/system/target-riscv.html

This has risc-v: https://www.raspberrypi.com/products/raspberry-pi-pico-2/

Thank you all for your excellent responses. No thanks to the fools who downvoted this.

Reply on news.ycombinator.com