Learn Rust by writing a small OS
os.phil-opp.com
os.phil-opp.com
While I have no interest (nor time!) in writing an OS, just skimming through this you can see the author put a significant amount of work and polish into this.
That said those are the only nightly features we use, staying on stable when you can is good, and many projects will be able to go to stable due to this, which is awesome. Been waiting on this for years.
I'm mostly a back-end engineer, but I've read the Rust Book[2] and I'd like to learn more about systems programming.
The other resources I would point you towards are Rust by Example:
https://doc.rust-lang.org/rust-by-example/
And the Rust Cookbook:
https://rust-lang-nursery.github.io/rust-cookbook/
Though the cookbook is kind of out-of-date. I'm actually in the process of updating / expanding it, but that effort is far from ready for presentation.
[1]: https://www.oreilly.com/library/view/programming-rust-2nd/97...
If you're comfortable in Rust, it is the next step up for intermediate developers.
It contains a good amount of low level details, but is still pretty fast paced.
I did, and I didn't like it much, but YMMV.
First: sadly, as common practice in the Rust books world, the book devotes 1/4th of the content to an utterly useless Rust guide. This is a marketing device (it's scammy for me, but it's arguable) to illude readers that they can read a book on learning Rust _and_ apply it in a certain context - but it's not possible to meaningfully learn Rust in 100 pages (not even in 400...). Those 100 pages would have been much better spent on-topic.
Rust language sections are also added to various chapters, which again, are redundant. Chapter 10 is entirely dedicated to multithreaded Rust programming, which is not systems programming.
Ultimately, it depends on what one exactly wants to learn and what they intend to do with it:
- if one wants to learn O/S programming, this is definitely not an O/S programming book; just the last two chapters are.
- if one wants to learn interfacing with system components (e.g. the network stack), especially with the intention of just reading without actually applying, this can be a fun book.
I personally don't think that the latter is systems programming, and I find the book misleading.
https://doc.rust-lang.org/nomicon/
Please ignore all the gatekeeping marketing on that front page - it's very counterproductive and not at all accurate. IMHO all of this material should just be included in the standard Rust book, especially the sections about ownership and implementing Vec.
The Nomicon is an extremely useful resource because it "desugars" all the magic that is happening when you write Rust, so you are left with a competent understanding of what the compiler is doing on any piece of code you are looking at or writing.
Valuable reading for people wanting to build real systems in Rust, IMHO.
It's intentionally different than most other text books. That means that each book in the Rust ecosystem complements the others. It's full of large worked examples from different domains. For example, you implement an NTP client, a database and a CPU emulator. It sacrifices idiomatic Rust for ease of learning. For example, it largely avoids higher-order programming to be accessible to as many people as possible.
Its role is primarily intended to teach you Rust, and to bring you up to speed with the jargon and concepts used in systems programming so that you can tackle dedicated material.
Let me know if you have any follow-up questions :)
It's not clear to me from either the top-level docs or a skim of the update posts whether this effort is essentially a loose collection of (useful!) blog posts, or whether they close the loop to form a coherent and complete whole.
Question: at the point my impression is that Rust is the consensus best overall systems programming language (assuming you’re starting something new). Is that just me or so others share that perception?
I’ve mostly done high level programming in my career but have wanted to eventually move into some barebones hardware stuff for fun. So I’m curious if Rust really is the best way to go these days (versus my old assumption that I should try to master C).
Rust provided a way into low-level programming without having to deal with any of these things at all. All I had to do was learn a few concepts like allocation and ownership. Easy-peasy compared to the above! This was especially true as when I learnt Rust I had need of it in production at work. I would not have had the confidence to put C code into production as a novice C programmer with no oversight, but with Rust this wasn't a problem at all.
Of course, you could emulate this behavior with Rust using extern and unsafe and whatnot, but the affordances of Rust steer you away from those details in favor of the higher level abstractions it offers. Which is what you want for 99% of software development, but when you’re writing platform specific code (e.g. OS startup, context switching, optimized SIMD) that a compiler can’t reliably generate, it helps to be able to quickly prototype something with C and then tweak certain instructions in the generated assembly subroutine until you get what you want.
That said, C is still a perfectly fine, very mainstream choice. If you just want to learn systems programming (rather than simultaneously learning systems programming and a new language), it might be the right place to start.
In terms of playing with hardware: did you want to learn rust, or understand low-level programming? I think rust will help you make a more robust, correct application (with a lot of time and effort understanding rust itself), while C won’t do much to help you but you’ll be able to be right next to the hardware. Personally, I think if your goal is to understand how hardware works, I’d use C. (But if you want to learn rust, use rust.)
It's not a fun language if you want to experiment or move quickly as the type checker might act as a brick wall in those cases. But as it prevents a whole class of bugs that are related to the majority of security exploits I think it's a pretty good choice if memory safety and performance are the main priority.
I think I'd probably agree with that impression. It's the route I've taken into "systems programming", and I don't regret it at all. The big advantage of going Rust-first for me is that C and C++ have so many unspoken rules that you need to follow in order to avoid security issues and hard to debug errors, whereas Rust codifies most of those as compiler errors. That makes it a lot more accessible for the beginner to learn not just the basic syntax, but a best practices and good habits.
C and C++ are still the mainstream at the moment, but I think they've peaked and would expect their popularity to wane over the next 10-15 years. On the other hand, Rust has only just hit the mainstream in the last year or two, so there's still a few missing pieces and adoption is not yet that high, but I think it's by far the best intro low-level language overall.
Yes, I agree this is the big advantage for me too. Rust allows mediocre devs to build more ambitious systems than they would have otherwise attempted in other languages. You can certainly build anything you want in C++, but from my experience due to the number of, as you put it "unspoken rules", novice developers will encounter a lot of foot guns. It leads to code that works but is very brittle; if you look at it the wrong way, it ends up segfaulting.
Rust says "You can't run this until we're sure it's not going to violate any of my assumptions of how a system is built" and it goes through your code with you to check off all the boxes. Is this thing mutable? Does it have more than one owner? Yes? Well then Rust says that's going to lead to pain in the future and prevents you from doing it. C++ will let you do it and hope that the learnings from the pain you encounter in the future due to your poor choices will prevent you from doing it again.
When I started learning Rust in 2015 the biggest thing in C++ I had built was a robot, and in that world you do most of your work as message passing. It's really more like a style that Erlang devs would find familiar. It's really not equivalent to doing systems programming in the OS/compiler sense. I found Rust very hard to use because I had poor habits in terms of object lifetime and ownership management. But over time the I figured out what the borrow checker wanted and in doing so, it made my code sounder, and therefore far more robust than what I would have put together in C++.
Going forward I apply these ideas to all languages I write in, so this is why I teach Rust in my PL course: even if students aren't going to write Rust in their future career, I've found it makes them think harder about variable lifetimes when they switch back to C and C++ in the OS course.
The concepts in Rust are a loose/spiritual superset of C; you'd be able to pick up C easily after learning Rust.
Also, I learned Rust with the linked series. It's extremely well though out and guides you into the mental model of "rustisms."
If you're learning low-level/barebones hardware programming, this is both a blessing and a curse. If you already know the low-level stuff, then Rust/C++ abstractions let you organize your code in a much nicer way, but if you're learning the low-level stuff, then those abstractions are actually in the way.
C won't get in the way, for better or worse, so it's definitely worth learning, at least to appreciate the safety valves Rust and C++ give you after you've repeatedly shot your own foot (specially Rust which learned from most of the historic mistakes of C++).
If you don't want to deal with C-nonsense, then Zig is probably the best alternative for that sort of low-level programming, it even has freestanding (no operating system) as a first class target, stack-traces and all.
Interfacing with other libraries, is ofc, a different matter, but you can usually wrap the library API in your own functions to minimize this mismatch.
You can also write literal C and use the C std-lib in C++ but it is the same problem. You're not using C++ at that point
Rust's stdlib assumes not only that there is malloc behind the scenes, but that it is ok to panic. Rust itself, including its stdlib, lacks a stable ABI. It also assumes that lots of little allocations and deallocations with a single owner is a valid memory pattern, and that writing directly to specific memory locations is something you never want to do. These are all wrong assumptions for someone writing baremetal. Look up abi_stable and no_std.
If you're learning, however, you know have to understand why you can't use generics (monomorphisation means you're not getting a stable ABI), why you shouldn't use native tagged unions and instead should write your own with unsafe (for total control of how the tag is handled), why you can't use the standard library, why you need to tag most things with repr(C), why the standard ownership patterns the borrow checker beat into you are in the way when writing your own allocator (with plenty of unsafe). Do you see my point?
Both Rust and C++, when writing baremetal, ask you to ignore how to write idiomatic code in their respective languages. This is fine for experienced devs, who will make good use of the additional features for abstraction, but not for newbies IMO.
C and Zig, on the other hand, do not have this problem, as idiomatic C barely uses its standard library (which is garbage anyway) and idiomatic Zig is designed to work baremetal, including the standard library. I certainly don't recommend using C for new developments in most cases, but for learning, well, you need to learn it anyway, even if you're using Rust baremetal. It's effectively what you're writing, albeit with lots of repr and unsafe strewn around.
if std::collections::HashSet::is_empty(&basket) {
if basket.is_empty() {
Under UFCS I could extend your Rust type by simply writing a new free function which matches the type, and this is in fact deliberately forbidden. (I can write the free function, but, doing so does not extend the type and I can't call it using the "method" syntax).
C++ has in the last decade provided measures for memory safety. Online resources are plentiful and references are easy to get at a bookstore. There are many different compilers (a plus I'd say compared to the monocultures many languages have), but Clang and G++ are ones to look at in particular. They share many extensions to C++ which are useful in systems programming. Clang can be installed on Windows using MS Visual Studio, while G++ is Unix-like only. C++ has many different build systems to choose from, but CMake a common portable one.
Ada came from a similar time as C++. It focuses less on memory safety than Rust, but has more of a focus on overall program correctness. A subset of Ada, SPARK, can be formally verified. The language's culture has more of a focus on embedded systems than general systems programming, and has less of an online presence than the other two. You will have to use reference materials and books more than online guides compared to Rust or C++. The open-source compiler of note is GNAT, it supports both Windows and Unix-like systems. It comes with a build system.
Rust, as you probably know, has a huge focus on memory safety. Rust is in culture much more like newer languages such as Python. Forums are active, updates to the compiler are more frequent, and online resources are plentiful. The book describing the language is a living document available online, unlike the other languages listed. The compiler, build system, and package management system is the same for every supported platform. This to me is a big plus, though package management is not very important for kernel-level development.
* Rust doesn't have volatile variables, you use volatile intrinsics on the loads and stores.
* C declares that arithmetic on 'char' and 'short'-sized variables gets promoted to 'int'; Rust actually has proper u8/u16 arithmetic support.
* Rust supports something akin to multiple return values via tuples, which means you can actually get operations like checked-overflow arithmetic supported in the core language, unlike C.
Wrapping::<i32>(i32::MAX) + Wrapping::<i32>(1) == Wrapping::<i32>(i32::MIN)
Now, on one hand, this seems like a very clumsy thing to write. But then on the other hand, did you actually want CPU-style wrapping integers? No? Then the ones you actually did want are easy to work with in Rust, but don't kid yourself you wanted a "high level assembler" if you can't even handle modulo arithmetic.
My take: Coding in C will teach you a two unique skills:
First, mastering working with pointers will inevitably lead you to debugging cache-miss related performance issues, virtual vs physical addresses when dealing with MMUs and a couple of other low level CPU details that are hard to learn about any other way (yes, this can also be done with C++).
Second, Because C is a barebones language, you’ll have to build everything yourself. Linked lists, hash tables, queues, binary search trees - all of it! And since you are working with pointers, a lot of the data structures and why they matter will make sense at a level that is impossible to grasp with say python (For example, any C programmer who picks up the usual university textbooks on algorithms and data structures looking for a reference to implement a hash table will end up disappointed- most books tell you how hash tables work, but very few tell you how to implement a hash function correctly - and with C, this matters a lot!)
C++ trades off some of C’s language simplicity in exchange for developer velocity. Hash tables, vectors etc are all taken care of for you. But this is actually a dangerous trade off: C++ gives you a language that will not fit in your head, and you can never be entirely certain about the behavior of code hidden from you by design.
Rust makes a ton of sense as a C++ replacement. It takes the same trade off that C++ did (give up language simplicity for developer velocity) but also adds guard rails around to help keep things sane.
I am convinced that rust is the future in every place where C++ makes sense today.
C is different. Yes, it suffers from many of the same bad things as C++. However, people who write C also work differently: they are used to building everything from scratch, they know their projects need more time to complete, they can look at almost every single line of code and tell you roughly what machine code it will compile down to. The language is small enough that it everyone knows all of it, most agree on the best way to do things and critical bugs are often spotted by just recognizing that some code doesn’t appear to follow well known patterns for implementing something (see how the OpenBSD community find bugs for example).
In short, it’s hard to recommend rust over C because they kind of come with different developer ethos. But Rust over C++ is a no brainer any day. And Rust over C makes sense any time C++ over C makes sense (which is the case for most C projects)
One interesting quote about rust in the context of OpenBSD [1]
> For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.
C is not going away anytime soon.
Good on you if you can pull it off though!
Is there any similar OS project (or even book!) that uses the latest operating system principles and non-hypothetical (i.e., unlike MIX or MINIX) languages?
Writing an OS in Rust: Async/Await - https://news.ycombinator.com/item?id=22727985 - March 2020 (103 comments)
Writing an OS in Rust: Advanced Paging - https://news.ycombinator.com/item?id=19017108 - Jan 2019 (141 comments)
Writing an OS in Rust: Introduction to Paging - https://news.ycombinator.com/item?id=18903235 - Jan 2019 (79 comments)
Writing an OS in Rust: Hardware Interrupts - https://news.ycombinator.com/item?id=18274235 - Oct 2018 (63 comments)
Writing an OS in Rust, Second Edition - https://news.ycombinator.com/item?id=16556481 - March 2018 (40 comments)
Writing an OS in Rust: Handling Exceptions - https://news.ycombinator.com/item?id=13961020 - March 2017 (41 comments)
Writing an OS in Rust: Returning from Exceptions - https://news.ycombinator.com/item?id=12548066 - Sept 2016 (54 comments)
Writing an OS in Rust: Better Exception Messages - https://news.ycombinator.com/item?id=12218867 - Aug 2016 (31 comments)
Writing an OS in Rust: Catching CPU Exceptions - https://news.ycombinator.com/item?id=11791694 - May 2016 (41 comments)
Writing an OS in Rust: Remap the Kernel - https://news.ycombinator.com/item?id=10822479 - Jan 2016 (25 comments)
Writing an OS in Rust - https://news.ycombinator.com/item?id=10807816 - Dec 2015 (34 comments)
Writing an OS in Rust: Allocating Frames - https://news.ycombinator.com/item?id=10569463 - Nov 2015 (14 comments)
Writing an OS in Rust - https://news.ycombinator.com/item?id=10448136 - Oct 2015 (34 comments)
Kudos.