Using Rust for an Undergraduate OS Course | Hacker News Reader

Using Rust for an Undergraduate OS Course

rust-class.org

126 pointsbrson71 comments

> Go. Go is a systems programming language

Not those kinds of "systems". It is funny, it seems initially it was meant to be those kind of "systems", and then it pivoted as we like to say, to become a "distributed-server-network-backend systems" not "hardware-kernel-OS" kind of systems. And then creators kinds of winged it and remarked how "well, that's what we meant when we said systems".

As for Rust, yeah, there are already a few projects trying to build a kernel or drivers in Rust. Here is one for example:

https://github.com/charliesome/rustboot

> it seems initially it was meant to be those kind of "systems"

From the announcement talk: "And it's a systems language in the sense that we intend it to be used to write things like web servers" http://www.youtube.com/watch?v=rKnDgT73v8s

I don't think that Google ever wanted to develop a new OS kernel.

> From the announcement talk

From the announcement blog (and that was the widely circulated quote in tech news not the video transcript):

---

Go combines the development speed of working in a dynamic language like Python with the performance and safety of a compiled language like C or C++. [...] And the compiled code runs close to the speed of C [...] Go is a great language for systems programming with support for multi-processing ...

---

http://google-opensource.blogspot.com/2009/11/hey-ho-lets-go...

> I don't think that Google ever wanted to develop a new OS kernel.

As the meme goes, they keep using the word "systems", I don't think it means what they think it means.

It's a tough crowd when being the authors of both the Unix and Plan 9 kernels isn't sufficient to get the benefit of the doubt on this matter!

That's probably the problem. When those guys say "systems", people expect they mean OS/kernel-level stuff.

> As the meme goes, they keep using the word "systems", I don't think it means what they think it means.

I'm pretty sure it doesn't mean what you think it means. The only title I have on my bookshelf which has "Systems Programming" in the title is System Software: An Introduction to Systems Programming. The book is about creating assemblers and compilers for a made-up architecture called SIC: http://en.wikipedia.org/wiki/SIC/XE

This is neither here nor there, but the point is that a fair number of accomplished people in the field were apparently at one time less hung up about precisely what "systems programming" might mean and (this is just conjecture on my part, since honestly, who cares?) probably just used it as a contrast with applications programming.

Jeez.

These dudes were writing "systems" long before most folks here on HN were programming.

To me, and to them, "systems programming" has a particular meaning that does not include kernels. Like it means assemblers, compilers, web servers.j

I think they have more credibility about what it means than perhaps you do.

Before many HNers were even born too.

This is possible the worst example of selective quoting I've seen on HN.

From the bit you left out:

Want to write a server with thousands of communicating threads?

It's always been clear what kind of apps Go is targeted at, and it's never been operating systems.

I think that is the definition of "systems" the class is using though. If you look at the programming assignments, they are all about implementing programs in userspace and not hacking on the kernel.

Yeah, I was confused at first too. Not sure how you can call it an "OS class" if students never touch kernel code. Although, apparently one project group did manage to write a kernel in rust based off of rustboot (https://github.com/wbthomason/ironkernel).

It looks to be inspired by CMU's "Introduction to Computer Systems" course (http://www.cs.cmu.edu/~213/). The professors who teach that course wrote a textbook, "Computer Systems: A Programmer's Perspective" (http://csapp.cs.cmu.edu/public/students.html).

I was a TA for a year for a version of that course adapted to the school I did my PhD at. We did the same thing: this course became the required one, and OS became an elective. I was skeptical as well, but after TAing it for a year, I think it is absolutely the correct thing to do. This course instills an understanding of the whole stack of a computer system that exists in user-land: assembly; how programs are executed including the stack, data segments, dynamic memory; code generation; process management and concurrency; memory allocation; network and server programming. The point of the course is how different levels of the system interact, not just studying one component in isolation. It is the kind of course I wish I had as an undergrad; instead, I had to build that knowledge piecemeal during grad school.

More people will do user-land systems programming than kernel systems programming, so I think it makes sense to make the user-land course required, and the kernel level one an elective.

Yeah I am a little confused by it. Actually Go or Erlang would be a good choice for this "OS" class and they do seem to have adopted the other "systems" definition.

Rust is a high-level systems programming language. Go is a low-level managed language runtime

This is probably the best clarification I've read regarding the difference between Go and Rust. They operate in different domains, and any competition is out of confusion over definitions (the source of many needless disagreements throughout human history).

Other than the word "systems", was there any other indication that the Golang team meant to target OS kernels?

Not sure about OS kernels, but their intent was to attract C++ programmers, although, they ended up mostly attracting python programmers.

You mean, if we assume that the word "systems" doesn't have a well defined meaning, that especially the Go team would know all too well when it used it?

I define "systems programming" to be programming which must interact directly with the kernel or devices. Doing so requires knowledge of how those systems work. It includes, but is not limited to, writing operating system kernels.

Systems programming does not mean operating systems.

To me it means "infrastructure" - things like web servers, database servers, or the wide range of distributed systems that are currently written in Java (eg, Hadoop).

I am more into D and Rust than Go nowadays, but I think Go can be used for writing an OS if one so desires.

The language is quite similar to Oberon in features. You just need to provide a syscall package for bare hardware.

If powerful single user graphical OSs were written in Oberon, why not Go?

I don't really get why this is a problem for a kernel programmer:

> lack of any intrinsic support for concurrency

An important part of an OS is that it implements support for concurrency - it builds processes and threads out of the raw materials such as page tables and timer interrupts. A kernel also builds its own spinlocks and higher-level waiting primitives. If you are relying on some high level language runtime to do the heavy lifting, is that really learning about how an operating system works?

For a kernel I actually see a lack of language support for concurrency as a plus. You need something to sit below the higher level stuff.

Rust doesn't have any built-in concurrency in the usual sense either (well, except for thread-locals and atomic intrinsics). What it provides is a few simple features (uniqueness and safe references) that allow programs like kernels to create their own safe, race-free abstractions. Instead of providing features directly ("scenario solving"), Rust's concurrency features are more like building blocks to help libraries build the features they need.

Is there any support for the article's assertion that C uses ‘=’ for assignment (and ‘==’ for equality) “to save memory”? dmr simply described it as “a matter of taste”¹ and hinted that he was simply used to typing ‘=’ for assignment². [Edit: And the compilers didn't keep source in memory, anyway.]

¹http://cm.bell-labs.com/who/dmr/chist.html (also published in HOPL-II) ²History of Programming Languages II, Transcript of C Presentation, p692

(Nice to see Rust trying to bring progress to the C niche, BTW.)

Not to mention that Rust uses = and == the same way as C, although the expression "a = b" has unit type.

This just doesn't make a lot of sense to me, given that almost all systems programming in the real world is done in C and C++, and given the complexity of OS kernels, we're not going to see a general purpose commercial grade one written in Rust either ever (if it falls out of popularity) or for ten to fifteen years (if it becomes very popular). This isn't like application or web programming - in comparison to web technologies it moves at a glacial pace. So I think teaching an OS course in Rust is going to leave them underprepared and be a big shock if the students actually go on to do systems programming on real systems.

The other thing is that I did an OS course in C, where we got dirty playing around in the depths of the Linux kernel, replacing the scheduler, writing a filesystem, and so on. It was damn hard, but probably one of the only courses in Uni that really pushed me as a programmer generally (everything else I found easy). I think that it really helps to have experience in unsafe languages like C, just like you really need to do at least a bit of assembly when you teach computer organisation.

I'm not sure the lack of safety is the pedagogically interesting part though. The interesting parts of writing a kernel for x86 are the low-level interfaces: assembly, I/O ports, the 8259A PIC, the fundamentals of scheduling, writing a slab allocator, and so forth. Rust allows unfettered access to those just as C does.

The lack of safety in C just means that the projects take longer to debug because the compiler doesn't catch as many mistakes; I don't think it adds much to the learning experience.

Most of the real world uses C and C++, because of the vicious cycle he mentions on the post.

You don't need C's lack of safety everywhere for doing an OS. There were OS written in Mesa, Modula-2, Modula-3, Object Pascal, Oberon, Ada and many others.

C's strings and vectors are recipe for security exploits without any benefit for kernel programming. Or lack of type safety between compilation units.

It is a very good thing students learn C and C++ aren't the only way to write an OS. Sometimes the world seems to forget other languages have been used for the same purpose.

You may be imagining that it will be complete OS/systems written in non-C language from scratch, but it doesn't have to happen that way.

With languages with no GC or optional GC, calling back and forth between the new language and C is easy and direct, and so writing only new (or maybe historically problematic) pieces of a system in the new language becomes feasible.

I'm also teaching an OS class, and IMHO using some new shiny thing is not the right way to approach it, because the students will be put in an ideal imaginary world, and neither will face the real-world practice, nor understand how it all came to be the way it is now. I think, it may be a good follow-up course (like "System Programming in Rust") after the "normal" C-based course.

I also don't agree with the notion that other option is "to teach students to write C code riddled with security vulnerabilities, memory leaks, and race conditions". This is exactly what such kind of course should teach them not to do, clearly pointing these problems and explaining how to deal with them. But not having exposure to that in the learning environment will only make them repeat these mistakes in the real world.

Btw, I'm a Lisp programmer and not a C fan.

It is however a very good idea to teach them that C is not the only way to write operating systems and it is currently on this position, because of a few historical accidents.

Which we pay every day with extra tooling to cover for the language deficiencies on the area of security.

I was one of the students in this course. If you're interested, I would recommend reading the professor's "course wrap-up"(http://rust-class.org/pages/course-wrapup.html) - it discusses his philosophy going into the semester. In particular, it addresses the most common comment I've seen in this thread, namely the rationale behind choosing to focus on general systems coding rather than kernel hacking.

Personally, I found the course enjoyable and worthwhile. It was not what I was expecting, and I was admittedly disappointed that we didn't go much into actual bare-metal programming. That said, I found it to be a good introduction to the concepts of systems programming, with a few detours. As I am interested in OS coding, I'm performing my own independent study to learn the material which might have been covered in a more "traditional" OS course, but the course did a solid job of covering concurrency, memory allocation, etc.

As for the choice of Rust: I really like the language, and plan to continue using it for personal projects. I'm not sure it was quite mature enough for exclusive use this past semester (the 0.7 to 0.8 switch midway through threw the TAs for a bit of a loop), but I've been very impressed so far, and think that this coming semester of the course (which will also use Rust) should have a much better time of it. The one annoyance I had with Rust was the lack of documentation, but the Rust community (including a fair number of those in this thread) were incredibly helpful, and I'm currently working on a project to provide an alternate tutorial, using a series of small "stepping-stone" programs to build from "Hello, World" to a MapReduce implementation. Personally, I think I might have picked either Go or a blend of C and Rust for this past semester, but Rust is becoming a better choice as time proceeds.

I was also one of the members on the student team that made the ARM kernel in Rust. Due to the timing of the project (right in the middle of preparation for final exams and the rest of our end-of-semester project due dates) and some "lazily evaluated" work ethic on the part of our team, it was fairly rushed, and is much less complete and polished than I would like. I'm planning on continuing work on it. Though I would disagree with the assertion that separating Rust from its runtime is as "painless" as it has been said to be, it wasn't terribly difficult, and solutions for that particular problem are also constantly being improved.

> its (C's) lack of any intrinsic support for concurrency

If you are writing the operating system kernel, who is going to give you concurrency support? Does rust running on the bare metal still have "spawn" and the other concurrency primitives?

They aren't language primitives; they're just part of the library. So presumably you'd implement them yourself using a little bit of inline assembler, and they'd then be on equal footing with the versions in the Rust standard library.

They are not writing the kernel. See the other discussion in the thread regarding the term "systems programming".

> I considered five possibilities: Java, Python, D, Go, and Rust.

It's a pity that people rarely consider Nimrod (especially when it is far more suitable than the first two possibilities). Here is an example for anyone that is interested: https://github.com/dom96/nimkernel

"The Rust compiler normally disallows any code with race conditions"

Lol what? Unless tasks cannot communicate at all, or they are scheduled entirely deterministically, how can the compiler eliminate race conditions? How can a compiler even determine what is a race conditions and what is intended non-determinstic behaviour.

The "default" communication primitive is message passing; and Rust has linear types, so passing anything from one task to another will disallow the use of that thing in the first task. e.g.

  let value = something();
  channel.send(value);
  // I can no longer use `value` here, it was moved into the .send call

(There is some subtlety here; some values don't move when passed around by value, like primitive numeric types, but in general, anything that could cause a race condition (i.e. pointers) moves.)

However, this isn't the end of the concurrency story; as mentioned in that article there is the `unsafe {}` escape hatch which lets you do things that could cause crashes/race conditions/etc. that the compiler wouldn't normally allow (`unsafe` is basically the programmer saying "trust me, I know what I'm doing" to the compiler).

This unsafe hatch allows one to implement shared memory with safe wrappers[1], Arc (Atomic Reference Counted) for immutable shared memory (without any locks[2]) and RWArc and MutexArc for lock-protected mutable shared memory.

[1]: http://static.rust-lang.org/doc/master/extra/arc/index.html

[2]: Rust has very good immutability support, so it can express "this value can never be modified" in the type system, so we can have shared memory without locks entirely safely, because we know that there is no modification possible and so no race conditions possible.

(I'm answering the data races question, which may be a non-sequitur if you're asking about race conditions in general, as pcwalton points out.)

Two tasks A and B both send a message to task C. Does the message from A or the message from B get received by task C first? That's a race condition.

"A race condition or race hazard is the behavior of an electronic or software system where the output is dependent on the sequence or timing of other uncontrollable events."

"It becomes a bug when events do not happen in the order the programmer intended."

http://en.wikipedia.org/wiki/Race_condition

If I thought that the message A would definitely get there first - who knows why but that's what I thought when I wrote the code - this is a race condition and a bug.

Please explain to me - how does the compiler prevent this race condition and bug?

It doesn't. It prevents data races.

(But that said, there was some experimentation a couple of years ago with "channel contracts" in Rust, which allow you to solve that problem in many scenarios by explicitly enumerating the state transitions in your channel. We have the mechanisms still in place to do this as a library if we want to.)

From your link:

C++11 introduced formal support for multithreading, and defined a data race strictly as a race condition between non-atomic variables. While race conditions in general will continue to exist, a "data race" must be avoided by the programmer, who must assure that only one thread at a time may access any variable if the access is for writing.

What does C++11's definitions have to do with anything? This is Rust.

As I edited in at the bottom of my answer, I slightly misinterpreted what you said and answered the question restricted to data races only.

I believe the correct term would be "data race", which is a subset of race conditions. If you have no unsafe blocks, it's impossible to race on memory.

Races related to messages are of course possible.

Usually we say data races to be clear. Rust has full support for shared memory, but the compiler ensures that you are not accessing it without locking if it's mutable. (If the memory is immutable you can access it without a lock.)

You can of course have races in message passing, or the disk, or the network, etc.

...but you can still have races with locking.

Data races don't mean data corruption. It simply means the result is dependent on the scheduler or something else out of your control.

Two threads set a shared variable to different values. They both lock it before the set it. But that's still a data race, as which thread write first is dependent on the scheduler.

What exactly is your goal here, with this reply chain you're making? No, Rust has no magical solutions to message passing order. There aren't any, because your "message A" and "message B" could well be network transactions, which will come at you in arbitrary order from two different machines. Given that there's no "solution" to this problem anyhow, what's the point of trying to attack Rust here?

I'm defining "data race" in the same way that race detectors do—concurrent access to a piece of memory without use of a synchronization primitive. Rust rules those out at compile time.

No compiler can reason about whether some high-level piece of code is semantically order-of-execution dependent or not; not every non-deterministic piece of code is a race-condition.

You: "not every non-deterministic piece of code is a race-condition"

Wikipedia: "A race condition ... is ... where the output is dependent on the sequence or timing..."

Boom! Headshot!

The wikipedia article means the sequence or timing of the execution of the instructions in relation to each other.

Non-deterministic code does not only encompass code of which the results vary with the timing of instructions relative to each other, but also code of which the results vary with the timing of instructions relative to the state of the universe or anything in it.

    Boom! Headshot!

That adds nothing to the discussion and makes it feel more adversarial. It reduces the quality of discussion for no gain. Please don't do that again.

Being nondeterministic doesn't necessarily meant the output differs. (E.g. a program that increments a global counter (with a lock) once from each of ten threads and prints the count after all threads are finished; the increments happen in a nondeterministic order but the output is always 10.)

Cool, but i think most people would not call this an OS course.

A similar kind of course for graphics programming is highly desirable.

Also, I might like to add, PS4 toolchain fully supports LLVM- Clang. It might be wonderful if rustc starts running on it too...

"I'm not aware of any other language that has concurrency constructs as elegant and easy to use as Rust's spawn, and I don't know of any language that comes close to the race-free safety guarantees provided by Rust."

I'd recommend having a look at Erlang, it allows you to start lightweight pseudo processes for concurrency and provides a robust data sharing model.

Not so sure about low level access, we used C for those parts in the OS course I took.

Erlang's a weird choice for an OS class. An Erlang program is run in a VM that aggressively hides from you the exact things you want exposed in an OS class. As a user, that's great, as a way of teaching OSes, it's not so great. Plus the VM simulates a machine that is quite unlike today's real machines from an OS perspective.

Granted for an OS class they just seem to be writting a lot of web and network servers. Given their assignment list Go or Erlang would actually be a fit.

I for example remember writing a kernel driver and learning about virtual memory and io schedulers. We wrote file servers in the "networking" class but hey, that was years ago and a different college, so maybe I am just old.

Implementing Erlang on bare metal would be a good idea for an OS class, writing code in it, not so much.

> I don't know of any language that comes close to the race-free safety guarantees provided by Rust.

STM in Haskell.

Can anyone elaborate on how "Rust changes the way you think"?

With Rust you need to be explicit about everything's lifetime. You cannot allocate some chunk of memory and freely borrow it; when borrowing you need a certificate (i.e. compiler check) that the borrowed reference has a shorter lifetime than that of the referenced memory. It is tremendously different from the absence of such guarantees (like C) or the GC-based weak guarantees (like Java and Go, which does not solve the logical memory leaks anyway [1]).

[1] Not that Rust completely solves the logical memory leaks, but it makes the leaks explicit.

c++ smart pointers under the hood? its actually cool.. to bake some of this directly in the language.. can prevent some memory copying by playing with raw pointers indirections under the hood in the lang runtime, etc..

But you already has to think like this when you use smart pointers in C++.. "who owns this reference?" etc..

But in C++ this is a library.. not a syntax.. point to rust for this

You already have to think like this when you use smart pointers in C++! But C++ compilers don't help you with that! The Rust compiler does. Using Rust, you get surprised how often you get ownership wrong, because the compiler tells you every time. The same analogous code in C++ will still be wrong, but you wouldn't know, since the compiler tells you zilch.

hello sanxiyn! (fyi all, sanxiyn is one of the rust compiler developers, unless i'm mixing up names :) )

(There are quite a few of us in this thread, fwiw: brson and pcwalton are Mozilla employees working on Rust and Servo; sanxiyn, bjz_, steveklabnik and I are volunteers, just from the handles I recognise.)

sanxiyn is actually a Samsung employee, and on their behalf has contributed to Rust in the past. IIRC he's not currently employed to work on Rust, though he's still an active community member.

For balance, here is an experience report teaching D: http://forum.dlang.org/post/yutwxozxoqrfalzfswht@forum.dlang...

I'm the instructor for the class (and author of the post). I'll try and clarify a few things here.

If anyone is interested in participating in the next version of the course (which starts Jan 14), please submit the form here: http://rust-class.org/pages/spring2014.html

- "Operating systems" vs. "Systems programming"

If your definition of an operating systems course is a course where you implement your own OS or hack on the Linux kernel, this wasn't an operating systems course. But, most people have a broader interpretation of operating systems courses today, to include courses where you learn about the layers between high-level programs and physical things, and about how to build robust, scalable, and secure computing systems. This is the second type of course. Deciding do a course that was not entirely focused on building an OS was not related to the choice of using Rust (I already knew I didn't want to do a build-an-OS course before thinking about which language to use), and I don't think its controversial (this is what the majority of top programs already do). I discuss more about this in the general course wrap-up: http://rust-class.org/pages/course-wrapup.html

- Eliminating "Race Conditions"

You are right in pointing out that Rust doesn't eliminate all race conditions (and no language that allows multiple threads and any interaction with the the external world really could do this). My wording here was very careless. What Rust does is use language/compiler mechanisms to eliminate the kinds of pernicious data races (multiple threads reading and writing the same mutable state in uncontrolled ways) that are a very common and hard to find and fix problems in most multi-threaded programs.

- C's assignment operator

My perhaps somewhat hyperbolic diatribe about C using ‘=‘ for assignment is meant to illustrate how design decisions C's designers made for good reasons given the computing systems they were using in the 1960/1970s, would not be the best decisions if one was designing a language from scratch today. I think the choice of ‘=‘ illustrates this well, but there are dozens of other more serious issues in C's design that were good or necessary choices in 1972, but are undesirable legacies today: not having bounds checking, not specifying the order of evaluation for many constructs, allowing arbitrary and unchecked type casting, unsafe memory management, etc. Other languages (including Rust) that are strongly influenced by C syntax have also adopted the ‘=‘ symbol for assignment, but that doesn’t make it a good thing. For languages like Rust that are targeting experienced programmers it is probably the right choice; for languages like Python that are intended as first languages, it is really unfortunate, and many smart people who might otherwise turn out to be talented programmers are unnecessarily put-off by this. (I don't have concrete data to support this, but from having over 350,000 students in my open intro CS course that uses Python, I have plenty of anecdotal experience with students being confused by this.)

Vocational value

This was a course at a public university, so its content should not be primarily driven by immediate vocational concerns. There's nothing wrong with vocational courses, where the primary goal of the course is to improve the immediate job prospects of students who take it, but was not and should not be the goal of courses at public universities. (That said, the students entering this class are already very well qualified for the job market, and many of the 4th years in the class already have job offers, so are not in danger of not being able to get an interesting job since they only have C experience from one previous course. Knowing Rust has actually been helpful for some students in the job market, and I don't think having less C experience is a major issue for many positions.)

From a purely pragmatic viewpoint, the majority of the costs of the course are not covered from student’s tuition, but are paid by the US and Virginia taxpayers. So, our main goal should be to do what we can to enable and encourage students to do things that make the world a better place (the best ways to do that, of course, are debatable, and maybe it is learning how to hack the Linux kernel, but I think that’s a much tougher argument to make.)

It's a good start, but I'd honestly feel cheated if given such a course. I agree that the C tide may need to change, but using a language pre-stabilization AND that has literally no job opportunities is a real drag. So, the same goes for Go as well. And the fact that python or any other such language was even an option scared me just a little. I believe in Rust, and hope it succeeds, but don't feel this was a good avenue for it at this time.

Seems like a fun course. Just not an OS course.

Why wasn't PHP on the shortlist?

Reply on news.ycombinator.com