Reenix: Implementing a Unix-Like Operating System in Rust [pdf] | Hacker News Reader

Reenix: Implementing a Unix-Like Operating System in Rust [pdf]

scialex.github.io

187 pointssteveklabnik21 comments

"The only problem is that rust was built with the idea that it will be used with a malloc style memory allocator. This kind of allocator is rather hard to implement using slab allocators since malloc must be able to allocate buffers of any size."

...

"About the only somewhat difficult challenge was that I was unable to find any clean way to switch between different virtual terminals. I ended up having to just update a pointer so that the interrupt subsystem could know which TTY to send input to." (discouraged in rust).

It's a good exercise to try to take a language with promise and use it for a systems task to understand what's possible. But sometimes some languages need to be systems languages, and others need to be application languages, because there are different assumptions and default cases we want to make easy/possible for developers.

Well TTYs are apparently they own kind of hell.

We are talking about a system that can trace a lineage back to the Morse telegraph...

With bits like schedulers and device drivers in an interrupt-based system a lot of ideas we'd like to about runtime state as modeled by your language of choice (the stack, lexical scope, memory, thread context, reference counts) become exposed and the implementation details have to be known, or you have to be comfortable tuning/self-restricting use of features around such a tricky, dragons-be-here scenario.

This isn't a problem specific to TTYs, but indeed they are their own special little hell no matter how you choose to tackle them.

I like OSv, just because it doesn't have TTYs.

Allocators are being worked on. It's not a fundamental language design issue. It didn't make it for 1.0, but the infrastructure to eventually supporting custom allocators has been part of the design of the standard library for quite a while.

I ended up having to just update a pointer so that the interrupt subsystem could know which TTY to send input to." (discouraged in rust).

Why would that be discouraged, and what would be the "Rust way"? That sounds like the simplest, most straightforward way to do it.

You don't _generally_ do arbitrary pointer arithmetic in Rust. You can, with the 'raw pointer' type, but this paper attempted to use all of Rust's higher-level features. Raw pointers need `unsafe` to dereference, and unless you're doing something like this, aren't generally needed, so are 'discouraged' in a certain sense.

> what would be the "Rust way"?

I'm not sure there is one TBH.

So basically TUNIS http://en.wikipedia.org/wiki/TUNIS for the Rust era and language.

It makes sense. Some of the best Unix implementation work I've seen (e.g. the DG/UX reimplementation of SVR4 UNIX for multiprocessors, large storage farms, and manageability) came from the "let's consider refactoring everything" that comes with a clean-sheet design.

"This boot-loader, unfortunately, did not support loading any kernel images larger than 4 megabytes. This turned into a problem very quickly as it turns out that rustc is far less adept than gcc at creating succinct output. In fact, I was hitting this problem so early I was barely able to make a “Hello World” before having to stop working on rust code."

This really caught me. I have not played around with rust that much but having the executable be over 4MB, with such a small program such as just a Hello World is kind of nuts. I know that they still have lot of compiler optimization, but I just thought that was a very interesting portion of the read.

The actual kernel binary was smaller but the loader s limit was based on the amount of memory the kernel laid out in memory takes, including anonymous sections.

Also turning on optimizations in rust and the linker helped a lot but were not enough in the end.

I guess you are the author of linked paper. Well done.

I wonder if you have any opinions on how useful higher-kinded types (HKTs), one of the most requested features for Rust, would have been in implementing this OS. For example the absence of HKTs means that Rust can't have smooth and general handling of monads that is comparable to Haskell's. Would Haskell-style monads or Arrows have been helpful?

Thanks.

I believe HKT would have been useful but am not really sure how much HKT would have helped me overall. They would likely have been most useful (if at all) in the VFS/S5FS/VM stages that I was unable to fully get to.

Honestly, I have not used haskell enough to definitively say whether monads/arrows would have been useful.

I guess the only way to find out is to await the emergence of low-level languages with HKTs and then run suitable experiments.

Ah, so all memory used by the kernel must be listed in sections loaded by the bootloader? As opposed to the normal situation where the kernel just starts grabbing it from the hardware?

I know no specifics of the situation, but llvm optimisations can be manually toggled through rustc. Maybe there are more special-case optimisations that could help.

    > having the executable be over 4MB, with such a small
    > program such as just a Hello World is kind of nuts

Presumably it's a pretty small class of program for which this is a Real World Issue.

Unfortunately, the kinds of people who care about that sort of thing overlap heavily with rust's intended user base.

Also, end users might not care that much about binary size, but distribution maintainers do -- they want to fit as much functionality they can within a reasonable-sized iso download. If including a rust program in the base OS means omitting a dozen C programs that would have fit in the same space, that's a hard sell.

I am optimistic that these kinds of issues will be worked out as the language stabilizes and the tools improve.

PDFs discussing topics like that used to look much more bleak. The code listings here look better than most of what I've seen before. Even the (dis)assembly has some syntax highlighting. This content deserves to be that nicely presented, I love it.

I wonder how this compares with the benefits of writing a kernel in a high level language as seen in the Singularity project.

http://research.microsoft.com/pubs/52716/tr-2005-135.pdf

http://research.microsoft.com/en-us/projects/singularity/

I think it would be cool to study defect rates of a serious kernel implementation vs historic rates in linux and others.

Reply on news.ycombinator.com