Is It Time to Rewrite the Operating System in Rust? [video]
youtube.com
youtube.com
Rewrites from scratch are time consuming, and at risk of second system syndrome (especially when you suddenly have all the shiny new language features).
Direct 1:1 rewrites are underwhelming. After the initial C to Rust pass you end up with the same program, with a C-like architecture, and all you've done is replaced gcc with a slower rustc.
But wrapping a C implementation in a Rust API works great. You can translate C's implied ownership semantics into explicit Rust types. You can wrap all these C structs/handles with their init and free functions into RAII that never leaks.
It's something you can do in days, not months to years. You keep a fully working program at all times. You benefit from Rust for new development, and you can gradually refactor C out of it when and where it's sensible to do so.
I think before we jump into writing OS in rust we should first understand what wrong with current dominant Linux. I think we could make a major revision on how computers are built and how should be OS architected to last for another few decades.
Lack of an async API. Bad IPC primitives (how many are there? They are all terrible). The whole "everything is a file" idea. Real time audio. Bash. File systems. The process vs thread distinction. Package management. Permissions. Security model. Scheduler. General cruft. X Server. Battery life. OOM killer.
I would much rather hack on a Rust kernel than C.
The advantage of Rust is that you can split things into modular code bases. Redox is split very nicely.
I think that one could start a hosting company and write a hypervisor in Rust.
I'd hardly call Redox serious, it's a "research OS" with no explicit goals to actually be useful.
And that's fine really, but it is representative of the larger problem: no one capable of pulling it off is willing to be so ambitious. No one is willing to commit to the hard work of making a usable and practical platform, which will undoubtedly involve distasteful compromises and uninteresting work sanding down the rough edges.
I'd be glad to be proven wrong on this, but I just don't see it happening.
I'm saying in the next 10 years, there will be a serious OS written in Rust.
But as nickpsecurity likes to point out, C projects can make use of good, if labour-intensive tools for ensuring safety, so that seems like the logical choice instead of rewriting to languages which haven't proven themselves at the target time and complexity scale.
I think this has been repeatedly proven false over the last several decades. There really is no way to make C programs safe in practice. It’s time to move on.
> Go, JavaScript and others are garbage collected, making interacting with C either impossible or excruciatingly slow.
I don't see why interacting with C is a requirement. Android (SDK) could be considered an OS that doesn't have this either.
edit: I wasn't at the point you mentioned yet. Well, IMHO, it is not just the GC, it is not having access to native pointer prevent you to implement algorithm that are keys in term of performance for a kernel. IIUC more savvy dev than me it might not even be possible to implement a kernel in Rust without "unsafe" (which give you access to those raw pointers)
Happy to be proven wrong though.
It make me think of PHP. The language is actually really slow but because most of it's standard library are wrappers around actual C library, a lot of people perceive it as a really fast language.
But compared to raw C/C++/Rust, it is still slow.
Edit: I don't know much about this but it might be a better example "GRiSP 2 – Erlang and Elixir on bare metal." I'm assuming this is possible and includes the BEAM VM with GC.
That isolation model probably wouldn’t translate to the greater code ecosystem even though it’s very effective for BEAM applications.
I was under the impression that generally VM are idealized hardware that is friendlier for software development but they would make terrible actual hardware. I suppose it also depend on the language/VM.
But I'm totally out of my comfort zone. Let wait and see if someone more knowledgeable chimes in :-)
The language does allow for direct memory access though through some special functions that the compiler converts to direct memory accesses.
Both JavaScript and Go are fantastic of examples of languages deeply incompatible with OS work, as they expect the ability to manage threads, memory, and even networking as a core part of their language runtime and feature-set. None of this is going to work when you're trying to implement those building blocks itself. It is also not acceptable for a language to decide to do something else outside of the authors' control when working on a kernel, like both JavaScript and Go wishes to do.
Attempts to write kernels in Go have been made, but doing so involves defining a subset of Go so small that what you end up with is really just a small new language with a Go-ish syntax, without any of the features.
So no, C interaction is by itself not a requirement, but it's more of an indicator for the ability to be in control.
These can be implemented using a subset of the language that does not need them, pretty much like C++ requires 'new' but when writing a kernel you do not have enough functionality to use it - so you don't, until you have provided enough infrastructure for its use. Or in a similar fashion...
> None of this is going to work when you're trying to implement those building blocks itself.
...most C stdlib functions are also not available when writing a kernel in C (e.g. malloc/free, fopen/fclose, etc) and you simply do not use them until you have enough infrastructure in place to implement them.
Definitely. That subset for languages like Go and JavaScript is, however, so small that nothing is left. And for a kernel, that subset would have to be used in the vast majority of the code base.
If your goal is to write C with Go/JavaScript syntax, then that might be good enough for you, but that's hardly the point of "Should we rewrite in Rust" talks.
> ...most C stdlib functions are also not available when writing a kernel in C (e.g. malloc/free, fopen/fclose, etc) and you simply do not use them until you have enough infrastructure in place to implement them.
Of course, that's also how you'd do it in Rust: Don't use things that aren't there.
The Go runtime won't give you that choice, though, so you'd have to remove that, which removes pretty much all the language features that makes Go... Go.
Even the stable NDK APIs are actually JNI wrappers, with exception of the 3D and real time audio ones.
Regarding firmware, if the talk was about bare-metal embedded development, then I think Rust is the solution to a problem that no one has.
Since the food on my table comes from firmware development, I passed several days trying to understand the reasons to move to the approach described here: http://blog.japaric.io/brave-new-io/ (Embedded in Rust blog, great reading BTW).
But found little. It's a nice toy but it may be too much. If that is the trend, then I think Rust for Embedded is the answer to a question no one asked, because in my work, if I need to set a bit into a register, I have to be sure it's fking set; not playing around "just because it may be problematic".
Otherwise I agree that a rewrite is nonsense. In these kinds of huge projects, only a gradual solution that can co-exist with the existing code base could work, if anything. An example would be tightening the safety of the existing code through static analysis of the kind offered by Frama-C.
Some history of OSes, a bit about Rust. Is it worth re-writing the OS? No, but what about incorporating Rust into OSes, instead of creating a new one from whole cloth? There’s three approaches: in the kernel? Probably not worth it. In userland? Probably worth it. For firmware? Probably worth it.
https://lobste.rs/s/qz9d82/is_it_time_rewrite_operating_syst...
Overall I was sort of disappointed in this video. I would rather hear about someone writing their OWN code in Rust. I watched one of Cantrill's previous talks about this which I liked, so I watched this one too.
It's not that interesting to hear "well Rust has problems for MY domain (kernels), but OTHER people should write their code in Rust (user services and firmware).
There's an obvious asymmetry there, because you know all about the problems in your own domain, but you're less familiar with the problems of other domains (e.g. ones that require a lot of async code, a lot of text parsing, etc.)
-----
I just watched this video. I liked the part about failed C++ operating systems from the early 90’s (Apple’s Copland, an effort from Sun, and Taligent).
Overall he’s excited by Rust, but points out some problems with using it for kernel development:
- Multiply-owned data structures are all over Unix kernels, and Rust’s ownership system doesn’t like those. (i.e. the doubly linked list problem)
- Rust doesn’t allow you to handle memory allocation failure. This is being worked on? I didn’t know this about Rust and it seems odd for a low level language.
He says that instead of kernel development, Rust could be used for:
- user-level services like systemd, or
- firmware.
This doesn’t seem like anything more than a wish though. I don’t see that Rust really shines in those areas.
- I think you could probably write systemd in Go. I don’t see that Rust has any advantage there. Whereas a kernel in Go has some obvious downsides (I know it’s been tried).
- Rewriting firmware in Rust sounds nice, but I don’t think vendors will do it. They have enough problems writing decent C, and they don’t write open source code. He is wishing for them to write Rust and release it as open source, but I’m not optimistic about either of those things.
I think the kernel and the browser are where Rust is interesting – when you need both performance and memory safety. I’m not really excited about Rust for other applications like web services, i.e. doing the job that Go, Python, or Ruby can do. So it does seem unfortunate that there are still some open problems with using Rust for the kernel.
He also points out the many nascent Rust OSes kernels, and the fact that they are “from scratch” systems, i.e. not compatible. I agree that you need a compatible kernel to have any hope of adoption.
The standard library panics upon allocation failure, but allocation is a library thing, not a language thing. You can handle failure if you do it yourself. Eventually we have a path forward to handling it for those data structures too, but we’re not quite there yet.
This wasn’t a big concern because when you’re doing OS level stuff you’re generally not using the standard library anyway.
Although on second thought, I guess you can handle it because it's an exception rather than a panic. But exceptions have other downsides so people don't use them in that domain.
With a lot of low-level operations, go may have issues with mapping its threads to native. I can't remember which kernel interface was it, (there was a post about it on HN) but you have to call it for the same object from the same system thread. Since go didn't guarantee the mapping, you could randomly fail there.
Then again, I don’t remember seeing a crash in either Safari or Chrome in ages, but the point being that Firefox seems at least as stable as it’s competition.
I've used Mozilla M-series browsers back at the turn of the century and we crashed those "hundreds of times per day" but that's because we were DOM mangling back when that was new and exciting, not how everything works in every popular web page. Also the M-series weren't finished, hence the milestone number instead of a release version.
But for programs that I haven't written: no, they generally don't crash too often.
Unless it's a piece of code I wish to help improve, I see no reason to waste my time with broken tools.
I can't think of any software I use with any regularity that crashes daily (well, other than code I'm working on), and yes, I use Firefox as my main browser, so I'm not seeing the behavior mentioned here. I tend to leave applications open for days on end (or until my computer crashes). If something crashes often enough that it becomes expected, I look for solutions and replacements.
So yes, crashing isn't normal.
Rust does not disallow crashes, it disallows that your code will run in unsafe manners, either with compile-time errors or runtime panics (with some exceptions for `unsafe` code).
Browsers are to some extend more complicated than the operating system they run on, so it's no surprise that there are bugs in there.
If you experience crashes, please do submit the crash reports, and if you already know what the cause of the crash is, please file a bug on bugzilla. Thanks!
To be picky, it specifically allows unsafe code as long as it’s marked up as such.