Committing to Rust in the Kernel

There’s a recent drama where the Rust folks asked some people to clarify some of the semantics of some of the filesystem APIs, and this request wasn’t taken well. There’s been a bunch of hn threads about it.

About invariants generally: Rust wants to know when memory behind each pointer is immutable, or mutable by Rust only, or could be mutated by something else while Rust has a pointer to it. Rust also encodes which types can't be moved to another thread, and which pointers own their memory and need to be freed by Rust.

These are part of the type system, so they need to be defined precisely. The answer to these questions can't be just "it depends". If there are run-time or config-dependent conditions when the same data is owned or not or immutable or not, Rust has enums, unions and unsafe escape hatches to guard access to it.

This is true in a literal sense, but there’s some nuance. They were working on upstreaming them, but there was some strong disagreement about one of the earlier patches and so at least for now they’ve given up. And that patch was pretty small, it’s not like a massive change was required.

That is factually misleading. Look at LWN's coverage since the recent controversy over a Rust developer quitting the project:

1. Rust-for-Linux developer Wedson Almeida Filho drops out - https://lwn.net/Articles/987635/

2. Airlie: On Rust, Linux, developers, maintainers - https://lwn.net/Articles/987849/

LWN never once mentioned any blogs or opinions from kernel developers or otherwise unless they were in support of Rust. There has also never been an article highlighting the very real challenges not only with Rust but also the attempt to integrate it into the Linux kernel project. Scroll the home page for recent articles and it's pretty evident where the bias exists.

Meanwhile, The Register[3] cited at least Drew DeVault among others:

3. https://www.theregister.com/2024/09/02/rust_for_linux_mainta...

Yes, The Register wrote a more balanced article on the topic than LWN.

> He’s no more an evangelist for Rust than Linus is

Linus does not have a personal website where he is only publishing positive articles about Rust.

I was hunting for what Windows libraries used a particular new API and saw it in my scrolling!

You can see all the panic and error strings, and some internal package paths if you run strings over it. Win32k looks like it got split pretty hard into a couple of sub libraries in recent versions though (win32k, win32kbase, win32kbase_rs etc)

Thanks! I’m aware of most, but not all of that. But that stuff isn’t kernel stuff, though obviously much of it is serious systems work. That’s what I was referring to specifically.

At least Ted Tso has clarified his position greatly from what initially read as “I refuse to learn rust”. Instead yes asking for a guide to help him understand the Rust:

> There is a need for documentation and tutorials on how to write filesystem code in idiomatic Rust. He said that he has a lot to learn; he is willing to do that, but needs help on what to learn.

> since fixing stuff may require a more specialized expertise and knowledge

That's why sign-offs from reviewers are "required", no?

It seems the correct mentality to encourage people to do hack on the kernel, show that things are possible, they are "working on my machine", and then ... yeah, they will need to go through the usually length process of getting it merged.

But starting with "synchronization between different teams" usually gets shrugs and "let's get back to it soon"-s.

One of the advantage of big monorepos (and Linux really is just one big "monorepo") is that you can change $whatever as long as you also fix everything that depends on that.

Sometimes this is just simple stuff, like adding a function argument, changing a type, or something like that.

"Break whatever" is a bit of a crude way of putting it, but yeah, it's definitely an advantage IMHO. Not just for the final patch, but also prototyping, experimenting, showing ideas, etc.

And yet, Rust is the exact opposite. Why try to make the two meet? The argument for Rust in the kernel over memory safety can be remediated by better memory safety tools for C can't it? (Not that the Linux kernel project doesn't have any already.)

Why not devote the time to writing a better memory safety tool for the Linux kernel (or C in general) rather than keep trying to force two disparate cultures and ideologies to meet in some fantasy middle?

ThePrimeagen actually had an interesting opinion on this recently too, worth a watch: https://www.youtube.com/watch?v=62oTu9hjxL4

Oh interesting. I was expecting to see nonsense in that video (probably triggered by the first line -- C and C++ are basically the same thing, in that you can, and folks did forever, implement all the C++ fancy stuff in C -- just because something isn't in the core language, doesn't mean it wasn't done).

But actually it's suggesting how I've been wanting memory to be managed for a couple decades (and have achieved long ago in C projects). Basically some sort of "domain-orientated arena allocator". A simple example is in a network server -- you receive a request from a client, do a bunch of stuff to service that request, and now you're done. Please blow away the memory used to service the request, thanks, all at once. Of course it's not quite that simple because there will be some "chaff" objects thrown off in processing the request that we need to keep around for a while later (e.g. for logging to drain).

I suppose the reference to Jonathan Blow should have been a clue that this would be something worthwhile.

Rust can do these things. It is maybe true that culturally many Rust programmers do not. There’s both good and bad reasons for this.

> RAII and smart pointers and borrow checker are all signs of "individual element thinking"

I'd call it object-oriented thinking. To me Rust and modern C++ are attempting to "OO-ify" systems programming and I think you get push back from folks who view their resources more holistically.

I took my time to watch part of this. I don't entirely agree, however. I'm not really a large systems programmer (I'm a scientist, actually). I really do like the group oriented thinking, but it does seem like there is space for "individual element thinking" at times. This sounds a lot like the philosophical notions of reductionist thinking vs. holistic thinking which is something I think about a lot when it comes to science and physics in general. The way of thinking I've come to value most at this point in my life for understanding the world that might sound a bit silly applied to this is a synthesis (so called hegelian dielectic), which is this case for me means that reductionist thinking and holistic thinking are not really opposites but are simply modes of mental modeling, and can be apply in different ways and different times to your code. Sometimes, it is valuable to have simple, self-contained types which is reductionist or individual element thinking. However, nests of pointers are always much more complicated and unnecessary than programming at the system level for a group of related objects. Which you use depends on the context, and I don't really like identifying them as opposing forces more than different modes that can be drawn from at different times.

That said, most of the time, I find myself utilizing group oriented thinking in my codes and I avoid atomistic, reductionist thinking whenever analyzing a problem at a first pass, but reductionist reasoning does help at times, it just depends on the problem and the context. It is also true, unfortunately, that in science teaching at least we teach students to be reductionist first, and then that reductionist thinking clouds their understanding and it is something young scientists need to break out of at some point, and some never do break out of it. May be in that way it's similar to what this person refers to here (n vs n+1) I just didn't also get stuck thinking reductionism is always bad and avoided it as a rule but I draw from both sides, so to speak.

There are dozens of Arc implementations in the kernel. Atomic reference counting is an important way to manage lifetimes and is used extensively in things like making sure that shared resources (files, pages, …) are managed correctly.

The rust notion of lifetimes is one of the most c++ things I've ever seen. I've always assumed it grew out of the same culture that produced c++ smart pointers and RAII.

> I can't say I've ever seen someone manually implement Arc in C

In C++, shared_ptr in the standard. But this was common even before the c++11 standard introduced that. I've rolled my own before, more than once. Microsoft ATL had CComPtr.

It wasn't surprising to me either that rust came from Mozilla, which made heavy use of COM, an object model that is very heavily based on reference counting.

I think that's misunderstanding their point a little bit?

I agree with them that in any well-written program (in any language really, not just C and C++) the ownership and lifetime constraints are not necessarily enforced by the compiler - that is, they are not expressed through types or even necessarily through code at all - but they are definitely existent in the design and behaviour of the system. If the programmer cannot explain who holds what and for how long, and therefore when it's safe to read or write particular resources, then that isn't well-written code IMO.

> I can't say I've ever seen someone manually implement Arc in C

...an Arc is literally just a shared pointer. It's in the name, Atomically Reference Counted. Reference counted resources that use atomic operations to adjust the count are a dime a dozen in C projects in my experience.

Rust did not invent the concept of a multiple-reader/single-writer lock either, e.g. the Linux kernel has the `rw_semaphore` type. I don't understand treating types like this as something arcane simply because they're given an explicit tag in Rust.

Sure! I'm just disagreeing that "C is anarchy because anyone at any point can do anything". You can't do anything, Linus will eventually yell at you in an email. To me, the only philosophical difference is that Rust wants to automate Linus

> but doesn't unoptimized C map closely to 1:1

What is a variable in C? A register? A memory location? The language doesn't have basic concepts needed to map anything 1:1 to assembly and the ones it has usually come with half a dozen standards worth of required error handling, because having single instruction features like sqrt return -1 on error wasn't enough.

Unoptimized C is not something anyone actually uses. And it maps less obviously to assembler than C with some optimizations, because C compilers in no-optimization mode generally do brain-dead things like allocating all variables on the stack.

ADTs and such don't actually make the mapping less obvious. Async kinda does, but again it's not hard to have at least some mental model of how an async function will turn into a state machine implementation. C, C++, and Rust are all about equal in terms of how well I can predict how a given function maps to assembly, which is that if I care, I need to check, but I'm rarely completely bamboozled by what I see.

> I get that GCC and Clang does all sorts of optimizations, but doesn't unoptimized C map closely to 1:1?

Nope. There's actually a number of "optimizations" that get applied to "unoptimized C" code. For example, gcc decides to apply even/odd mathematical function laws to the math library functions even with -O0, and both gcc and clang are very happy to throw "unused" code at -O0 that prevented me from doing jump table shenanigans.

C fundamentally has no idea of the distinction between registers and memory, and this is probably the most important distinction in modern assembly languages. It's especially obvious when you get to exotic architectures that have thousands of registers and a relatively thin memory pipe. Making a C compiler get out the assembler that you expected is a lot trickier than you might expect, and when you need exactly some assembly, you'll find that most compiler engineers will tell you "the compiler won't guarantee that, please use assembly" while the people trying to do so often end up spiraling into a rant about how compiler writers are idiots who can't write working compilers because it won't give them the assembly they need.

> I thought the thing that was making it not map 1:1 was actually all of the extra features in Rust, like the ADTs and async and all of that. Is that not actually the case?

People use a variety of different definitions of "map 1:1" that makes it hard to really answer your question for certain. What you seem to be getting at is the notion that C's ABI is predictable. But there are plenty of C features whose mapping to assembly is as unpredictable as Rust's ADT or async features are: C's bitfields are the most notorious example, but I'd throw in variable arguments, atomics, and the new _BitInt into the mix. Which is to say, if you're an engineer for whom this stuff matters, you'll know how the compiler is going to handle these constructions for your targets of interest, but that's not the same as saying that those constructions will always work the same way on all targets.

You are probably writing your comment via lots of software written in C++. It's a great language with lots of flaws introduced by legacy decisions that would be made differently today.

You definitely need to be able to build the kernel that shipped with the distro using tools from the distro. Thats basic table stakes. Needing newer tools for building from the latest upstream repository is fine though

Distros have historically been very concerned with "real development"!

Debian, for example, packages all of the tools and libraries needed to build any package in Debian, so users can easily modify and recompile any package. Because there are a lot of packages in Debian, it's become a great, stable, vetted source for general-purpose compilers and libraries.

Rust really is an outlier here -- its marketing has managed to walk a delicate tightrope to be considered "stable" and "mature" enough to use for important projects like Linux, while also still being new and fast-moving enough that it's unreasonable to expect those projects to use anything but the most recent bleeding-edge nightly build. And that will create problems, if only for distros trying to build the kernel binaries that they ship to their users.

We might get a stable and performant upstream driver in that case

You're welcome.

> Any concerns of the same kind of thing?

Here's the canonical list: https://github.com/Rust-for-Linux/linux/issues/2

There's a lot, and I don't know the status of many of them, personally. But I don't see anything there that I know is not gonna work out, like for example, they aren't using specialization. Most of it feels like very nuts and bolts codegen options and similar things.

That said, back in August, the Rust Project announced their goals for the second half of this year: https://blog.rust-lang.org/2024/08/12/Project-goals.html

They say that they're committed to getting this stuff done, and in particular: https://rust-lang.github.io/rust-project-goals/2024h2/rfl_st...

> Closing these issues gets us within striking distance of being able to build the RFL codebase on stable Rust.

So, things sound good, in my mind.

The kernel doesn't offer a stabke ABI for modules regardless of Rust. So what you're describing is already the case.

The RfL project lead is Miguel Ojeda. He hasn’t gone anywhere.

I think that was the lead for the group getting RUST in the kernel.

RUST itself is still strong outside of the kernel.

Google funding is a positive signal?

https://killedbygoogle.com/