Hacker News Reader

> It would make me 10x more likely to work on the kernel

Seconding this. I don't think I would ever touch the Linux kernel with a twenty-foot-pole in C. In Rust? Maybe I'd give it a try and contribute one day.

Knowing the exact set of constraints on my code, having them enforced, not having to hold the entire system in my head just to avoid a catastrophic memory error. These things make contributing to a project - especially one as incomprehensibly large and complex as the Linux kernel - orders of magnitude easier.

its not that bad writing kernel code - just remember if you get anything wrong you'll hose your filesystem beyond repair ;-)

what about interacting with the existing kernel code base? would data coming back from the kernel into rust space need to be wrapped to provide safety guarantees? or would it be necessary to turn safety features off?

a bit confused about this.

That is what this patch series is about. It will require using unsafe code to some degree, yes.

So... it doesn't really impact this discussion other than "shouldn't be an issue". I'll try and give a summary of what's happening technically, but HN is frankly the wrong form for a "how to use the C ffi in rust" tutorial. Also a big disclaimer that I don't know what precisely this project is doing, so I'm talking about C/Rust projects in general.

Rust talks the C ffi really well. It can call external functions that follow the C abi the same way it can call native unsafe functions [1]. You can tell it to layout a struct the same way that C does, etc.

Because calling the C abi requires unsafe code, it's common to provide wrappers around the C abi that are safe against missuse. I.e. that make it so that the only way to call C functions is the correct way. This is doing things like making it so the only way to get a `struct` is to call a (safe) `new` function that calls the (unsafe) C initializer internally, and exposing the C "methods" on that struct (that expect it to be initialized) as safe methods on the rust struct that internally call the unsafe C functions (and they can do so because they know the struct has been initialized). Obviously for any particular C api you have to look at what it requires to be called safely, and then figure out how to encode it in the type system, but that's usually surprisingly easy.

Calling rust from C doesn't really require any "unsafe" code (other than the fact that C is basically a giant unsafe block by nature), because the assertion that you're calling it correctly happens on the C side of things, not the rust side of things. Just like rust can call C abi functions, rust can make it's functions follow the C abi by simply saying

   extern "C" fn foo()

instead of

   fn foo()

But many of the data structures you might pass from C to rust will need a wrapper to use "safely". E.g. if I pass a doubly linked list, it's going to need raw pointers more or less by nature (at least if rust wants to be able to mutate it), and someone is going to need to do a similar wrapping thing where they expose some functions that correctly work with the list, that internally use unsafe, but expose a safe api.

[1] So what unsafe here means that the compiler doesn't know that the function is safe to call, so you have to tell it "I checked and how I'm using it is fine" by putting the call inside an unsafe block. This looks like the following. Note that you can also have unsafe native rust functions (e.g. if you want to index an array without checking the array bounds that's an unsafe function implemented in rust)

    unsafe {
        c_function_here(arg1, arg2)
    }

thank you for this. this helps.

but let's say one is writing a filesystem in rust, so you're implementing most of the functions in "struct file_operations", and moreover you are passing "struct inode" , "struct page" etc ... back and forth between c and rust. with such heavy handed interaction, aren't we basically doing c in rust by necessity of the interface? by which i mean "unsafe" the way you defined it?

are there examples where you see a clear win?

You'll have to excuse a bit of unfamiliarity with linux internals here, I'm taking a guess, but I expect that filesystems are an example where you would see a clear win.

My assumption would be that a file system is calling the same methods on a few different objects repeatedly. E.g. "read me some bytes from this page" or "get the id of this inode". For each of these APIs you once write a small amount of unsafe code that encodes into the type system "and this is how you can call it safely", and then you repeatedly get to make use of that code with guarantees that you aren't making any mistakes that are too terrible (logic bugs still exist obviously, which on a file system could delete or corrupt files, but you aren't going to corrupt some random kernel memory by accident). That's a pretty big win in my mind.

Meanwhile file systems probably include a lot of non-ffi things I think rust is substantially better for too. Like handling of a ton of different error's (oh no, the disk failed to give me bytes. Oh no, these bytes make no sense. etc) in the codes "happy"(ish) path. And like parsing data structures out of bytes (correctly). Tracking exclusive access to various resources. Implementing compression algorithms. Etc.

The case where you would see the sort of issue you're discussing is where all the code is doing basically unique ffi calls, so you don't get any reuse out of safe abstractions. I don't know of any great examples of this, maybe things like boot sequence code where you're running a lot of unique things exactly once to initialize the hardware?

thanks gpm for taking the time. let's see how it pans out. rust is definitely interesting.

now let me not impose on your kindness further and go learn a little rust.

Networking, especially wireless (as it's more complex and potentially more dangerous: attacker needs not even a wire).

Google is developing a bluetooth stack in Rust.

[1] https://blog.desdelinux.net/en/google-desarrolla-una-nueva-p...

hmmm ... good point.

>- I think it would increase the general productivity in kernel development. A better kernel helps everyone out.

Don't you think the massive increase in compilation time would negate any productivity gains and probably decrease productivity overall?

No, because

- I don't think it will be massive.

- I especially don't think it will be that large for incremental builds, which is the main thing that matters. (But I'm not involved in this project, so I don't actually know how incremental the builds are...)

- My experience is that the vast majority of programming time is spent fixing mistakes, not compiling. Rust reduces the amount of time spent fixing mistakes a lot more than it increases time spent compiling.

- Rust moves many errors early in the compilation process (instead of when you try and test your code), which reduces iteration time instead of increasing it.

I'm not involved in this project, but I imagine it's at a point where you could get some numbers for the fixed overhead that adding rust adds to the compile times. I'd be interested in seeing those numbers.

Have you compiled the Linux kernel before? I doubt Rust will be the bottleneck, it's a massive project - a tiny fraction being in Rust will be a blip.

Pretty frequently, yes. It doesn't take that long on modern, moderately-powered devices. Even when I was using a mid-tier device from a decade ago, compilation time was still around half an hour, which was less than most Rust projects I've encountered are on modern and reasonably high-end hardware, despite doing much more and being much larger.

Yep, that sounds exactly right to me - about 30 minutes on older hardware. That's very odd to me that you have 30 minute rust build times - as someone who works on a rust project professionally, with 10KLOC, that isn't my experience at all. If Rust gets into the kernel I would expect it to account for <1% of the code, so even if it were 100x slower to compile, which it isn't, I don't see it having an impact.

Man, 10KLOC is a very small project. Obviously it compiles quickly. Linux kernel is almost 30 million lines of code.

Yes... exactly. The Linux Kernel is 30 million lines of code - so how exactly will some minimiscule-by-comparison amount of Rust code slow down compile times considerably?

If it's always going to be miniscule then why bother? If it has potential to grow to something not miniscule then it's important to it compiles quickly.

1. Even if we start today, and all new code is written in Rust, it will be years before Rust has an impact on compile time.

2. A relatively small portion of code can account for the majority of exploited vulnerabilities.

3. Nothing about Rust is fundamentally slow with regards to compile times. There's plenty of time to work on it, and we've already seen significant efforts make headway.

Compile times are a nonissue in any practical sense for this work.

I wrote 10kLOC last week. Most of the code didn't require much extra thinking, it was mostly a translation of an old TypeScript project to Zig. But measuring compile times with 10kLOC is really not a good argument.

Well, 15KLOC, and of course not including dependencies. But my point was that there will be a tiny, tiny amount of Rust in the kernel by comparison to the 10s of millions of lines of C code. Rust would have to compile radically slower than C, like hundreds or thousands of times slower, in order to be a limiting factor.

I've been coding predominantly in rust since 2018 and have never encountered a build time > 10min.

Worked at a place where our c# build times for a large project, many millions of loc, were 40 mins or so.

"Productivity" is notoriously hard to measure in software.

While slow compile times may slow you down, and hence reduce your productivity, if the compiler prevents hard to fix bugs, it still may increase your overall productivity. Consider things like https://hacks.mozilla.org/2021/04/eliminating-data-races-in-...

> Overall Rust appears to be fulfilling one of its original design goals: allowing us to write more concurrent code safely. Both WebRender and Stylo are very large and pervasively multi-threaded, but have had minimal threading issues. What issues we did find were mistakes in the implementations of low-level and explicitly unsafe multithreading abstractions — and those mistakes were simple to fix.

>

> This is in contrast to many of our C++ races, which often involved things being randomly accessed on different threads with unclear semantics, necessitating non-trivial refactorings of the code.

Maybe the C++ was faster to compile, but in the end, fixing these issues took more time. There were more of them, and they were harder to track down.

Nobody truly knows the answers to these questions in the general case yet, of course. My point is just that "slow compile == bad productivity" is not inherently true.

Faster compile times are, of course, always desired no matter what.

> "Productivity" is notoriously hard to measure in software.

We can look at history and results. In these terms, C (and perhaps C++ to some extent) is, I believe, the only productive programming language for making the low level parts of non-experimental kernels.

As far as it can be publically measured, Rust so far has proven itself for application programming in a certain niche and not much more. It would be cool if it could prove itself in kernel space too -- we certainly need less system crashes caused by bad kernel-level code. Curiously though, it has been a very long time since I've last bumped into such a thing in Linux. This makes me suspect that Rust is trying to fix a problem here that's already been fixed in another way.

As for ease of coding, C seems like a massively easier language to learn than Rust. But I might be wrong there. Any data about that, I wonder?

Yes, that is true. But the only way to get that data is to do it. This work is one part of doing that. Someone has to be first :) (There are of course a ton of kernel-level things in Rust that don't pass the "non-experimental" bar for various people. As always, depends on exactly what you mean.)

> Any data about that, I wonder?

Possibly one of the only things harder to measure than productivity is ease of learning, haha! I had programmed in C for decades before Rust even existed. We do have a lot of people who say that they think Rust was easier to learn for them than C was. And of course many who believe the opposite. Not sure anything is conclusive in any direction. For example, it's quite possible that some people find C easier, and some people find Rust easier, and there will never be a clear winner.

Time will tell.

I've got some vague opinions about "easier to learn" that I'd like to hear some disagreement on, to help me work out my thoughts more. Please forgive me if I'm not very clear here.

I don't know if this is what you mean or not, but I've seen a lot of claims that a language is "easier to learn" that seem to be considering "learning a language" as a valuable topic on its own, separate from "learning to write and maintain correct nontrivial programs in the language", and that seems wrong to me.

There's a part of this idea that does seem valuable to me, in that at the beginning of your learning process, there are a lot of benefits from being able to quickly get to a point where you can successfully write small programs that do something. It helps your motivation. It helps you reach some amount of productivity faster. When a language is better at early onboarding, it's more-useful to people who have smaller needs and more-constrained use-cases. Python being so easy to learn to glue together some libraries makes it a fantastic, valuable tool for many people.

The part of this idea that I really disagree with is how it applies to non-trivial, non-beginner use-cases. There are topics and skills that languages vary in their coverage of, but that you still need to learn about and deal with anyway for many types of programs. Memory management, resource handling, ownership and sharing, concurrency, nullability, error handling, composition, organization, abstraction, refactoring, testing, debugging, etc. A language including more or less that directly addresses these topics doesn't necessarily mean you won't still need to learn them.

To me, the relevant question isn't "Which language is easier to learn in isolation?", but instead "Which language is easier to learn to implement safe, performant, efficient, reliable, concurrent code with?".

If you take a new engineer who has "learned C", how easy is it to train them to get their rate of memory safety errors, thread safety errors, missed error checking, etc. down to the same rate as you'd get from a new engineer who has "learned Rust"?

Without tooling support like you get from Rust, you instead need to learn safe idioms, learn strategies to minimize your exposure to errors, train yourself to always always check everything at all times, learn how to write tests to discover mistakes you've made, learn how to use a collection of third-party tools you can use to approximate some of the benefits of Rust's compile-time checking, and train yourself to always use it. That's not "learning C", but it's still required in order to implement something like Linux.

Rust's bet is that there are ways to reduce the overall complexity of everything involved in implementing high-reliability high-performance systems by moving some of that complexity into the language. If you don't think it's accomplishing that goal, that's fine, but make that case directly.

I agree that the C programming language is smaller and easier to learn in isolation. It's not so obvious to me that something like "C + Valgrind + ASan + TSan + UBSan + ..." is easier to learn than Rust.

On the other hand, for many classes of errors that C offers no help with, Rust's compiler will directly point out where you've made a mistake, why it's wrong, and often offers advice on how to fix it. When learning a new language, having that kind of tooling support universally available is extremely helpful.

To be clear, Rust doesn't handle everything, and there's still a lot of benefit you can get from dynamic analysis tools, fuzzing, etc. There are also levels of reliability and assurance that aren't currently feasible with Rust. Rust has a long way to go.

The point I think I'm trying to make is that Rust really raises the bar in a meaningful way. There's some nonsense and awkward bits in Rust, but a lot of what you need to learn to be effective with Rust are things that you'd need to learn anyway to be effective at this level with C, and I think it's easier to learn those with Rust's help, and it's significantly easier to build systems with a much lower rate of these problems by using Rust.

Sorry for the length, and lack of organization. This has been rattling around in my head for a while, and I wanted to get some thoughts out in writing.

100% this. People forget when using dynamic languages they are trading up front cost - its easier to write the code but harder to test. In trivial or exploratory coding the tradeoff can be good, but it is a tradeoff.

That being said, using rust can be really nice for exploratory coding. If don't worry about edge case (use unwarp()/panic!) and don't worry about memory efficiency (use clone()) it still produces fast, memory efficient code.

This will also be a bit messy, as I am on my phone and don't have much time to write this down.

I have 10 years of writing Java on both small hobby projects and massive systems on my job. I dipped my toes into C/C++ a few times and I can write hobby project sized code that runs, but I can never really trust that code like I can the Java code that I write.

I had this issue especially when writing my first TeamSpeak Plugin: I could never be certain if I had to free the data or if TeamSpeak would do it for me after my function was called. I had to look into the documentation, as TeamSpeak is closed source.

About one and a half years ago, I finally sat down and learned Rust. I had tried to learn it several times before and always ran out of motivation.

This time, I got to hobby project level proficiency within a month or two. It was really fun and I rewrote several Java Spring projects in Rust's Rocket.

As Rocket starts just about instantly, even with the long compile times, the Rust rewrite takes less to compile than the Spring projects took to start up.

Coming to the point: I am already more confident about my Rust code than ever was about C and would even go so far as to say that I trust it more than my Java code, despite the big difference in experience.

I believe that I am ready to jump into a big Rust codebase and become productive as soon as I have understood the business-logic of the project in question.

I doubt I will be able to say the same about C/C++ in the near future.

If you are talking speed to production ready code then rust is really productive. The rust tooling picks up a lot of errors and leads you to spending more time fixing coding issues rather then compile to test your code.

How much is 10 x 0? ;)

What makes you assume the probability of contributing is 0 though

> It would make me 10x more likely to work on the kernel, just because I enjoy programming in rust more than I enjoy programming in C

That sound was a thousand people nodding in unison

And it would be a million people if it were Python. That doesn't make it a good idea to encourage submissions to the kernel in Python. There is much more to kernel programming than the programming language. C might be inconvenient and lacking expressive power, but if people can't handle pointers and goto, what mess are they going to make with different address spaces and interrupts?

> C might be inconvenient and lacking expressive power, but if people can't handle pointers and goto, what mess are they going to make with different address spaces and interrupts?

I have terrible news for you. People (well, humans, and I don't see anybody else signing up to maintain Linux) cannot in fact correctly handle pointers and goto. That's why they keep making mistakes.

It's actually to be hoped that Rust can usefully express constraints it has today to prevent some of those problems onto things like address spaces. It'd be great if say, driver code which can confuse a virtual address with a physical one just won't compile rather than compiling and then mysteriously not working as expected or causing occasionally BUG() reports.

> but if people can't handle pointers and goto

Mitre shows that in fact people can't handle pointers and goto