Next steps for Rust in the kernel
lwn.net
lwn.net
Glowing endorsement from Torvalds right there. I'm really curious about how this will work out, Rust is my favourite programming language by a fair margin and seeing it being used more and more in places where C or C++ has traditionally dominated is very exciting. The future is looking bright (for those of us who like Rust).
Heck, so my secret project of slowly replacing every Perl script in the kernel with C might even get approved...
https://www.youtube.com/watch?v=yA_wUiNuhSc
C isn't a good software engineering language but it might be better than Perl. Or in this context, maybe not.
User-space part only. Kernel is C++.
I don't think that Rust has the same issue: it has a clear (albeit potentially complex) vision of what it is trying to do, which is distinct from C and more consistent than C++.
Of course, there are areas where Haskell is stronger than Scala (hint: modularity, crucial for good Software Engineering, is not one of them). And Scala has its own way of doing things, so just imitating Haskell won't work well.
Examples of this "better Haskell" are https://typelevel.org/cats-effect/ and https://zio.dev/ .
All together, Scala may be a better choice for you if you want to do Pure Functional Programming. And is definitely less risky (runs on JVM, Java libraries interop, IntelliJ, easy debugging, etc...).
None of the other languages you mentioned are viable in this sense (if also you want a powerful type system, which rules out Clojure).
I agree that Rust's identity is pretty clear: a modern language for use cases where only C or C++ could have been used before.
It is _the_ language where most of the Functional Programming in the industry takes place (that's not to say that Haskell, F#, OCaml or Clojure are bad, they just aren't as widely used). And that also means that there are lot of open job positions available.
I like FP and Scala comes with some awesome libraries for concurrent/async programming like Cats Effect or ZIO. Good choice for creating modern-style micro-services to be run in the cloud (or even macro-services, Scala has a powerful module system, so it's made to handle large codebases).
https://typelevel.org/cats-effect/
The language, the community and customs are great. You don't have to worry about nulls, things are immutable by default, domain modelling with ADTs and patter matching is pure joy.
The tooling available is from good to great and Scala is big enough that there are good libraries for typical (if not vast majority of stuff) and Java libs as a reliable fallback.
Jokes aside, even Scala proper sees itself as an experimental testing ground more than a complete idea realized. That's why dotty became Scala 3, with a huge amount of backtracking.
It pushed the state of the art across the JVM landscape, but I think it's lost its purpose with Java, Kotlin, and Clojure around.
That's just something that Java, Kotlin, nor Clojure don't offer. And there is significant demand for Scala (and its pupose) in the industy (it just not may be as huge as Java or other more mainstream languages).
As in: keeping it clean and well structured is impossible and the impressive flexibility of the language is huge downside on larger projects.
I feel qualified to say such things as I've worked on a billion-line perl codebase before.
> As in: keeping it clean and well structured is impossible and the impressive flexibility of the language is huge downside on larger projects.
Perl gets quite unfairly picked on here, to the point where it's basically a meme. It's not like, just to pick a somewhat popular language, Ruby doesn't provide plenty of ways to write completely unreadable code that can only be understand by running it through irb, if even then.
I agree that there is plenty of bad Perl code out there, but there's plenty of bad code in any language that achieves general popularity for a time. Disciplined use can produce perfectly clear and readable code and the module system is perfectly adequate for separation of concerns.
The language does give you an absurd amount of rope to hang yourself with though. If I were managing a large Perl project today I'd have an extremely restrictive style guide and would enforce as much of it as possible via pre-commit hooks.
> I feel qualified to say such things as I've worked on a billion-line perl codebase before.
For a language that permits such densely expressive code, that's appalling. I'm so sorry.
This was surprisingly doable? Marshaling/unmarshaling Perl objects into Python ones isn’t difficult these days, and you can run Perl code via FFI so the transition can be piecemeal.
I have no opinions on Perl as a language (some constructs / patterns felt a bit hard to think about, but I’m far from an expert), but ultimately it’s a high level language and has been left behind tooling wise to another high level language, so it’s time to move. For us, one motivator was that having a large legacy Perl codebase (even one running well for years) hurt recruiting.
I feel like - opinion, not based in fact, I've never worked on any codebase >1M LOC - the only way you can keep a codebase like that maintainable is by very strict code style and architecture rules - e.g. no functional programming, no feature X, no feature Y, strict code style rules, strict architecture rules, etc.
Personal experience. We have some java code from 15-17 yrs back. About the time “design patterns” and all the other “pattern” stuff was big in everyones mind. It’s a mess of abstraction for simple tasks. Couple all that with the fact that class instantiating is possible via reflection and at this point people maintaining it have kinda given up. It works - don’t touch it.
This isn’t an indictment of Java. Just resume driven design.
Programmers are highly trained like to see the world as if everything can be cast the same way. While that IS a good way to see a large variety of software problems, when you isolate down to a companies' use cases, it quickly goes away in the pragmatic sense. It then becomes something the company is paying for because they chose Java and Java developers like that sort of thing.
Long ago I worked primarily in perl and ... yeesh, I recognize maybe 30% of it when looking, and that's the directly-c-adjacent stuff.
To be fair if I walked away from Rust for a year I'd probably be similarly lost.
The Amazon retail website (amazon.com and related versions for various countries) used the Perl Mason templating system (https://masonbook.houseabsolute.com/) to allow various teams to build web site components in a performant and secure way.
It worked quite well since it ran quickly and isolated components so that only one small change could be made without breaking other components.
What, how?
Thats bigger than windows linux llvm chromium and 5 other compilers combined according to google
Linux is a pretty small codebase compared to most proprietary codebases in large companies to be fair though.
The commands supported by `sh` are... let's keep it at "fossilized from somewhere in the 80s", and so the only alternative was perl which has a ... history of being not exactly user-friendly to code for.
[1] https://www.debian.org/doc/debian-policy/ch-files.html#s-scr...
1 - http://archive.debian.org/debian-archive/debian/dists/buzz/m...
fn eat(food: Edible) just takes an actual Edible. Afterwards the food is gone, perhaps this function stashed it somewhere, or gave it away, but you don't have it any more† so maybe it was Dropped (destroyed)
fn sniff(food: &Edible) -> Smell this time our function only wants an immutable reference to the Edible. The food isn't changed in any way by this predicate, it's still yours, and now you also know how it Smells.
fn lick(food: &mut Edible) -> Review but this function modifies the Edible via the exclusive reference. You still have the food, and now you've got a Review, but the food may be changed, perhaps irreversibly by being licked.
† If Edible is Copy then Rust will just copy it, so the caller still has it, types which are Copy are typically small, and they can't implement Drop, so if they were destroyed nothing happens anyway. An integer is Copy, a String isn't Copy, and neither is a File, or a network Socket, or whatever. You can make your own types Copy, if you want, and if they qualify.
I was referring to the `Box<dyn Trait>` one, as I'm at least 90% sure `Box<X>` passes a owned, dynamically-allocated X by reference (ie, Box<X> is a pointer to X).
If the function only wanted a reference to it, that would be &Box<dyn Bird> and then I wouldn't give up ownership by passing it to the function. But that's also a very strange thing to ask for, since we can instead pass a reference to a Bird, no need to Box it.
No, a Box<dyn Bird> is a pointer to some sort of Bird on the heap. Unless Rust has wildly changed how it implements function calls/stack frames, local variables are stored on the stack. A pointer is a reference to the thing it points to (in sense used in the phrase "pass by reference").
> then my call moves the Box<Goose> into the function, I don't still have that Goose
Yes: you passed a Box<Goose> by value, you passed the Goose it points to by reference, and you transfered ownership of both.
> are you promising to deallocate and otherwise clean up the object when you're done with it, versus are you promising that the caller can keep using it after you return.
Whether I ask for a Goose or a Box<Goose>, it's mine now. The caller no longer has it, it's not just that they shouldn't use it, they can't, the compiler won't let them. Since it's mine, it is now my responsibility as owner either clean up, give it to somebody else or explicitly leak() it.
Rust doesn't promise how exactly this is implemented, and it is likely to be different for an object of 4 bytes than for an object of 400 bytes.
It is true, though, that there are differences. With the generic, you can refer to the type again later, whereas the type is anonymous with `impl Trait`. And `dyn Trait` gives you dynamic dispatch. But in my experience there are usually some differences between the multiple ways to do something in Perk as well.
Or should I learn C/C++ first and then learn Rust?
I've just done projects in Python,JS & Java so far.
If you want just to "play" and do some tiny projects as low level as reasonable - go for learning C first. You'll learn a lot - and then when you learn rust, the reasons why things are done that way will make a lot of sense. Also, so many low level utils are written in C - having a basic knowledge of it is really useful. But don't try to build a big complex thing in it unless you have a really good reason to (or you fall in love with C. Some people do!)
If you want to actually build something big and complex and for real world use - probably go for Rust.
If you want to build a big production system using an already established tech that's based on C++, learn that. (Eg. one of the C++ based game engines, or a C++ GUI framework, or whatever).
C++ has everything C has plus many layers of complexity deposited over it that can make you more productive but also obscure your relationship with the machine. If you're going to go on to Rust I wouldn't bother with C++ at all right now.
If you're coming from Java, the most difficult thing you'll need to learn is memory management, as all of those languages do that for you. The other difficult thing is threading, but that's 1) unnecessary and 2) not actually that different from Java.
Rust forces you (unsafe aside) to do memory management in one specific way, and verifies it in the compiler. C and C++ let you do whatever you want, though C++ encourages (without requiring) a Rust-like approach.
I think Rust will probably be better if you want to have a constrained environment that tries to prevent mistakes. C will be better if you want to explore on your own, learn from the mistakes, and thus understand why Rust wants you to do particular things. C++ frankly doesn't have a whole lot to recommend it unless you are working with a C++ codebase, already know C, or really need one of its features.
I'm coming from a largely Java background (though I program a lot of C and assembly as a hobby), and this statement is very true. Getting used to the borrow checker and reference lifetimes has been the largest obstacle so far. But once that is understood, the rest of Rust is basically learning the idioms and ecosystem.
I will likely change how I write C based on my experience with Rust's memory ownership model..
Moving from high-level languages to Rust is going to be a nice punch in the face :)
Some people suggest to learn C/C++ but that ultimately depends on the resources available. If one has infinite time, certainly it's ideal, but with limited time, I think learning Rust will already be very demanding.
My suggestion: find some small projects you find interesting. Follow the usual Rust learning path (reference book first, then find other books), and apply it to the projects you've chosen; after an year or so, ask the question again.
Other small suggestion: don't start from book that pretend to teach Rust by learning another topic, because the amount of Rust that can be taught in 100 pages is useless (a lot of books pretend to do so). On the other hand, there are way more books to find fun projects/topics than there were an year ago.
IMHO, if your end-goal is Rust, learn Rust directly.
The angle I am coming from: I've been doing C++ for 20 years and Rust for over a year.
Learning C/C++ can provide great value on its own, through gaining knowledge of how lower-level programming primitives work, but many of your C/C++ skills won't be subsequently applicable to Rust, except perhaps for core concepts such as a stack, a heap, owning memory, a `struct`, etc. This is caused by the fact that Rust takes many programming abstractions much _farther_ than C and in a substantially _different_ direction than C++. For example: where C uses manually managed pointers, Rust uses compile-time checked ownership and borrowing; where C++ uses inheritance, Rust uses generics and composition; where C uses void pointers ;) and C++ uses pure virtual functions, Rust uses traits and `where` clauses; where C and C++ use... a programmer's eyeballs to guarantee multi-threaded correctness ;), Rust uses compile-time "marker" traits; and so on.
Don't get me wrong: C and C++ are extremely able, powerhouse languages worth your time in their own respect, it is just that most of the C/C++ -specific knowledge you invest your time to learn beforehand won't be directly applicable to Rust later on, because Rust just does things in a significantly different way than C/C++.
Oh, don't forget to RTFM - Rust is easy to learn by reading (in order to understand its unique concepts), but Rust is hard to learn by experimentation alone (trying 100 different syntax variations to see what compiles).
You don't need to 100% the language, a few months experience will be be handy for the rest of your computing career. It'll be a perfectly reasonable stepping stone for reading C++ without really needing to learn all the details to write it.
As much as any language can be: C is still the lingua franca, a relative common denominator. As you explore different parts of the computing environment around you, you'll run into it constantly.
Learning about python? C. Digging into how your program talks to the OS? glibc and the man pages and most examples will be C oriented. Any documentation ever explains how something is laid out in memory? It'll be explained in C structs. How function calls work? The assembly will be explained in terms of C code and C's calling convention. Any language's FFI? C. Digging into the JVM, browsers, Javascript? It's C++ but you'll know enough to eyeball the code.
This is mostly about all the things you'll want to be able to read about well. Anything you'll want to know has been done in C or C++ — Nothing big exists that you'll want to explore is mostly written in Rust or Zig yet. And there are still things that can't reasonably be yet either (oversimplification).
There's no real problem starting with Rust, especially if you have some experience with other languages. The biggest difference is going to be getting used to dealing explicitly with memory, since all the languages you listed have a garbage collector; but this is true of all of C, C++ and, Rust. Be kind to yourself -- don't expect it all to click instantly.
Rust is it's own thing; I personally really like it but going from C to Rust is harder than just going to Rust as you have to "unlearn" C paradigms.
You'll probably be "better" if you go C first, but it will take a lot of time and energy.
In the end its completely up to you.
It will teach you how a PDP-11 thinks, at best. The cache hierarchy and speculative execution are completely hidden at the level of C, to start with.
You want to learn how computers think, pick up an assembly language. (Any one will do.) You want to build portable low-level software, learn Rust, Zig, C, or C++ (in my personal order of preference).
[0] https://www.intel.com/content/www/us/en/developer/articles/t...
The performance gains of fine-tuned code is typically two orders of magnitude higher
A quick glance at Intel's Software Developer's Manuals [0] falsifies this:
>> TLB and Cacheability control: CLFLUSH, CLFLUSHOPT, CLWB, INVD, WBINVD, INVLPG, INVPCID, and memory instructions with a non-temporal hint (V/MOVNTDQA, V/MOVNTDQ, V/MOVNTI, V/MOVNTPD, V/MOVNTPS, V/MOVNTQ, V/MASKMOVQ, and V/MASKMOVDQU).
> [..] speculative execution are completely hidden at the level of assembly too
Section 18.1.13 specifically mentions side-channels for speculative execution, and there are more than 100 other matches for "speculative" across the document (some of which also refer to load barriers).
So no, these things are not completely hidden at the assembly level, and at least in the case of the (or one of the?) most popular consumer CPU architecture in the world, they are actively documented in the primary reference for an assembly programmer.
[0] https://www.intel.com/content/www/us/en/developer/articles/t...
And error recovery (that is fucking crazy on x86), and wide instructions, and I/O, and stack management, and... I am not sure C is even a good approximation for the PDP-11.
For which you'd need to be familiar with a low-level language like C.
It's a good way of helping people with no programming knowledge get past the stage of "what are those magic colourful words you type that somehow make the computer do things?".
https://tobiasvl.github.io/blog/write-a-chip-8-emulator/
I haven't made a Gameboy emulator, just learned the assembly so I could modify some roms.
You can check out my blog post[1] if you're interested how I went about it (there are a couple resources at the bottom I used to build it, including the one that HideousKojima recommended).
As for language, in hindsight, C was fine. CHIP-8 is so basic that you don't really need to worry about performance or to implement a JIT compiler unless you want to learn how to do those things specifically. Just pick any one out of the three (assembly is a bit of a weird choice though, why not write a CHIP-8 emulator and then a brand new program to run on that emulator in CHIP-8 assembly if you want to learn assembly?)
As for Gameboy, it would have more instructions, a different graphics system, sound etc. and overall be more complicated than CHIP-8. Try it out if you either feel a bit more adventurous or have implemented CHIP-8.
[0](https://sr.ht/~gotlou/chip8-emulator) [1](https://gotlou.srht.site/chip8-emulator.html)
Check this post out - Should you learn C to "learn how the computer works"? (https://steveklabnik.com/writing/should-you-learn-c-to-learn...). Spoiler alert - no.
I think summarizing that article as "no" really undercuts how diplomatic and reasonable Steve Klabnik is in it.
But is it necessary to understand now a computer works? No.
You can also check out "C is how the computer works" is a dangerous mindset for C programmers (https://steveklabnik.com/writing/c-is-how-the-computer-works...)
If, however, we're talking about the kernel then perhaps we should listen to what Torvalds says about this topic.
"People who designed C, designed it at a time where the language had to be geared towards the output, so when I read C I can think about the Assembly that it will create":
http://www.youtube.com/watch?v=MShbP3OpASA&t=20m45s
Consider that he said this in a time with C++, even if Rust was in its infancy: it is not as tied to the assembly it produces. Not by a long shot.
The issue with Assembly directly is that it's absurdly non-portable, too low level and too unstructured to do anything reasonable with it.
There's a reason that Linux is written in C and not ASM Directly.
Even interpreting this as charitably as possible--that you don't mean literally learning assembly but rather learning everything besides specific assembly syntax that you would have picked up from learning assembly--this claim doesn't hold water.
For starters, C isn't that much closer to hardware than, say, Java. The main differences you have with C are that a) C has a pointers-are-almost-integers model [1] and b) C has a more limited runtime than other languages. C is still a fundamentally based on an abstract machine semantics model, and that abstract machine doesn't have a close bearing on modern hardware.
In particular, C does not distinguish between registers and memory, and if I had to pick the single most salient feature of modern hardware you need to understand well to be able to say you understand how computers work, it's the register/memory distinction. If you think processors are mostly like they were in the 1970s--when you could say "add 1 to this memory location"--then C's shrugging off the register/memory distinction makes sense, but this is a situation that hasn't been true for several decades.
Another failure of C is that it's far less expressive than assembly. There are features in other languages that are impossible to express in C. How do you write a function with multiple entry points, as you can in Fortran? Or a coroutine, as you can in Algol? Or nested functions, as in Pascal? Or try-catch, as in C++? Or functions with multiple return values, as in Matlab? Indeed, how do you write something like JMP *reg in C, a useful primitive for state machines?
No, C does not come close to being a good description of modern processors, where "modern" means "anything that came out since I was born."
[1] Not going to open up the can of worms that is pointer provenance.
Just nitpicking, but doesn't C have the register keyword? (I know it does nothing at all nowadays, but AFAIK it did make a slight difference back in the dark ages.)
Your C compiler might conclude - even if it doesn't decide to put the variable in a register - that since the variable notionally doesn't have an address some optimisations are available which it can't otherwise be sure are safe.
[1] Okay, the official semantics of "volatile" strictly speaking are completely orthogonal to memory, but the practical effect that everyone agrees on (don't optimize this memory access) is inherently oriented towards memory, and the fact that the semantics are so vague about what it actually does is partially a reflection of the fact the semantics themselves don't distinguish between register and memory.
I don't believe this at all. You won't learn how to deal with limited numbers of registers. You won't learn about method prologues and epilogues. You won't learn about oodles of stuff.
Those things takes like an hour to learn, translating C to assembly is trivial, it takes time and will be verbose but it isn't hard at all. So even though you don't learn all the details you still learn all the higher level stuff related to assembly programming when you program in C.
This isn't true for other languages, translating arbitrary C++ or Java or Javascript to assembly is really hard, so they don't teach you to think like an assembly programmer.
I mean; you will likely end up stepping through debuggers where you will see registers being used. At least if you learned C like I learned C.
Similarly for Prologue and Epilogue.
"why does a function call have a cost" is a common question and it's solved forever after an hour googling.
If you're willing to accept, at face value, that certain easy tasks are convoluted for Good Reasons, you can go straight to Rust. If you want figure out why for yourself, start with C.
I always want to learn bottom up, and personally recommend starting with C.
A)
Rust gives you many great abstractions that you will want to use, even if you skip the standard library. This gets in the way of really learning to work with and understanding pointers and low level mechanisms. You can certainly do that in Rust, but you have to force yourself to ignore most oft the language.
B)
You will have to read and interact with C code eventually, and be it just calling out to libraries or wrapping libraries in Rust interfaces.
A solid understanding of C is critical to do that correctly.
There's also a lot of safety tooling that you rarely need with Rust, but should be comfortable with. (Like valgrind)
After learning Rust you will hate working with C (well, I do...), so it's better to do it early.
You don't want to get fooled into thinking C's abstract machine is what your actual machine is like, because it really, really isn't. A bunch of the stuff C is deliberately vague about because it wasn't portable in the 1970s is very concrete and important to understand about on a real computer. Rust's Wrapping<u8> and Saturating<i16> have behaviour that you could buy with a real CPU (indeed your actual CPU almost certainly can do Wrapping<u8> with no extra work) but C doesn't promise anything like this and names types things like "short int" which could be anything.
On the other hand, some things C makes seem obviously concretely real, are just being simulated convincingly and that's not necessarily how your CPU works at all. Nobody really promised "pointers" are just integers you can go around doing arithmetic with for example.
If your endpoint is Rust anyway, just start there. You can always learn C (and/or C++) later.
I don't think it matters (much) which language you start with. Any one of the suggested languages (or commonly suggested languages in general) will be fine. I think this applies in general too, at any levels of skill/experience, but probably matters more for someone with less experience.
Between C and Rust, Rust has better tooling and an easy to use library ecosystem. C is a smaller and easier to grasp language.
I would start with Rust if you want to write a real world project. And probably start with Rust if you just want to write some leetcode algorithms as well (but C is fine here too)
People that learn C or C++ first often end up fighting the Rust compiler a bit until they adjust their mindset.
Rust's borrowing and ownership semantics are not bizarre and novel solutions to things that aren't problems in other languages. Rust's borrowing and ownership semantics are a compiler-level reification of problems that exist in all languages, and especially multithreaded languages. This 100% includes C. You may not adopt Rust's exact solutions in all cases, but if your are programming at a system's level at all and you're not thinking about the problems that Rust exposes directly, you're in trouble and you don't even know it. Using Rust is a great way to work in an environment where the compiler is forcing you to learn how that all works, and will train you good and hard about how to think about issues of ownership and what code is allowed to touch what.
I primarily work in Go, a garbage collected language, and I heavily use the concurrency features. That doesn't mean I don't have to think about ownership because the language lacks any support for it, or that I don't have to worry about who is responsible for deallocating things because I work in a GC'd language. It means I have to think about those things, but I lack support from the language. My personal career experience means this is not a terrible tradeoff; as a result of all my time in Erlang and Haskell it is no big deal for me to operate concurrently in a manner that I know will work, because while they are different than Rust, they were also harsh taskmasters in terms of only allowing me to do certain things that work. Other tradeoffs make Go worth it for me despite this lack of language support. But the lack of support doesn't make the problems go away. It just makes it so you can't see them immediately at compile time.
I would expect that someone who learned Rust for a year, then learned C for a year, would probably produce better C code than someone who learned C for two years. Eventually, the C programmer will, by hook and by crook, learn all the things that Rust would have taught you, but you'll be learning it in an enviromnent that lets you make mistakes, then build on them for months before they finally blow up in your face. It is far easier to learn about what mistakes you are making in an environment where it instantly tells you that you made a mistake.
(Also, I should clarify something: By "C" here, I mean programming in raw C, without any particular additional support. For the sort of purposes I'm talking about here, I actually consider "C with a strong static analysis tool like Coverity" a separate language. If you use such a tool pervasively and work in a code base that has either been cleaned to its satisfaction or started from scratch under the analyzer's support, you can also learn in a fast-feedback environment what not to do. It won't be the exact same lessons as Rust, but it'll be good enough; there are many paths to enlightenment in this case, just as I trod a completely different one as mentioned above. The important point is to use one of these, or perhaps another, and not simply stand in front of the monster that is Raw C with nothing but your own wits to guide you. I don't care how smart you are. It'll eat you alive. You do not want to be responsible for trying to tame the thing yourself under your own power and nothing else.)
The story I’m seeing at most companies is that they have a few old guys taking care of maintaining their legacy C/C++ codebase, and there’s a diverse bunch of young folks shipping new components in Rust.
When the time comes you can start with Rust
Guess I should start my 5th attempt at finally getting good at Rust.
Probably this one:
- Rust has the clear goal of memory safety, which is an active issue in the kernel (see: Kees Cook's Kernel Self Protection Project), even more so when it comes to drivers which are a lot more variable in quality and have very variable levels of oversight, which is why those are the primary use case.
- Rust has a much more expressive type system, especially without the need to go into template weeds.
- Rust has a lot less implicit behaviours.
- Rust features tend to be more orthogonal and mis-interact less with one another.
- While Rust has a fairly steep learning curve, short of unsafe it doesn't kneecap you, if it doesn't like what you're doing (right or wrong) it tells you, it doesn't go off into the weeds doing the wrong thing.
https://rust-for-linux.github.io/docs/kernel/
Out of that, core is basically just what you get from Rust's core itself, alloc is similar to Rust's provided alloc but is fairly heavily customised to meet the requirements of the kernel and of Linus. But kernel is custom code for the kernel specifically. Overall that's a lot of effort.
We didn't see a similar effort for C++. Would that have overcome Linus' reluctance? Well, it can't if it doesn't exist.
Do you remember the Paul Graham essay on the Blub programming language? (http://www.paulgraham.com/avg.html)
C++ is kind of like the Blub language. If you know C++ well enough, you can do anything you need to do, so something like Rust just seems too esoteric and weird and pretentious.
Fortunately, I spent many years using Rust recently (coming from primarily a C background) and when I'm on a project using C++, it just feels too weird and rudimentary and horribly over-complicated.
To end with a quote from the PG article:
_But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub._
I would have said C is firmly a Blub, though.
C++ is full of footguns and it is easy to make mistakes, but it is also one of the most powerful and expressive languages out there.
Not to mention that it is blazingly fast.
You can argue that what you give up in exchange for this freedom is more valuable, but don't twist the definitions of words.
It depends on which "common definition" you're working from. To make an analogy, both GPL advocates and MIT/BSD license advocates argue that their conception of freedom is "more free."
Rust is closer to the GPL here. By limiting certain things that you can do, you are free to do things that would be harder if you're allowed to do anything. The canonical example here is Stylo; the project was attempted with C++ multiple times, but was too buggy. But Rust's restrictions allowed the Rust version to succeed. You can argue it both ways: Rust limits certain kinds of code patterns (outside of unsafe, of course...) but that may enable you to do things that were too hard to do when there were no safeguards.
(A more generalized version of this debate is the distinction between "positive liberty" and "negative liberty," this debate transcends software.)
The restrictions merely apply to provably correct code.
I'm not sure how telling the compiler to disable the safety features so you can do your thing is unbearably limiting.
C++ templates are hard to learn and understand, but they are one of the most powerful constructs we have in any programming language, I miss them when I work in other languages.
Though as far as compile time templates go I think Nim templates generally meets or exceeds C++ templates in most areas. But I've become addicted to compile time type ducking, which is antithetical to Rust's vision of programming.
I love Rust; I’d do any new project in Rust instead of C++.
It’s worth the effort to learn.
- Have hidden costs
- Have sneaky failure-modes
C just... doesn't add any abstractions. And then Rust adds abstractions, but makes a concerted effort to avoid ones that have these two problems
Rust though? That could make the kernel more secure in the long-term and rule out a whole class of bugs. This makes sense.
The thing that finally made Rust click for me was doing a usb HID project. Suddenly all the machinery around ownership and multi-threading made sense and I was able to get things done, and I could suddenly also do all the things I previously attempted to do in Rust.
I guess i should do the same too but I have really and man the language is not easy to grok. These days it also feels like if you don't know Rust, you are somewhat "old school".
You say that like there’s some scalar measure of “better”.
The gccrs and rust_gcc_backend projects both are progressing well and would presumably help enable support for other targets only supported by gcc.
As far as its affect on Rust, the last year or two, a significant amount of development and affordance has gone into features for "no-std" and its likely that will only accelerate now that Rust is going to be in Linux (and have an opportunity to prove itself there) along with the number of Rust developers that are interested in kernel development.
The performance of the NVMe Rust driver in Linux persuaded some of the skeptical kernel devs. With Rust's greater static analysis (and a lot of work around formal methods), in the long run, there will be optimizations that are at least easier if not impossible in C-based codebases.
The main advantage is, as you said, the static analysis that radically reduces pointer-related bugs and security vulnerabilities. Another advantage that I'm personally a little skeptical about, but has been expressed by more than one kernel developer, is attracting new talent to becoming kernel developers as the current group is aging.
Do you have a link to some of the discussion around this? I'm curious to hear what kernel devs thought of the driver and how it was introduced to them. Is there a talk about it? Or mailing list archives? (Or both?)
IMO Rust is a really good language, and it deserves it's place in the toolbelt of most systems programmers. It's not always the right choice, but it's usually a good choice.
This is one of my few big gripes with Rust too, but it should be pointed out that it's dynamic linking with other Rust code that isn't really supported. Dynamic linking with e.g. C libraries is a breeze.
Part of the discipline of writing C++ and C libraries is writing around a stable interface, that you can link against. In C++ that means using the pImpl pattern.
Which is, of course, the actual gripe we're discussing. The lack of dynamic linking is just a consequence.
This has happened and will absolutely happen again. Just being "memory safe" will not prevent protocol errors, bad IVs, preventing timing attacks, or the other subtle things involved when doing cryptography on the wider internet.
There's a huge list of tradeoffs between dynamic and static linking, and Rust is betting that in today's world, with modern tooling, internet availability, hardware speeds, and memory size, the benefits of static linking outweigh the costs. I guess time will tell
But- traditionally Linux has really leaned into the opposite approach, so it'll be interesting to see how that dissonance plays out, and whether one or both is forced to compromise, as Rust gets more embedded in the Linux ecosystem
As a Rust fan who subscribes to the classical Linux distro model (and even exclusively gets his Rust crates from there), I disagree. What you suggest comes with a cost, namely having to accept software in constant flux. To me, that cost in unacceptably big.
I write a bit more about this gripe here: https://news.ycombinator.com/item?id=32925491
(Of course the problem is technically hard, or possibly unsolvable, so I don't blame the Rust folks for its existence.)
I did for work as well and in my case I do not want to touch it again for quite a while. Cargo is a terrible system with all kinds of hidden and possible broken dependencies you introduce and everything depends highly on the version of rust you use. It's not even close to mature enough to consider using except for test projects. It doesn't fix programmer errors despite every rust user trying to convince me otherwise that all imported crates are perfect.
I like the idea of the language somewhat but the entire ecosystem ruins it for me.
I've encountered none of this. Of course Cargo is going to have broken libs, it's just a package repo like npm, pypi (+ pip), etc.. The differences between "editions" is pretty large, but I haven't seen much reason to not use the latest release in most cases. The tooling out there is very good at supporting multiple toolchains simultaneously (though, not in the same project, necessarily).
> It doesn't fix programmer errors despite every rust user trying to convince me otherwise
No engineer who knows what they're talking about will ever say this. You're either spewing lies or are believing the words of people who have no idea what they're talking about, neither of which are good. Rust helps prevent a specific class of memory errors that are very common in C (and to a lesser extent, C++). It still gives you the `unsafe` escape hatch that lets you do almost anything you can do in C.
> It's not even close to mature enough to consider using except for test projects.
Our production hard-realtime control system begs to differ.
Can you give a concrete example of this?
The rust portability story is in the process of being addressed, by two different projects:
gcc-rs[0] adds a Rust frontend to GCC.
rustc-codegen-gcc[1] adds a GCC backend to rustc.
Both are progressing pretty well, but it will likely still take a while until they reach maturity.
> Or will this change restrict the number of platforms the Linux kernel will be usable on?
There won't be any Rust inside the Linux core, only for drivers. So no, the linux kernel will always be usable on all platforms. However, the drivers written in Rust won't be usable on platforms not supported by LLVM until a GCC alternative becomes mature.
> what will the advantages be?
There are multiple: Safety, improved stability, and developer productivity. Compile time will likely take a hit (Rust is famously slow to compile), but I don't expect a big shift in runtime performance (could be slightly better due to the extra opportunities the compiler has for optimization, but it's not super likely).
As for having any effects on Rust: it already has! A lot of unstable features got prioritized due to being necessary for the kernel, such as fallible allocations.
[0]: https://github.com/Rust-GCC/gccrs - monthly reports at https://thephilbert.io/category/gccrs-status-updates/
[1]: https://github.com/rust-lang/rustc_codegen_gcc - monthly reports at https://blog.antoyo.xyz/
That’s the biggest one, anyway. I picked up Rust for the first time 8 years ago. I liked it: it felt comfortable in my “worn” C++ hands. But the community outside of a very few core individuals who care deeply about the language’s success can be quite toxic and dismissive of anyone who suggests Rust isn’t basically perfect. And it was a bit of a turn off for me, personally.
They’re simply right about one core thing and they’re loud about it when it comes up. Until something proves them wrong and steals their thunder… I don’t know what to tell you.
Not sure what to do about that really, but I think it caused many C++ programmers to hate the language before they even tried it.
Experience shows that memory safety issues are the most common cause of security issues.
Microsoft:
>As we’ve seen, roughly 70% of the security issues that the [Microsoft Security Response Center] assigns a CVE to are memory safety issues. This means that if that software had been written in Rust, 70% of these security issues would most likely have been eliminated.
https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe...
Google:
>Roughly 70% of all serious security bugs in the Chrome codebase are memory management and safety bugs, Google engineers said this week.
https://www.zdnet.com/article/chrome-70-of-all-security-bugs...
Microsoft:
>we’ll peek at why we think that Rust represents the best alternative to C and C++ currently available.
First, there are plenty of fantastic memory safe languages already available and widely used inside and outside of Microsoft, including .NET languages like C# or F# and other languages like Swift, Go, and Python. We encourage anyone who is currently using C or C++ to consider whether one of these languages would be appropriate to use instead.
We, however, are talking about the need for a safe systems programming language (i.e., a language that can build systems other software runs on, like OS kernels). Such workloads need the speed and predictable performance that C, C++, and Rust provide.
When thinking about why Rust is a good alternative, it’s good to think about what we can’t afford to give up by switching from C or C++ — namely performance and control. Rust, just like C and C++ has a minimal and optional “runtime”. Rust’s standard library depends on libc for platforms that support it just like C and C++, but the standard library is also optional so running on platforms without an operating system is also possible.
Rust, just like C and C++, also gives the programmer fine-grained control on when and how much memory is allocated allowing the programmer to have a very good idea of exactly how the program will perform every time it is run. What this means for performance in terms of raw speed, control, and predictability, is that Rust, C, and C++ can be thought of in similar terms.
https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe...
Sure thing but can Rust live up to its memory-safety promise? Or even data-race freedom promise?
I must say that, coming from systems programming background, I was always skeptical about those promises because IMO these problems fundamentally cannot be solved in bare-metal programming language. They're implied by the complexity of HW not loop-holes in programming languages.
That said I see nothing wrong with Rust but I think that people are vastly overselling its core features while at the same time a quick glance over https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=rust will give you exactly a list of bugs which Rust seems to promise not to run into.
Another very recent example, and a very good blog post about memory corruption taking place in Rust: https://www.graplsecurity.com/post/attacking-firecracker
More seriously, the number of peoples you can have a sane discussion about unsafe use is quiet low because when you need to use unsafe is because of a complicate/hard to explain/understand problem.
This combined with the (99% of the time valid) safety dogmas brings some to instantly tell you how to not use unsafe with blatantly bad workaround without considering why you _would_ use unsafe at the first place.
This safety/anti-unsafe dogma makes some peoples far too vocal.
I consider Rust community to be nicer with time, seriously, but I can't say this "I don't understand your problem, but I saw a unsafe, here is a code that doesn't use unsafe" is not a thing. Of course, it's changing, because Rust is more widely use, and "here is when you need unsafe" patterns are emerging. As a Rust user, I'm happy.
There will always be outliers, but I actually think the Rust community has done a really great job around discussing unsafe in the years since the actix-web issue. See a really great discussion at: https://www.reddit.com/r/rust/comments/we91es/good_example_o...
Just some anec-data, but I had some benchmark code which was unsafe (indexing into an array of bytes, no UTF8 validation), that I wanted to rewrite as safe (by leaning on the std lib, so plenty of unsafe under the hood) just to see how much slower it would be, and the code ended up being ~20-30% faster. Sometimes the unsafe patterns we bring with us just don't match the Rust model.
I agree, I started Rust before this drama (I wouldn't call that an _issue_, it was the whole culture that goes wrong here), now peoples try to understand why you need unsafe, but it hasn't been the case and for _years_ it was almost impossible. actix-web is the pivot point but the simple fact you need a sad situation like this show how crazy the safety discussions were before, really.
Once again: Things are changing and I'm more than happy to see pragmatism in the Rust community. But you can't behave like all of this didn't happens.
> Sometimes the unsafe patterns we bring with us just don't match the Rust model.
I agree and noticed this too in some cases (compiler does miracles), that's why discussions have to be serious.
I think that was always the case. actix is a singularly unique example, and I had never seen anything like it before or since. Not even close. There just hasn't been any Rust project with that kind of popularity that also contained egregiously unsound code. So we're not just talking about unnecessary use of unsafe here, but egregiously unsound usage of unsafe.
The attitude toward 'unsafe' has pretty much always been the same. Use it. It exists for a reason. But make sure it's justified and strive for soundness.
The crab is cool though. As language mascots go, it's a lot more adorable than the Gopher or that LISP alien thing, and almost as adorable as Bjarne Stroustrup.
You really don't? Every time I read a comment like this, I think, "Who hurt you?"
Still love the language though, and in general the community is very friendly and welcoming (if not always open to criticism).
I can already hear faint exclamations about the language's age, but are Rust programmers justified in putting forward claims about safety? Or do they shift to using the word in the sense of "fewer bugs in this particular class" (and nevermind any other effects the language may have on development).
Even if we take such a narrow and distorting view of "safety", we can look at, say, "The Power of Ten" and ask how does Rust use fare with each rule, and is it any more advantageous than other languages in use by safety-critical systems.
I understand there's the Rust Book, Rustlings, and Rust By Example at https://www.rust-lang.org/learn. Are there other good resources? Does anyone have a strong suggestion on which of those official resources I should start with?
The rust book. Reading it start to finish provides a very good basis.
And if you're tempted to "learn by doing" with the usual linked lists, go and read "learning rust with entirely too many linked lists": https://rust-unofficial.github.io/too-many-lists/
Rust has enough rare or novel concepts that trying to learn on the job with no book learning whatsoever is really reserved for the rarefied few. Though I understand that a lot of C and C++ concepts port quite easily, it also differs from those languages (especially C++ for advanced features) that doing a bit of a reset will avoid future pains e.g. Rust's concepts of copy, move, references, ... are very different from C++'s and going in half-cocked will be frustrating.
That means what the best way to start is depends on your background. In my eyes you can do nothing wrong with going through the rust book you already mentioned. Having read 4 different Rust books I have to say this introduction in combination with "Programming Rust" by Jim Blandy, Jason Orendorff, Leonora F. S. Tindall is probably the best way to get started. https://www.oreilly.com/library/view/programming-rust-2nd/97...
[ For example move really is as simple as Rust makes it look, it is a headache in C++ because they added it to a finished working language ]
Several people suggested books/ web pages let me suggest some videos:
Jon Gjengset's "Crust of Rust" Youtube videos are good once you reach the point where you can write more than "Hello, world!" with some confidence but certain specific things aren't clicking.
https://www.youtube.com/watch?v=rAl-9HwD858&list=PLqbS7AVVEr...
For example, Jon does a whole video on lifetime annotations, which are something you won't see in most popular languages, but also one on Rust's iterators, which are a familiar concept from both C++ and Swift but don't work quite like either (they're very different from C++ iterators, your Swift experience will help more).
I recommend watching specifically a handful on topics you don't feel you understood well, although you could watch all of them especially if you just enjoy Jon's style and have the free time. They're not fast paced, if you wanted "Rust in 60 minutes" this is not that, but each one is actually writing carefully chosen code that runs and talking through what's going on, not just clicking through slides.
Just go through everything in "3 Syntax and Semantics" and "4 Effective Rust", you already know how to program C and C++ so that is all you should need to know. Took me a few days.
(1) Go through the Rust Book,
(2) Do "exercises" (small side projects, advent-of-code, rustlings. ..etc),
(3) Go through "Rust for Rustaceans". It's specifically targetted at "intermediate Rust developers". If you have "good working knowledge" of C and C++, I will wager that you will absolutely love this book, because, to me, (1) it explained so much of Rust's design choices, and (2) you will learn so much so quickly.
It definitely took some time and learning but now I have 6 Rust API microservices, a few scheduled/queue-reading services, and a shared library for common models/utils/providers.
It appears Linus has said it is happening, and a bit how it should happen, but has not given strong guidance or even parameters on how long any experiment would last.
Instead, he says the maintainers can decide to accept or reject or use Rust however they like. He doesn't even say they should state their policy, so they can let people try and still reject them.
Giving maintainers the ultimate discretion I think tracks the incentive system in the kernel: maintainers, who do boatloads of work, get to be deciders. That also gives companies a strong incentive to employ maintainers.
On the tools front, Linus expressly said he wants not just an anointed compiler on kernel.org, but compilers from the distributors. He's driving Rust normalization and platform adoption as a condition of use, even though the kernel historically adopts a fairly narrow (if not archaic) set of tools.
Somehow this all seems like a bunch of rafts tied together with (platform) boats pushing at the edges -- somewhat tenuous, hard to drive, but vaguely heading in the right direction.
As others have mentioned, it's interesting that Linus and many maintainers are getting older, and instead of getting freedom and flexibility in their maturity, they can look forward to more low-level bouts of increasing complexity. When they talk about the integration of Rust, they're clearly anticipating it might not fully happen until after they're gone. That, too, could change the incentives.
But heck, we should TRY it. So I'm glad this is finally happening.
Just a few questions off top of my head:
* How much will it affect the development velocity? Existing devs (and maintainers) will at least need to learn how to read and debug Rust code. New devs will likely have no good competences in C which is 99% of the codebase so how good/quickly will they be able to blend in?
* How many existing kernel developers will community potentially loose because of the increased complexity?
* How much new talent will it actually be able to attract? Now people will need to learn two completely different languages and their vastly different ecosystems.
* Does Rust have high enough entry bar to produce competent kernel developers? Sometimes high entry bar acts as a very good filter to end up in a pool with a highly skilled (and motivated) engineers.
* Coordination of development efforts is now going to become much more complex. This includes writing new features (C or Rust or both?), maintaining the old ones (rewrite pieces in Rust or not?) but also code-reviews which will now pose a bigger challenge given that it will consist of mixed Rust+C code.
Having worked in codebases which mixed C and C++, I already understand what kind of issues mixing C and Rust is going to produce. And I expect them to be much more accentuated because distance(C, C++) << distance(C, Rust).
That's what you really want in Raspberry Pi sized projects.
Perhaps it's worth waiting a week before posting these? LWN's paywall is dropped at that point.
I am not Jonathan Corbet so I don't know what he considers occasional. But this is the fourth LWN subscriber link posted in the past 7 days, and the eighth in the past 14 days.[2]
For reference, LWN currently lists 10 paywalled posts, the oldest from 13 days ago.[3] Those posted on HN are nearly every paywalled article that they have.
[1] https://news.ycombinator.com/item?id=1966033
[2] https://hn.algolia.com/?dateEnd=1663790400&dateRange=custom&...
[3] https://lwn.net/
Not the OP, but a possible reason: not all people have credit cards. In fact, the very first thing I did once I finally decided to get a credit card (which, frankly, is only really necessary to buy things from outside my country; within the country, there are other convenient payment methods) was to subscribe to LWN. Before that day, other than a brief year in which a friend had paid for my LWN subscription with his credit card, I mostly waited the one week to read the paid articles (and, in fact, I'm still subscribed to the LWN mailing list which automatically tells you whenever a paid article becomes free).
esp the last paragraph: >As time ran out, Matthew Wilcox asked whether kernel developers should be writing idiomatic Rust code, or whether they will be writing "C in Rust". Ojeda answered that code might be more C-like toward the beginning; adoption of more advanced features (such as async) might take longer. Gleixner asked what could be done to prevent developers from using unstable features (once the features used by the kernel are stabilized); the answer was to specify the version of the compiler to be used with kernel development.
Some reasonable people in this sea of sectarian crabs, glad to hear that