Linus Torvalds: Rust for the Kernel Could Possibly Be Merged for Linux 5.20
phoronix.com
phoronix.com
They say that Rust support is optional, but when the first big driver gets written in Rust it'll become mandatory. So either the Rust integration will fail and nothing important will get written in it, or it won't be optional. I'm not sure what comfort that's supposed to bring.
They just assume you'll curl | rustup.whatever and install a new version of rustc every month from a third party website like every single Rust guide suggests. That's not optimal for the linux kernel. But maybe kernel dev types won't be as bad as the typical bleeding edge rust dev.
And given that rapid rate of change in Rust, I don't see how GCC can ever keep up. Maybe in 10 years when Rust is more stable.
Funnily enough, I'm using the rustc compiler from Debian repositories (who of course is not well-known for eagerness to adopt bleeding-edge), and I've not run into any Rust code that wouldn't work with that compiler.
The foundational libraries in the ecosystem tend to take a much more conservative approach to MSRV (minimum supported rust version), supporting older Rust versions for as long as possible, and only bumping the version when new features that significantly improve the library are released (and often only once the versions implementing those features are packaged in stable releases of major distros).
For example, the `rand` crate supports Rust versions down to 1.36 (released in July 2019) [0], the regex crate supports down to 1.41.1 (released Feb 2020) [1], etc.
[0]: https://github.com/rust-random/rand#rust-version-requirement...
[1]: https://github.com/rust-lang/regex#minimum-rust-version-poli...
Btw, I just attempted to install another rust program (legdur), and guess what, my rustc is too out of date to handle it ("failed to parse the `edition` key, this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions"). So that's 3 of 4 attempts to run Rust programs that have failed. Granted, my rustc is now about a year old, but that's still absurdly short.
Have you tried compiling something less than bleeding edge, with a year old compiler, or are you picking projects specifically to "showcase" the supposed failings of the Rust compiler?
Many libraries in the ecosystem have a MSRV (minimum support rust version) guarantee, with compile-time shims to enable newer features if a more recent version is detected.
You can pin your dependencies to those versions (and if they don't have an explicit MSRV, just pin it to a version by date or by running https://github.com/foresterre/cargo-msrv on the project to find the effective MSRV).
You can cargo install specific versions of a binary crate, and if they move to the 2021 edition, or use a recently stabilized standard library function or w/e, you can simply choose to install a specific version, that would work with your distro's rustc/cargo.
I'm not even talking about the completely valid, but last resort strategy of many non-bleeding edge distro package maintainers, of simply creating a .patch file and applying it. In legdur's case, --- edition = "2021" +++ edition = "2018" on Cargo.toml would probably do the trick. For libraries/binaries you control, you can use https://doc.rust-lang.org/cargo/reference/overriding-depende... and https://github.com/itmettkeDE/cargo-patch.
Giving up after the first minor roadblock and crying bloody murder is intellectually lazy.
They last just fine much longer than that, you may just have to upgrade your compiler if you compile from source (doing so is a single command, takes about 30 seconds max, and new versions are backwards compatible so code targeting older compiler versions will still work). Importantly, you generally do not need the Rust compiler at all if you only wish to run Rust applications. You can usually download a pre-compiled binary distributed by the program authors. Applications (not libraries) requiring a recent compiler / language toolchain and updating as and when is convenient to them is hardly unique to Rust.
Basically, if you're in environment where you can't or won't update your compiler to something more recent, then why do you expect to be able to use recent programs? Why not use older programs in line with your old compiler?
This is what I don't get about the Debian/Centos folks. They specifically use a Linux distro that gives them old software and then complain when they can't build new programs. Well, if you're using Debian/Centos, then you're committed to old programs. So build old programs, not new programs. Either that, or go install 'rustup' and bring in a newer compiler.
You know, like a sane person.
Oh hey, there's two of us ;-)
My shoulders relaxed significantly when I also put Cargo in offline mode, pointed it to the directory where Debian installs its librust-*-dev packages, and stopped downloading rapidly changing stuff off of crates.io.
Rust is good, but I also do not like the trend in every language and/or build system having its own package management with its own third party dependencies and its own custom installers separate from the distro. I understand the pragmatic problems this solves, but I think it has serious ramifications for reproducibility, security, auditing, etc.
In general, language package management systems, containerization and fully statically linked distributions (appimage, flatpak, snaps, etc.) are all a direct response to system package managers being woefully inadequate in all sorts of ways.
And unless something like Nix takes over which actually allows reproducible builds, we'll continue to see alternative solutions continue to encroach on system package managers.
I see why they happen. I don't blame the people to make them. I just think it's a potentially awkward situation when it comes to something as core as the kernel; which has traditionally only required GCC.
Clang is starting to leapfrog GCC here. IIRC, some features are better supported and/or more advanced on Clang, other features (e.g. LTO) are exclusive to it. Not to mention it's almost a certainty for Clang to land kernel PGO support long before GCC, in fact Google already uses it in production Android kernels.
Anyway, rust or not, linux is being decoupled from GCC.
This is in the context of the Linux kernel, which only mainlined Clang LTO support in 5.12. AFAIK, GCC LTO has yet to be accepted.
The trouble, of course, is that this is hell for a Linux desktop system, because you can't cleanly put the GUI stack in either "base system" or "everything else", especially with all the shared libs and such.
Having one for Rust reduce a lot of complexity (Same operation across all OS), and is the best place to take decisions (Rust developers know Rust needs, debian ones who knows?).
Let's consider my alternatives. 1. I can (and will) publish the source and give instructions on how to build it. This works but is a bit inconvenient for the user. They have to manually install and manually check for updates and manually install updates. 2. I can publish on language-package-manager. This is a simple and easy process. It can be figured out in less than 30 minutes. For the user they put the name of my project into a config file and that's it. 3. I can publish on top-5 system package managers. This will take days or maybe even weeks of labor.
So basically, it's just not viable for hobbyists. It can only work for companies that pay people a salary to deal with it.
I use Arch Linux and Most Rust programs I use are installed from the Arch repositories or AUR. Rust packages are very well integrated with the distro, they depend on distro packages and have other packages depend on it. As far as the user is concerned, the Rust build system is just a developer-only stuff like CMake or autotools or ninja or whatever.
Anyway I would like to point out that C++ also do something similar to what Rust libraries typically do, which is to use header-only libraries that don't appear as separate distro packages. It's as if every Rust library meant to be used by Rust programs (as opposed to libraries that expose a C API that can be called by other languages) were a header-only library. And this is actually great because Rust (like C++) monomorphizes generics, that is, if you call a generic function defined on another crate, the compiler actually generates a new function just with the type parameters you supplied, and there's no way the library can know upfront which generic instantiations will happen over all programs that use it.
On the reproducibility front, I think it would be great if C program actually did what Rust does and pinned the exact damn versions of all libraries they use (like Cargo.toml does)
Feels like a pretty blanket statement. I'd assume that they'd base the minimum version on Rust's editions. An edition is released every 3 years and is meant to be a solid LTS (as far as I'm aware) version of the language. If they use Rust 2021 edition, you can tell the compiler that (like C89 vs C11) and it will reject code that's newer. C has plenty of newer stuff too that isn't allowed in kernel code at the moment as well.
All new Rust features are available from all Rust editions. The 2015 edition has got new features 3 weeks ago, and will get more again in 3 weeks.
Old Rust editions are only for backwards compatibility with old source code, e.g. to let it use names that became keywords later.
Editions are more designed to manage backwards-incompatible changes, like new keywords or tweaks to corner cases in the type system.
You don't need a super modern version of the spec to use Rust.
As for Rust's incompatibilities, as long as you stay clear from methods marked unstable and the nightly compiler versions you'll be fine. Rust's libraries have breaking changes, but the stable language features haven't had any since the 1.0 release as far as I'm aware.
If your problem is forwards compatibility, well, you can't use clang10 to compile the kernel either. Sometimes the version gets bumped, but it doesn't need to happen too often. I think GCC's rust compiler will keep up just fine.
That said, I think for this first phase the kernel is planning on a specific stable version of the compiler with nightly features enabled, because they have a couple specific needs that haven't made it through the stabilization process yet.
I'd like to see the actual percentage but AFAIK it would take some analysis. Crates.io shows 86417 total crates, searching for "stable" shows 3346 and "nightly" shows 2527... So that's not a good heuristic, probably.
https://github.com/rust-lang/surveys/raw/main/surveys/2021-a...
You tried to make that sound like few Rust developers use nightly, but the same survey you linked to states that, while 7449 said they use the latest stable version for local development, 3333 said they use nightly, with 2853of them using the latest nightly and 481 using a specific version of nightly.
All in all, the ratio of Rust developers targeting nightly releases or betas represents something in between a third and a quarter of the whole developer community.
That's a lot of people in the Rust community delivering code targeting unstable and unmaintainable releases.
How did you come to this conclusion? Seems like it must be working for them, because most of them are using nightly out of choice.
Plus it’s a service to the community. Their everyday testing helps make the stable releases better. Imagine if no one used nightly - that would be much worse.
In my experience writing Rust professionally (including making heavy use of dependencies), I've only needed the nightly compiler once. And that was for our own code, not an external dependency.
It shows that the vast majority of people use latest stable. A decent chunk also use nightly, and the reason why “a crate dependency I need requires it” is pretty far down the list of reasons.
Oh and a pet theory: the way you feel about this is correlated with the kind of work you do. I’d you’re writing an OS, you probably use nightly still, and so your dependencies probably do too. But if you’re higher up the stack, where unsafe is less common and the appropriate language features are more filled out, you probably never use nightly. This makes talking past people unfortunately easy to do.)
(n.b. I’m glad they released the raw data, because otherwise we wouldn’t be able to share this, because it wasn’t included in the blog post this year. I brought this up before publishing but my advice was ignored, oh well!)
What exactly leads you to believe that?
From the Rust 2021 survey, 2420 said they use nightly to access one or more language features they need, 1028 claimed a Crate dependency requires it, and 876 claimed a tool they use requires it.
This, from a sample pool where around 3k users out of around 10k claim they use a nightly version.
As for the compiler, it will use a pinned, stable compiler, not nightly.
Why are you spreading FUD about nightly + Linux on this thread? At no point has this combination ever been considered.
The survey results are not relevant. The rust code must not use stdlib (the name escapes me) which uses an incompatible memory allocation mechanism.
The code used to build will likely need to be checked into the kernel git repo and not downloaded live online, I doubt that the mature development process of the kernel wants the problems that other package managers have.
I will fight to ensure that this is not the case in the kernels that I work with, because fetching dependencies is for people who can tolerate other people and networks being faulty.
I got shit to do.
Jokes aside, I bet Jakt (a completly new language) has introduced fewer language and compiler changes in the last 3 months than Rust (5-10 years old now?).
Rust's compatibility problems are quite irrelevant in my opinion. You can pin a language version and future compilers will still run your code as long as you don't use any explicitly unstable features (so no nightly compilers and no manually enabling unstable features). Yes, Rust moves faster compared to C's glacially slow language development, but almost every language does these days. Even C comes out with new versions every few years and C compiler versions get deprecated all the time.
Rust's editions are not language versions. They set aside syntax, so far there was 2015 edition (the original Rust 1.0 syntax) and then 2018 edition (a few tweaks, introducing the raw symbol which enables you to mention symbols that conflict with keywords) and most recently 2021 edition (which adds a hack to the older editions so that arrays don't seem to be IntoIterator in 2015 or 2018 edition even though they actually are now).
C++ ships language versions on its three year cadence. C++ 14, C++ 17 and C++ 20 are three very similar yet distinct languages while Rust's 2015 edition, 2018 edition and 2021 edition are simply syntax for the same language, Rust. You can write brand new code, today, in 2015 edition, and use features that didn't exist six months ago, no problem. You just can't use keywords (like async) that hadn't been invented in 2015 and thus you can't directly use features gated by need for those keywords.
MSRV (Minimum Supported Rust Version) is a different idea, it says this software needs some features which weren't in Rust until this version. For example if you write 2018 edition, that doesn't magically mean you used async, but on the other hand, if you did require async then no stable Rust versions from 2018 had working async, the keyword was reserved but not useful. So your MSRV might be much newer even though your program syntax is 2018 edition. Documenting MSRV is not a mandatory practice, although it's good to do it for important stuff or where the minimum version is likely to be a surprise.
[1]: like when PyPI cryptography first switched to Rust and broke ansible/openwrt/etc.
We're just not there yet.
Still, looking at https://llvm.org/docs/CompilerWriterInfo.html, it is hard to find an architecture that Clang won't support that is relevant to many people. And if someone does find one, well, Clang is open source and open to adding more architectures.
* The kernel requires stability. Building a kernel w/ rustc is like building a house on a flying bullet.
* GCC is insanely good at porting to other architectures, something vendors have been relying on for a long long time.
* Consistency in applying compile flags and optimizations.
Idk, these are all I can think of right now.
Could you give some examples of big features from the last 12 months? I can’t think of any big ones off hand. IIRC, just a large number of small improvements.
> The kernel requires stability.
I can’t recall a new Rust release ever breaking existing code. Is this what you’re talking about?
> GCC
Rust will have a GCC backend and/or frontend soon.
> Consistency in applying compile flags and optimizations.
Can’t say much about this but I’m fairly sure the kernel devs won’t let anything be merged until they’re satisfied. We can trust them on this.
I have one case of code from late 2015 that stopped working in, IIRC, 2016, due to unsoundness being found in the borrow checker and it being fixed.
error[E0713]: borrow may still be in use when destructor runs
--> src/writer.rs:121:21
|
47 | impl<'a, F: Write + Read + Seek + 'a> Writer<'a, F> {
| -- lifetime `'a` defined here
...
121 | Reader::new(self.file)
| ------------^^^^^^^^^- returning this value requires that `*self.file` is borrowed for `'a`
122 | }
| - here, drop of `self` needs exclusive access to `*self.file`, because the type `Writer<'_, F>` implements the `Drop` trait
I haven't had a case of code no longer compiling since then.
> Could you give some examples of big features from the last 12 months? > ... > I can’t recall a new Rust release ever breaking existing code. Is this what you’re talking about?
Let me show you two links:
1. Rust version history: https://github.com/rust-lang/rust/blob/master/RELEASES.md
2. GCC changes: https://gcc.gnu.org/gcc-12/changes.html#c-family
Compared to Rust, GCC, the C compiler, is absolutely boring. The C language has been stable for decades, so there's hardly anything to be done on the language itself. GCC has been used in all kinds of occasions, and has compiled all types of applications and system software. We know it works in every situation. There's hardly anything to be done. This is how the real stability looks like - boring as shit.
Now, what, 12 months? The number isn't simply getting there.
That's a bewildering statement considering that one of the bells and whistles that Rust offers is memory safety.
It will take quite a while yet before something important is made in Rust, and this early work can be done in parallel with the GCC-Rust work.
Huh? What relationship the Linux kernel and GCC has?
They're independent projects. It might be handy/nice/convenient to not need non-GCC compilers to compile Linux, but it's not like it's some license requirement or project obligation on behalf of Linux
For C and C++ code you want to be able to build with different compilers.
I think that also makes sense for Rust, the language and the compiler should be two separate things and there should be multiple compiler suppliers.
Are there arguments why this would be less relevant for Rust than C?
So it's very much against the historical policy of the Linux kernel to not be tied into a single compiler, even if it is a welcome change in policy.
In addition to the large maintenance effort for llvm to support so many additional back-ends, there is a non-negligible number of folks who would be very opposed to being forced to use a non-gpl compliler to build a gpl kernel.
https://en.wikipedia.org/wiki/GNU_Compiler_Collection#Archit...
https://en.wikipedia.org/wiki/LLVM#Back_ends
https://en.wikipedia.org/wiki/List_of_Linux-supported_comput...
If other compilers implement compatible features such that they can compile the kernel, fine. But gcc has been the supported compiler.
One is a Rust frontend to GCC (https://github.com/Rust-GCC/gccrs) while the other is a GCC backend to the existing Rust compiler (https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-12). It might take a while for these to bear fruit, but it’ll be awesome when they do. That should address the very valid concern that the full Linux kernel won’t compile for all supported architectures.
And it’s not really a problem that the GCC effort will take time. The headline of this article implies that the Rust integration is nearly done but it’s only the beginning. Based on the issue tracker there’s years of work left to get everything working perfectly. That’s enough time to get GCC supported well.
That's the problem. There are no GCC-based Rust compilers, nor is there a fixed timeline for one to be delivered, specially in the short term.
Therefore, as the OP pointed out, as things stand either Rust in the Linux kernel is dead in the water or it poses a major problem for the Linux community.
Let’s assume for a second that Linus knows what he’s doing. Now look at his mail from April 2021 on what he views as show stoppers (https://lkml.org/lkml/2021/4/14/1099) - OOM panics and 128-bit floating point. The former has been fixed now, which is why Linus appears bullish on adding Rust support to the kernel. Check that mail for the number of times Linus mentions GCC - 0. It’s not a priority for him.
It’s easy to see why. He (and everyone else involved) are very clear that this will only be used for driver code on architectures that were already supported. So therefore, there is no need to supply a GCC compiler in the short term.
I pointed out two long term efforts to get GCC Rust support. These could unblock more widespread use of Rust in the kernel, but that’s jumping the gun. Let’s see even one real driver merged before thinking about that.
Whatever happens, the Linux developers won’t let the project break for millions of users overnight. That’s not how they operate. Your “Linux community” should know that.
~~It's not~~ (see below) It wasn't supported directly by upstream, and in fact historically was pretty much the opposite: the intention was to not care about portability between compilers at all.
(I do think that what "support" means is kind of amusing, I would also say that this counts as support, but above you have people arguing that only Rust's Tier 1 support is called "support"...)
It is wholly another to say that they need adopt a whole different code-generation regime. It adds risk, and adds to what they need to understand to know what they are building. A pro needs to understand where problems come from, and be equipped to dig down to them.
So, OK for hobbyists, but a burden for pros. It is the difference between "neat project, might succeed" and "mature". Rust stands a good chance of becoming mature in a few years, more than can be said about almost everything else.
Why?
For example this comment https://news.ycombinator.com/item?id=31849071 is much more interesting and on topic.
---
I want to provoke a little bit, but I'm genuinely curious about the following:
I feel like the Rust ecosystem is still quite immature in a lot of areas. Just a few indicators to illustrate what I mean:
- Both async/await and the question mark operator feel like rushed implementations, neither seem like the best long term solutions for Rust and are not in line with the otherwise solid foundation of the language.
- Open source code examples sometimes use an array of external dependencies that are unrelated to a given project and feel arbitrary. This reminds me of the JS ecosystem.
- Some projects pride themselves to not use "unsafe" code, including linking with battle tested C code, which seems like an arbitrary restriction that sounds better than it actually is. There is even an open source maintainer that got mobbed out from his own project because he used "unsafe" in places that others didn't agree with.
- Rust is fashionable. Putting "Rust" next to a HN title immediately gets clicks and upvotes. SO surveys and similar report high interest in the language. This is not an inherently bad thing, quite the opposite. But as a secondary effect it might detract from objective, technical decision making.
I really like the language, I have _way_ more good things to say about it than bad things. But am I alone in feeling like this? Other modern language communities come with a philosophy that promotes stability and simplicity. Is the Rust community riding on the merits of Rust's unique value proposition, while forgetting some important fundamentals?
However it represents something that I don't like, which is trading language surface complexity and readability for programmer convenience. It is not something that gives you more power, just something that makes code a bit easier to write. Also there was seemingly not that much push-back against the feature which makes me a bit uneasy. Not sure why that is to be honest.
I don't want to start a discussion about the question mark operator specifically. That's not important to me. What I like to gauge is the general culture around the language itself and the community around it. Half devil's advocate, half uncertainty driven.
In this case it's a welcomed convenience. Have you used `thiserror` and `anyhow` crates ? They may alter your judgement on that point.
Who needs a for loop when you have while?
> it means something more specific and restricted than a general while
Sure, but that's still sugar. That's what sugar is.
Want to know something really crazy? In Rust, the `if` keyword is also just syntactic sugar over `match`. Still finding bits of grey matter on the walls from when I learned that one.
(Although, in Rust, `if` is also actually just sugar over `match`, so, uh...)
I would say the question mark improves readability, compared to having to check/return error codes by hand after every function call.
To your more general point, the language is pretty much established and syntax features are added very rarely. I think one more feature, let guards, is coming down the pipe soon, but other than that I wouldn't be surprised if the surface syntax never changes again.
Which is one of my complaints about Go.
result, err = doSomething();
if err != nil {
...
}
result, err = doSomething2();
if err != nil {
...
}This is normal? not having unsafe code does not guarantee absence of bugs, but at least it isolates the problematic bits in the sections marked as unsafe.
If you need unsafe, you can always use unsafe.
Battle tested codebases like bind or openssh — and many others, independently of the language of implementation — had bugs in the past. It always helps to have extra assurances. Using safe/Unsafe is one more tool to have assurances.
This is a recurring comment, but I would argue that it's not "rushed" but rather "incomplete" from supporting ecosystem (like a futures library being reactor-agnostic).
Async/Await is great for a couple of things, but not a silver bullet. So, apart from high-throughput network applications, (like web/app servers), you should stick to regular threads and channels. You can leverage Actor-based concurrency in your program that isn't async.
> - Rust is fashionable.
Because it's genuinely great ! I've been writing Rust code since 2018, and at that time "Golang" in titles would get all the hype. I get the urge to keep things as a secret underground in a true Hacker spirit. Relax : nobody is going to spoil Rust and Rust isn't going to become a new Javascript.
The culture of exploding dependency trees is a recipe for disaster, especially when combined with the prevailing attitude of "use the latest version or bust".
Still, it makes dev a lot easier.
I'd disagree with both of these. They may not be to your style but that doesn't make them rushed. Both features had long debates around their adoption.
> - Open source code examples sometimes use an array of external dependencies that are unrelated to a given project and feel arbitrary. This reminds me of the JS ecosystem.
That is irrelevant to this project. You should read the rust kernel docs to see why.
> - Some projects pride themselves to not use "unsafe" code, including linking with battle tested C code, which seems like an arbitrary restriction that sounds better than it actually is. There is even an open source maintainer that got mobbed out from his own project because he used "unsafe" in places that others didn't agree with.
Also irrelevant. This may be a criticism of Rust but it in no way affects how rust may be adopted into linux.
> - Rust is fashionable. Putting "Rust" next to a HN title immediately gets clicks and upvotes. SO surveys and similar report high interest in the language. This is not an inherently bad thing, quite the opposite. But as a secondary effect it might detract from objective, technical decision making.
Linus is famous for doing fashionable things and forgoing technical decision making.
Has he changed his mind about those things? Or does he not realize they are there, and that kernel code will use them?
When I try to make a list one of the first things I find is "any programmer that would prefer the project to be in C++ over C is likely a programmer that I really would prefer to piss off, so that he doesn't come and screw up any project I'm involved with."
There have been several carefully planned iterations of the Rust for Linux effort. Every iteration has addressed some feedback from senior kernel developers including Linus Torvalds. The Rust for Linux folks then worked with the Rust project to introduce these features slowly into Rust. An example of such a change is fallible memory allocation. These iteration cycles have been happening for years.
Check out the issues, especially the “wanted features and bugfixes” for each rust component (https://github.com/Rust-for-Linux/linux/issues?page=2&q=is%3...) for more details into all the work that’s happened here.
Throughout I have seen nothing but professionalism, courtesy and hard work from all parties involved here - the kernel devs, the Rust for Linux devs and the Rust project devs.
You do them all a disservice by implying that this decision was taken without “objective, technical decision making”.
> not linking with C code
I would need specific examples of C code being better than the Rust equivalent but people using the Rust equivalent anyway. I can give several counter examples. Such as libgit2, a C library with Rust bindings. It is widely used in the Rust ecosystem, including by the Rust project itself instead of the nascent gitoxide project. This shows pragmatism.
There are other examples of projects being rewritten in Rust but usually that’s ends up with the Rust version being very good. Such as when the maintainer of rsvg rewrote it in Rust. Look at the test suite results (https://github.com/RazrFalcon/resvg).
Or when someone wrote a text search tool (ripgrep) in pure Rust instead of using PCRE bindings. Look at the benchmarks (https://github.com/BurntSushi/ripgrep/blob/master/benchsuite...) - it’s faster than any competing tool on nearly all of them. If you’ve used search in VSCode, you were using ripgrep.
Or when someone wrote a pure Rust crypto library rustls when OpenSSL bindings exist. I believe the results of the security audit speak for themselves (https://github.com/rustls/rustls/blob/main/audit/TLS-01-repo...). See the comments on code quality - “Cure53 was unable to uncover any application-breaking security flaws. After spending thirty days on the scope in late May and early June of 2020, the team of auditors considered the general code quality to be exceptional and can attest to a solid impression left consistently by all scope items. Naturally, this is partially thanks to the usage of Rust as the preferred language for the entire implementation of the rustls project”
These examples show that when good developers rewrite something in Rust, we all benefit from the results. If you have an example of a Rust rewrite giving worse results, and being adopted over the C bindings, that would be helpful. I can’t recall any such instance offhand.
> Open source code examples sometimes use an array of external dependencies that are unrelated to a given project and feel arbitrary. This reminds me of the JS ecosystem.
Could you give examples? From what I’ve seen Rust library authors take care to only pull in what’s necessary. I’d be interested to see these libraries that pull in “unrelated” dependencies.
Note, developing in Rust doesn’t suddenly make mediocre code great. I’m sure there’s mediocre code out there, just like there is in every single ecosystem. The question is, can you get stuff done while only sticking to high quality dependencies? I think you can.
> async-await and question mark.
There is a language strangeness budget each language gets to use. Use up too much of the budget and the language feels alien to newcomers.
Rust already introduces new strange things like lifetimes. New syntax like the question mark and post-fix await strain the budget further. I’ll admit, it’s hard for new people. But I also think this syntax pulls it’s weight. This is subjective and reasonable people can disagree.
That said, these syntax decisions are set in stone. I wouldn’t wait in the hope that they might change.
Just wanted to comment shortly on this:
> Could you give examples? From what I’ve seen Rust library authors take care to only pull in what’s necessary. I’d be interested to see these libraries that pull in “unrelated” dependencies.
I would hate to call out specific authors and crates. I would rather see a general awareness and discussion around the issue. Your comment is reassuring though.
Yeah I feel you. No need to call anyone out when they're just putting out their work for free for us to use.
That said, I think generally the code that I've seen is good quality. Everyone uses rustfmt, everyone tries to address clippy lints. I've seen PRs in popular libraries dropping optional libraries to get slimmer. Rust libraries aren't perfect, but they're pretty good.
For Scala devs this is the sane way :) (Or for anyone who likes RPN ^^) [Or for anyone who simply encountered a bit too many "(await (await x.y).z).w" expressions in other languages.]
https://users.rust-lang.org/t/name-squatting-on-the-crates-i...
- parking_lot
- hashbrown
- ryu
- heck
The longer there's a shared namespace, the stranger the names get as the "good" names are already in use.
The contrary case gets confusing fast. "Oh Known Algorithm doesn't work" "Huh, I read a paper which proved it works" "No, I mean, Known Algorithm, not known algorithm". "What?" "The implementation by Popular Author. It's broken for the case we care about". "Can't they fix it?" "They say that although their implementation is called Known Algorithm it's just better that it not fix this because now Known Algorithm is different from the known algorithm". What?
Knowing there's a problem, or a different performance trade off, or a lack of maintenance for Haphazard (Jon Gjengset's Hazard Pointers implementation) is very different from having a problem with Hazard Pointers the idea.
I am not sure the assembler thing would handle UTF-8 as well as my incredibly low effort Rust solution tho.
What I want to say with this is: There are real, tangible advantages to the language when some noob who would shoot himself into the leg three times while writing the C/C++ equivalent manages to write safe code that performs well while not investing a lot of time.
What if there's nothing wrong with its intended purpose? The wheel has been around for some time, there has been research into other axle-mounted components surely we replace the wheel then?
I don't understand the "it's old therefore bad" argument.
But there is. See the long, long, so very long list of vulnerabilities caused by incorrect memory management.
GP was just emphasizing how old and crumbling that fossil is with the 50 years old remark.
>What if there's nothing wrong with its intended purpose
And what, exactly, is its intended purpose ? It was to rewrite a 10K LOC kernel maintained by a core team of 3 developers from PDP-7 assembly, in an age before the personal computer and the internet. It was literally created as an ad hoc, bug ridden and unspecified implementation, starting with the thinnest dressing over assembly and adding the absolute bare minimum required for a human to write ~10K LOC without gouging their own eyes off.
There is plenty wrong with it.
>I don't understand the "it's old therefore bad" argument.
It's easy. Fields where there are progress invalidates and supersede their own widsom. You wouldn't go to a doctor trained 50 years ago if you could help it. You wouldn't drive a car made 50 years ago. The only fields where that's not the case is stagnating one, like building houses or making furnitures. Things where 'progress' consists of irrelevant-to-quality changes.
Programming language design is in the first category, and not the second. Especially in the period from 1975 to 2000, it learned and discovered so much that languages made before that period might as well be cave man scribbling on cave walls.
[1] https://quotepark.com/quotes/1741351-c-a-r-hoare-about-algol...
With that being said, can you recommend a modern language designed around a model of computation flexible enough to target e.g. non-flat memory models?
I'm additionally interested in targeting MCUs with no ISA-supported stack and ~2KiB of RAM.
It's a bad argument, and it's not the argument the person you're replying to is making.
C is bad for reasons that are mostly disconnected to its age. Better languages were written in the 1960s and 1970s, and worse languages are written today. What the GP is saying is that C has not meaningfully improved over the last half century.
So is it your position that C is the pinnacle of systems programming languages? That no significant improvement in PLs has been made... that could ever be made?
I'm a Rust fanboi, but I totally get why some people don't like it. And why many people believe that something better (in one or more different directions) is possible. Or that something else would an even better fit for Linux kernel development.
If the formally proven stuff gets more traction, I'd likely jump ship to something like that. Though a lot, lot of work needs to be done there, especially when talking about interacting with hardware... such a headache. But if our base computing infrastructure could be proven to be correct (hardware and software), that could dramatically improve the entire software ecosystem. There would still be problems, but if we can at least move them up a level or two in the software stack, we have an easier time finding and fixing them. This is the difference between the Spectre attack and leaving the permissions for a password file wide open.
I don't know what a better future is going to look like exactly, but I know that we're not going to get there with just good old C code.
Actually, a third language. The Linux kernel already has two: C and assembly.
wasm kernel extension does sounds nice, especially after the meltdown and spectre.
Rust is superopinionated. Big and complex as well.
Currently, it compiles into C++ as it's written specifically for an entire operating system written in C++ but I can see native compilation becoming an option down the line.
It's far from stable but it's worth looking at: https://github.com/SerenityOS/jakt
\\I really don't
\\understand why people
\\have issues with
\\Zig's multiline syntax- Can't easily copy paste
- Can't easily edit
- Looks ugly
- Harder to read
Reasoning: https://github.com/ziglang/zig/issues/162
- programs ought to be UB-free by default, and overriding those defaults should be rare
- mutation is allowed, including shared mutation across threads
- runtime assumptions similar to C and C++, i.e. no garbage collector, and a useful subset of the lanugage doesn't require heap allocation
Given those requirements (in short, a "safe systems programming language"), I think you very quickly find yourself needing 1) lifetimes, 2) the no-mutable-aliasing rule, and 3) move semantics. Those things contribute a lot of the "feel" of Rust, and probably the majority of the difficult learning curve. But I think it's interesting to consider that "the type system should deal with lifetimes" isn't really a baked-in opinion that Rust has, as much as it is the necessary complexity stemming from the other opinions above.
Zig is not like that, it's pretty much just C, but simpler, ergonomic. It's design is a reasonably simple, portable abstraction of contemporary hardware. It gives you all the power to come up with your own abstractions on top of that.
Why not thought? Lots of drivers could benefit from async to simplify their state machine as they wait for interrupt for example.
The kernel has some different properties than userspace - e.g. context switches between kernel threads can already be cheap, which minimizes some of the reasons for using async/await. Then allocations for continuations using in-kernel slab allocators might also be cheaper then in userspace, whereas placing huge objects (Future)s on the stack (preferred mode of Rust Future composition) might be more expensive with tiny kernel stacks.
I guess it might be worth to do experiments for some use-cases to see how it actually would work out.
I've worked on C and Rust codebases, though nothing near as complex as the Linux kernel, and I'm excited for this change as it will reduce the burden on the kernel developers to reach the same quality output, so we either get drivers faster, or drivers for hardware we wouldn't have otherwise received, or both.
As far as the fear of Rust kernel drivers that don't compile with GCC's Rust frontend, I trust Linus to keep the bar high on letting this feature in, and that kernel drivers accepted into the tree compile with just GCC as they always have.
Third-party drivers may not, but that's also true today if someone wrote a driver that only compiled with LLVM. This doesn't really happen in practice as it goes against the path-of-least-resistance for the developer: they'd have to switch their toolset out when working on that driver vs everything else, and the auto-building by DKMS would probably fail on them if they used it during development -- all of the conventions used by the rest of the kernel infrastructure will keep them in line.
There was a bug with Intel GPUs that was unfixed for at least three kernel versions where plugging in an external monitor over an HDMI-to-Displayport adapter would reliably freeze the system the moment the kernel tried to modeset. That's finally been fixed from what I can tell, I believe it had something to do with a change to an NFS driver somehow.
Server hardware often doesn't need complex or bleeding edge kernel modules. You don't need sound or video, you don't need framebuffer resolutions, you don't even need that much in terms of keyboard compatibility. It's a lot easier to run a kernel of the most complex hardware out there isn't hooked up to your system. That doesn't mean Linux is bug free, it merely means that the kernel code for the limited hardware of your choice has been maintained well.
The real challenge for Linux or any kernel really is to run reliably on a laptop with an uncommon variation of common hardware, preferably sold for only a few months. That's where the real bugs come in.
There are also definitely kernel security problems (e.g. this from a quick google: https://www.cvedetails.com/vulnerability-list/vendor_id-33/p...). If Rust can eliminate a percentage of those then great.
(Frankly, I doubt that those would be resolved with Rust, or even if the developers all got attendants that would massage their feet.
The GPU driver developers are hugely busy with cranking support for new cards, and bugs in the older cards support are rarely tended to, when they are it makes headlines. I blame skewed incentives.)
You also don't need async in many if not most cases, especially if you're writing kernel code. I wouldn't want to use a kernel module that pulls in all of tokio/reqwest/hyper just like I don't want a kernel module that links against openssl.
Async makes a lot of sense for userland code, but not so much for kernel code. I haven't seen async get used in any demo for kernel Rust modules and I doubt it'll happen anyway.
Looking at a comparison between common kernel structures and theoretical Rust implementations of those (like on https://security.googleblog.com/2021/04/rust-in-linux-kernel...) I think the kernel can get improved security and reliability from Rust without any "fancy" Rust. Just the basic type system improvements, nullability improvements and explicit checks will be enough for some real benefits.
Have you seen how much communication with the hardware in drivers is actually very asynchronous? It’s all hand rolled C, of course, but there’s a lot of it.
But look at the bright side: people likely won’t be enjoying hand-rolling asynchronous Rust since language-level primitives exist, so maybe there will be enough pressure and enough of use case diversity to make a more solid implementation that doesn’t suck, or sucks way less than the current one.
What do you consider "hand rolled"? The kernel has extensive support for these use cases since, as you said, hardware is usually async. Async with hardware is rarely tangled, from my limited experience. It's usually event driven.
The reason Rust is interesting in the space is the memory safety without a GC. Zig doesn't have that.
Anyhow, this isn't all or nothing: a case could also be made one way or the other about how much 'unsafe' to allow in Rust in the kernel. It's a question of tradeoffs.
Async in Rust is incredibly well designed, and works very well for such a low-level design that minimizes heap allocations and virtual dispatch. Rust intentionally prefers locally explicit syntax for things that affect control flow, and wants to be back-end agnostic without a runtime, which goes against implicit built-in "colorless" magic.
At least in regards to the features that affect kernel-level development. Stabilizing that doesn't preclude active development in things like async or in the standard library (both of which I'd expect not to see much or any use in the kernel.) But I'd hope for decent work towards backwards and forwards compatibility in the set of language features used there.
The fact that it’s hard to even remember that these things have happened is a testament to how rarely it happens and how well the breakage is handled.
I suspect folks see the Rust release notes on the HN front page and assume that it must be a blockbuster release with lots of move-fast-break-things to make it there. But it’s actually like “new library API, some const fns, compilation time improvements”.
So then they think “this language is moving waaay too quickly, they need to do LTS versions”. But this is just speculation. That’s why I asked GP. Maybe they’ve actually experienced breakage.
I expect these to become even less frequent with ever rising amounts of Rust code that isn't on Github.
Strange you should mention this because they’ve been working on and will continue to work features to make Rust more suitable to the Linux kernel. This issue (https://github.com/Rust-for-Linux/linux/issues/355) for features in rustc was opened a year ago and it seems half done.
> backwards and forwards compatibility
Backwards compatibility has existed for 7 years, since Rust 1.0. If your code ever compiled under any version of Rust since 1.0, it will compile with all future versions. Guaranteed.
Forwards compatibility - not sure how this would be guaranteed. If your code uses a new library API, how would an old tool chain compile that? AFAIK, this is a non goal.
> calm down
I think it calmed down after async stabilised 4 years ago. I don’t recall seeing big changes since then.
On a broader note, Rust’s 6-week release cycle might make it seem like big features are being added all the time, but I don’t think this is the case. It’s usually just a few small features or conveniences, some bug fixes, minor improvements to compile times.
In my opinion the Rust language and eco system is not yet at the stability of C.
I feel certain that mixing two different languages inside the kernel will give all new and challenging errors to debug.
Also it is the fallacy of sunken cost.
So much work has gone into the Linux kernel, but efforts should be made to replace it.
What I would like to see is a project to create a new and modern kernel, taking a lot of what has been learned by the "prototype" Linux kernel and create something new and better.
It would also be free to take full advantage of the language improvements between Rust and C.
Sure it would take a lot of time, but I think it is time well invested, instead of slowing trying to port the current Linux kernel to Rust.
Plenty of people are working on new kernels and operating systems both by hobbyists and companies (c.f. Redox[0], Fuscia (zircon kernel[1], e.t.c.).
The main problem is that linux is an incredibly well developed piece of software, so it will take a very long time for anything to reach feature parity.
> instead of slowing trying to port the current Linux kernel to Rust.
I don't think anyone is suggesting to try and port the current kernel to Rust (and I'm fairly sure any suggestions in that direction would be met with a hard no from linus.) The current suggestion is just to allow language bindings to facilitate the writing of drivers in rust.
Even looking to the future which could see some critical components of the kernel written in rust, I don't think anyone is suggesting that the entire kernel should be ported, since that's a monumental task (and suffers from the exact same issue as above, for questionable benefit).
Also, having rust in a serious project like linux, (and omitting all the cargo madness), would be a great thing for the language.
Rationale: just like Rust does not let you segfault, a decent build system should not let you download shit from the internet. Much less so, without asking you explicit permission. Much less so, silently and by default.
...sure, by specifying the whole url and calling an external program where the actual shit downloading happens. The same as bash, C and so on. Of course, it is in very bad taste to do so, and by no means a standard or even common thing to do when using makefiles.
Maybe it would be OK if cargo let you download code, after giving it a sort of "unsafe" flag or something. But the current behavior is just bonkers.
Don't understand your gripe. Is it just that you don't like things to be this easy for the dev? Dependencies should not just be disfavored but actually hard to use?
[1] Comparison of memory safety of c vs Zig vs Rust (from elsewhere in the comments): https://www.scattered-thoughts.net/writing/how-safe-is-zig/
Specifically (and this is no longer quite eli5), Rust's freestanding mode--working without a standard library--is somewhat fuller and more fleshed out than the C++ freestanding mode.
Now, the sibling comment that points out that C++ gained a bad reputation pre-C++11 is probably a bigger reason for it not being seriously considered whereas Rust is, but there are some technical reasons where Rust does better than C++.
* It's very difficult to predict allocation patterns in C++ (even modern variants) from just the source code. An innocent looking object allocated on the stack might have a nontrivial constructor with all kinds of side effects. Rust, while being slightly less explicit than C, is much more explicit than C++.
* Modern C++ doesn't actually play that nicely with C. C++'s aliasing rules and casting rules keep getting more and more strict, whereas kernel programming lends itself to a lot of "bag-of-bytes-to-struct"-style programming. It would be very unfortunate if adding a C++ compiler to the kernel's build broke pre-existing C code by detecting UB in the context of C++ and decided to simply erase it.
And, Rust has even more incompatibilities with fast-and-loose C memory use. There are reasons why C and C++ compilers have "sanitizer" modes. Those identify places where code would be surprisingly elided. Yes, C has those too.
These 4 things do not happen invisibly in rust:
code running on struct creation (it will be an explicit ::new() call)
code running on struct copying (it will be an explicit ::clone() call)
code running on struct moving (moving is always a memcpy in rust, no custom code can run).
Error handling flow control (no exceptions, error handling will at minimum be marked with a '?'). (yes panic exists but this is basically for the same purposes as a kernel panic).
Rust favours making things explicit, much more so than C++.
In general, Rust is more implicit than C, but significantly less implicit than C++. Moreover, the ways in which Rust is implicit are more consistent and less likely to introduce bugs.
As an anecdote: I had a codebase where a nontrivial constructor contained a “registry” of created objects, for introspection. The design contract for that part of the API included the assumption that all constructions would be explicit, but it wasn’t actually enforced. Later, someone did some subclassing and added some APIs that inadvertently performed a converting construction, resulting in broken invariants around offsets and members in the registry.
Was it a good design? No, it was terrible. But nothing stopped someone from writing it, and the codebase otherwise “did everything right” (C++11, warning flags out the wazoo, fuzzing, unit tests, etc.)
It was a real pain in the ass to debug, and it fundamentally couldn’t have happened in a Rust codebase. And that’s just one tiny corner.
Code written to assume it will only ever run in context X being made to run in context Y instead is a problem that happens in every language equally. It is prevented by making a predicate Z to report whether X, and asserting Z where it matters. It is not a product of conversion.
In Rust, you would have had a factory function for type A, and somebody would have made a factory function for B that called the one for A, provoking the same failure.
We still need evidence of the original claim that implicit argument type conversion is a common source of memory corruption. We already know that bad design breeds bugs.
In Rust, the closest thing would probably be a shared Rc member for the registry and a non-owning indexing scheme (either weak references or something symbolic). But I’ve never actually seen a Rust codebase that needed to apply the registry pattern; it’s usually the wrong abstraction, like I said.
Edit: And no, replicating this pattern in Rust would not have produced the same failure. The failure mode in C++ resulted in exploitable memory corruption.
Also, GCC should fix their stupid libjitgcc so new languages can use it reasonably instead of making everyone use llvm and then complain that there is no gcc implementation
But why should Linux, or anyone really, let future promises block meaningful improvements today? If something comes along that's better than Rust for this, then another adoption/migration process can happen.
The issue is really starting considering you can't even call Debug's fmt.
And Rust will smooth the path for other language in the systems space. Rust had to work extra had because there has only be one dominant (pair of) languages for so long. Now that people have got used to choosing, hopefully it'll be easier next time for Rust++ or whatever.
But building a successful language and ecosystem takes a combination of visionary initiators, dedicated contributors, money, a significant value add, luck , and lots of time.
Rust has warts that could be done better by a newer language. There are some interesting experiments, but I don't know of any language that would fit the bill right now.
And if that language emerges it will need many years and probably a corporate backer to reach the level of maturity required for critical workloads like the kernel.
The world shouldn't be limited to C for another 10 years.
There is no plans to bring it into the core of Linux.
I'm going to call the new c only Linux cinux
But maybe I can elaborate a little further: it’s as fast as C, but more fun.
Besides, fighting with the borrow checker requires a lot of grit ;)
I wonder - doesn't that mean that either the borrow checker is a bit shit, OR you're systematically doing something wrong and don't realize it?
One of the reasons that "fighting the borrow checker" is such a common experience, is that it can be hard to tell the difference between cases like "this doesn't work because you forgot a small piece of syntax" vs "this doesn't work because you need a special helper type to make it work" vs "this doesn't work because fundamentally Rust will never allow this to work". For example, sharing objects between two threads often requires the `Arc<Mutex<...>>` pattern, which is a combination of two different helpers, and beginners who've never seen that combination before are very unlikely to discover it on their own. But once you have some experience with common patterns, and you know how to avoid writing code that will fundamentally never work, compiler errors are much more likely to be helpful.
A lot of people ultimately decide that "the compiler was right all along" and that their code is really better now that they've learned to satisfy the borrow checker. Personally, I think learning how to write Rust code is a helpful shortcut to learning how to write safe and correct C/C++, because the "borrow checker in your head" pushes you towards designs that work well in all three languages. But this is definitely a matter of opinion, not to mention extreme selection/survivorship bias, etc.
Linus has made it pretty clear that it's why he's allowing it; it provides something above and beyond other system languages wrt to writing correct code while still working with the kernel's structure and requirements (with a little elbow grease).
Additionally, Rust is a newer language that has incorporated some of the past 50 years of programming language design lessons that C has not incorporated (and likely never will).
introducing another language will create additional barriers for new driver developers. the kernel itself is already complicated enough and the focus should be on better documentation for new developers and a more welcoming attitude in general to improving (read simplifying) APIs like ALSA-SoC, for example. and please don’t tell me that the docs in the kernel tree are good enough.
as a device creator, i look at this from a completely different and more practical perspective than those who are obsessed about programming languages. in the end the kernel serves the hardware community and is not some kind of programming utopia to prove how smart you are.
i then have to maintain and keep up that know-how which adds more cost to my business.
and the kernel is not going to become a 100% rust code base anytime soon.
This is not the case and is not planned. Are you trying to make some kind of slippery slope argument or something?
Okay, so do you contribute? Because your wording here:
>> i will lose all interest in ever contributing
To me reads as a clear indication that you don't yet.
but.
C is language created by grugs in 70s. programmers born in 90s, 2000s very different. wear "programming socks". think C very, very bad -- danger!
rust is systems language for programming soxers. reality is, programming soxers now make big-BIG part of shamans in oss. like rust. want to rewrite everything in rust, think they can trap buffer overrun and use-after-free demons forever.
grug think there are perfectly fine grug languages -- ada, even lisp -- can trap demons same-same. grug also think tools can help keep demons out of C code base. but momentum behind rust now. grugs who maintain kernel must meet developers where they are. if kernel grugs resist change, programming soxers refuse to join project. go to press, say "these CVEs could have been prevented had they used rust!" gruggernews say "well, they're not wrong..." linux lose face. lose base of new programmers. very bad for project.
The fact is that new developers (and many old ones) see C as too risky and difficult to work in now, meanwhile JavaScript kids are transitioning to Rust and writing code that's nearly as performant as your C code but with fewer bugs.
As I like to say, Rust isn't for you... it's for your replacement.
This is not much of an argument. Yes, you have another use case, and maybe memory safety bugs are not an issue for you (which -- you hear this a lot among C programmers until they are bitten). But also clearly that doesn't mean those bugs aren't an issue for everyone else, or that the Linux kernel should prize your POV over all others.
Also, maybe your use is extremely limited? I mean "portable game device not connected to the internet" is pretty small share of the total Linux marketshare. It's probably a small share of the total Linux "portable game device" marketshare.