Rustaceans at the border
lwn.net
lwn.net
Kernel development is hard, and bullshit doesn't go very far in that context. Success for Rust in that environment (with some changes along the way) will be a proof of value.
Pretty sure async traits are coming soon (next couple of versions?) which is pretty speedy considering what I understand to be a semi-thorny problem.
> having a proper concurrency runtime story
Do you mean the language/stdlib shipping an async runtime?
> reducing the reliance on third party crates to ease error handling
I for one don't rely on 3rd party crates for error handling often? Anyhow is the most common one I use, but mostly out of habit and laziness, not actual hard requirement....
No. There are ideas for how to do it, but as far as I know they haven't been tested out yet.
It also relies on pretty large features that are not proposed for stabilization yet (GAT and existential types). When that is done and the implementation strategy is chosen there will also need to be a RFC cycle, a phase of ironing out bugs and finally stabilization.
It'll be ... quite ... a while... I can't see it happening this year.
Something something let the Java guy cast the first stone.
> reducing the reliance on third party crates to ease error handling does seem to take out forever.
I've written a ton of Rust and I've never used any of these third party error convenience libraries. As far as I'm concerned there is no issue in need of solving around error handling in Rust.
The "please don't offend this language I have intertwined with my own identity with" attitude in modern times is so tragic.
I have always been, and will be, plenty of things, and if it makes you feel good to call me Java guy, when someone criticizes your beloved Rust, please do.
But you're over-reading into my defense of Rust. My point is that the specific things you criticized as taking a long time for Rust are not actually taking a long time. My point with poking Java, specifically, is that Java is going on 30 years old and doesn't yet have an analogous feature. Moreover, since we're talking about Kernel dev, do C or C++ have a standard async runtime? Considering that there's little-to-no prior art for doing the kind of async API Rust wants without active garbage collection, I'd assert that it's hard to criticize how long it's taking. Do you know how long it "should" take? If so, how?
Even if C is not my primary language of choice, i would definitely try adapt myself to the ecosystem and not the other way around.
You have all the knowledge of other peers, manual, books, all the libraries, the whole ecosystem.. this cant be replaced.
Also there's something else about C nowadays, is the lingua-franca, the latim (or english) of programming languages. We use it to expose api's to others in any other language that want to consume it as a library.
There's something about culture that people often forget in tech.. it's the real backbone of any project that it's on its own feet.. and when you want to enter in a community you will be better of if you learn and adapt yourself into this community culture instead of creating cultural clashes into the community and try to overtake it (be it hostile or not).
People should be aware that this effort will make it possible to create rust-based kernel drivers and that's it. the RIIR folks are delusional and hype fueled and its better if the sane Rust community get away from them or start to get them back into reality as i bet they are not willing to expend 10 or 15 years of their lives rewriting big and complex piece of software for a likely no return as people will tend to keep using the software the have more community and that are stronger.
It's a much better approach for Rust or any other programming language to become research darlings and eventually become the primary ecosystem of a research OS that went well and is the thing that will replace Linux. The language alone wont do it, it must be able to be a contender to UNIX and POSIX, and whatever language that is in such a system will probably be the one that will become the dominant one in such a ecosystem.
Also another good approach is to virtualize the Linux Api like gvisor does is userspace or the fuchsia OS(and even FreeBSD) does in the kernelspace. So that you can create your OS and kernel in the best way you can looking ahead, and have this Linux compat layer where applications dont even need to be aware they are not actually running in Linux.
Yes, i'm sure there's an imaginary opponent downvoting my comment.
Also there's a lot into my comment, and people are not even noticing it in the whole context.
If there's no reason, why people get so upset? just move on if you are not being mentioned, as it will clearly be the case if i'm talking about "imaginary opponents"..
> The one proposing it is a long-time kernel contributor.
I'm not saying anything about people working on it specifically, if you read my comment, there's a clear separation between serious people and the hype crowd (which is not just RIIR, but now also cryptocurrency fellows, etc).. i can't say where the people working on this fits, i don't know them. Don't know from which part of my comment you took that conclusion.
There are quiet, clever, serious people doing the work, like Hoare, Matsakis and the people that are real enginners, i have all the respect for them (and im pretty sure Rust have tons of such a people). To be fair, all languages have all kinds of people but i don't know what happen to some of them that tend to attract a certain type of people more than others, like the feeling a got from Haskell community more often than others (but given the community was much smaller)..
For instance talking about culture, C succeeded exactly because it was a pragmatic language very simple and efficient like their founders to get things done. With this culture, things happened to be done around the language and we have the ecosystem we have today.. it's a great hacker spirit of more humble, hard-working, behind-the-cameras sort of people which i sincerely miss in the days of instagram, tick-tock and tech celebritism.
> * 11 hours ago | parent | prev | next [–] > Rust will quickly replace C in the kernel, I have no doubt about it
I've have seen tons of such a comment in all sort of products when the matter is discussed around here and elsewhere (twitter, reddit, you name it)
If you want to really get serious about this, i can feed this comment sections with tons of evidence over the course of the years.
I don't know what it is about Linux that makes people say this. Kernel development has most of the same constraints as any other embedded context, which Rust has plenty of focus on. No it's not as mature as C, but few languages are.
Plus if you go looking in the kernel you can still find plenty of bullshit hacky code. It's not special, it's just another random open source software. The quality very much depends on the individual maintainer of that subsystem and how much has been invested in that area.
Today Rust does not overlap Linux in terms of platform support. There are (small but very much alive) communities doing Linux on architectures that Rust has no support for and in some cases has no plans ever to support. So this makes drivers the only case where choosing Rust doesn't mean some people lose out, as a platform e.g. with no PCI bus doesn't get to run PCI drivers even if they were written in C.
I expect that over the next say, five to ten years, two things will happen to greatly improve this, maybe to the point where you absolutely could rewrite core Linux code in Rust if you wanted to. Firstly, Rust will get more platform support. Linux doesn't really need Rust's "Tier 1" (Linus doesn't check every kernel release passes tests on all real Linux target hardware as I understand it) but clearly you want Rust to at least build and take patches for every Linux platform some day. Secondly, some older platforms will "rust out". If your community is nursing 30+ year old hardware and increasingly more maintenance work is shared between fewer shoulders at some point "Linux-next" is not a priority and your platform will stop being supported while effort moves to exciting new hardware.
Nit, I believe actually the Rust and Linux platform sets would be considered "overlapping sets" in the mathematical sense :), since neither is a subset of the other.
e.g. Rust platforms include things like the NetBSD Rump Kernel and Redox and I think one would be hard-pressed to claim that Linux supports those as platforms.
I'm interested in kernel development and I like the idea of working on it for a living. Can you give more details about your job? What does it consist in? Is it mostly code-review? Or are you responsible for maintaining a part of the kernel. Who is the entity that pays you, and what are the criteria they'd use to pay a new contributor to work on kernel full-time? Finally, can you point me to beginner-friendly things to work on to get started? how do I know which part of the kernel I should study and contribute to?
Hence the crazy idea of now using WebAssembly in Firefox modules instead.
Probably less than 30 people in the world understand the thing as a holistic entity. From evolutionary terms, there's a giant risk of losing relevant technical knowledge to keep it up in the future.
I do not support adding it to the Kernel, I think we should just throw away the kernel entirely, but I understand why they're looking at Rust.
That's a pretty wild assertion and files in the face of the highly active kernel development process.
e.g. > Linus has released 5.18-rc1 and closed the merge window for the 5.18 release... 13,207 non-merge changesets were merged during this merge window.
https://lwn.net/Articles/890119/
> I think we should just throw away the kernel entirely
That's the dumbest thing I've see on HN in some months. The kernel is deployed in hundreds of millions of devices worldwide and continues to be the dominant OS in many many sectors.
No, OP is right. 99.9% of things added to the kernel in any given release are drivers (mostly for obscure server hardware that the average LKML lurker would never have access to). So while there's certainly active development, it's all centered around drivers, and the "kernel" part of the kernel are mostly stable (modulo the occasional greenfield project like eBPF). (Note that I said 99.9% of things added, I'm sure security patches make up a lot more than 0.01% of kernel patches).
There are billions of Android phones alone. Not to mention the huge numbers of servers, IoT devices, embedded computers, and all the SBCs in my closet.
Fortunately for the kernel, despite being niche, it has a rock-solid onramp forcing people to get into it. There will always be companies interested in the n'th degree of performance, both generally, and for some specific hardware. Someone has to go do the relevant kernel work for those things. So while you or I may never touch it, it is effectively impossible for a kernel like Linux to just rot away because nobody cares. It would require first a multi-year, if not multi-decade process of fading first.
It's on most of the smartphones on the planet. Plus it is the dominant server OS. Plus in the top 3 in many embedded categories (a highly diverse set of technologies)
Sadly enough, probably not for long: https://en.wikipedia.org/wiki/Fuchsia_(operating_system)
As we only have two other "big" (popular) kernels to compare to this one, do you think they (Apple and Microsoft) have more people "holistically" understanding the entire kernel or less? Since the nature of those companies are closed-source, I'm fairly certain even less people understand those ones "holistically".
> I do not support adding it to the Kernel, I think we should just throw away the kernel entirely, but I understand why they're looking at Rust.
How would that work in reality? Re-use the existing tests to build a new kernel from scratch? Sounds like a very far-out idea that wouldn't help with any of the current problems, but I'm happy to entertain the idea and hear your reasoning here.
While I would tend to agree that a full production replacement would be such a massive undertaking as to be impractical, https://github.com/nuta/kerla does something very like that - Linux userspace ABI on an all-new Rust kernel. (And even at this small scale, I find it mind-blowing that this worked)
I guess I need to go let them know they are Nobodies. oxff said so.
HaikuOS, SerenityOS, Redox, and ReactOS are all going strong. The BSDs continue to advance as well. Redox is written in Rust even.
I believe Google sees Fuschia as a true Linux competitor.
When you say “we should just throw away the kernel entirely”, what are you suggesting?
At no point did any active kernel developer express any such sentiment. Nobody is worried about not being unable to attract new blood.
What would you use instead?
That's simply false. I'd love to work on the kernel. I have immense respect for the people who work on it. Only reason I haven't tried contributing code is I don't think I'm skilled enough.
Probably a lot more than 30. There are probably more than 30 PhD students studying kernels at this very minute. However, I think you're right that no one is interested. People greatly underestimate the effect the kernel has on the full OS. They think it's "just" the kernel. This is why so many people keep saying that Windows will get a Linux kernel in Windows 10, no, 11, no, 12, etc. It's just a "kernel", like a grain of corn, something insignificant. So there isn't much interest in working on kernels. It's as unsexy as it gets.
This maintains cargo's dependency resolution/update checking/etc, but also allows for all dependency code to be kept alongside the kernel code and audited accordingly.
As said in the article, "the world is changing". However, the way the world is changing is that more and more people are aware of supply-chain attacks. While I acknowledge that Cargo isn't npm, it is still the case that of all the software that can not afford to just sorta grab things from the internet, the Linux kernel is arguably #1, straight up. There is no chance that the kernel developers are ever going to accept "well, I wanted some async stuff so I grabbed a v0.5.23 of a package that I found appealing". Even if they pull some stuff in, it will be through a review process and it won't be through cargo in general.
The argument against cargo or any equivalent being used in the kernel is stronger today than it was 10 years ago, and the first derivative is also positive. Probably the second one too, honestly. This isn't about kernel developers being old fogey sticks in the mud, this is about the kernel being such a high-assurance environment, the biggest, fattest target in the world, that things that make sense for most software packages don't make sense for it. And it has nothing to do with cargo specifically or Rust. It's just that the entire workflow afforded by cargo, or npm, or go modules, or the half-dozen Python package managers, or anything else resembling those things is simply not appropriate for the Linux kernel. The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow.
> The only way to make such a thing work would be to pin the versions so hard that you're effectively only using them as a downloader, not a package manager, and you might as well just have vendored code in the kernel repo anyhow.
This is the exact thing the person you replied to said should be done. `cargo vendor` fetches the sources for the package @ the version stated and embeds them in the repo. After that all deps would be sourced from the repository itself.
Nothing in the review process would have to change beyond adding a dependencies directory to the repo with a dedicated set of CODEOWNERS to handle reviewing patches with new dependencies.
What happens if a dependency adds a system call something? Libraries intended for kernel-friendly use cases really need that scope to be an intentional goal.
As for syscalls, Rust has a whole ecosystem of no_std crates for kernel and microcontroller development that already assume the lack of an OS. We use (and contribute to) such crates extensively in our product (which is already developed jointly with the Linux Foundation, though we're not working on the Linux kernel (well we are a little bit, but all of that work is still in C :P )) and can vouch for their quality.
They are not “too lazy”. It’s not economically viable for them to do so for various reasons.
AFAIK, in Haskell, any function that returns IO must indicate it in the type signature.
If you could find a way to take that idea, expand upon it with much more fine-grained types, then you could build an ecosystem where any system call, external network call, use of env variables, etc is baked into the type signature of every function and package. If you could do that, you could build pretty trivial checks to ensure that a given package doesn't perform any kind of system call.
So sort of like the types of permissions we assign to apps on Android, but represented (and enforced) directly in the type system.
It would be difficult to make something like this ergonomic but could be pretty cool.
Anything similar in rust would have to be enforced through code auditing tools, either forking the compiler, using some of its code as a basis or starting from scratch.
I should have prefaced my comment with a note that this was way more speculative :)
Everybody knows that there are escape hatches because they are actually sometimes absolutely required for asymptotics or just plainly because it may be too hard to "prove" that your code is safe to the compiler.
That isn't a gotcha -- you know exactly what code you need to audit extra carefully.
[0] That one's actually in ByteString to technically not standard Haskell, but I just like the name.
I wonder if this is a real and considered need, or a knee-jerk response to try to put everything in rust and directly in ring 0 just for the sake of it. I see no reason to try to break down the "divide between kernel and user space" in this way, and I wonder what's actually driving it here.
I wish this article went a little deeper into _what specifically_ these Rust users are trying to make modules _for_.
If you preserved an immutable tag of the source code and all its dependencies, a copy of the compiler version used and all build flags, then you’ve still got some big holes in your ability to reproduce a binary:
1. OS version & patches installed
2. OS configuration
3. Hardware used (processors can have weird subtle bugs, microcode can affect execution behaviour, etc etc)
4. Transient issues - the golden copy to be reproduced for some post event investigation might have contained a bit flip leading to impossible to reproduce verification signatures
Or is reproducability just a spectrum and you try to get further along it with some careful attention to detail, rather than an absolute to be achieved?
If the latter are you not cheaper just archiving binaries and tagging them with the source + deps + compiler + arch used to build them? Thats a 5 minute job to setup in your CI process and costs comparatively little to maintain vs wasting expensive human brains chasing down a futile goal.
Look for example at guix, that bootstraps explicitly with what they call "the maxwell equations of software".
Imposing a web-based build system, into the kernel of all places, is a kick to the face to people who care about reproducibility.
For something like Go it's trivial though. If you're using pure Go code then reproducible builds are pretty much as simple as "compile using Go 1.xx".
I don't understand why you think 3. and 4. are issues. Bit flips are very unlikely at compilation scale, and the hardware you use to compile something shouldn't affect the output. In either case you can just compile it twice and on different hardware and compare the result to confirm.
It is totally possible to use cargo without ever touching crates.io, but if you need any dependencies you'll of course have to provide them through some other means (local file system, git repositories, or a custom package repository).
cargo sounds pretty much like a new bespoke build process that doesn't even work across different languages on the same OS
I still think cross-language efforts (like bazel and I'm sure there are others, maybe nix?) seem generally better, but I suspect there is some fairly good reason they are less widely used.
I'm honestly astonished that programmers of a language that is deemed to be "safe by default" thought that this behavior was acceptable in any form, not to say the default. If downloading things at build time is somehow necessary, it should be an obscure option behind a flag with a scary name, like --extremely-unsafe-i-know-what-i-am-doing, that prompted the user with a small turing test every time that it is run. Cargo is just bonkers, it doesn't matter at all if it is "convenient" or not. Convenience before basic safety and reproducibility is contrary to the spirit of the language itself.
It's as if bounds checking in the language was deferred to a third party that you need to "trust" in order to believe that you won't have segmentation faults.
Edit: for things like the kernel, vendoring dependencies is still probably not a bad idea, of course
What happens when a given dependency adds new kernel-inappropriate features? Are kernel devs going to act like distro maintainers and decide between forking, maintaining patch sets, etc.?
You must specify --locked to get that behaviour
That's exactly what it does. The developer is not really expected to thoroughly review the codebase of every dependency.
Just like javascript, all sort of supply chain attacks are made possible.
A single malicious library can sneak into large ecosystems easily.
Wait till you find out about java ecosystems
I know investment bank dev teams pulling whatever they need from maven central with no oversight or introspection.
This is borderline inevitable for most modern development stacks, though .lock files can definitely help, even adding hashes to check against if you care about your dependencies being the same as when you first download/add them to the project and/or inspect the code.
As for worries about the things in those URLs disappearing, in most cases you should be using a proxy repository of some sort, which i've seen leveraged often in enterprise environments - something like JFrog Artifactory or Sonatype Nexus, with repositories either globally, or on a per-project basis.
The problem here is that all of these repositories kind of suck and that the ecosystem around them also does:
- for example, Nexus routinely fails to remove all of the proxied container images and their blobs that are older than a certain date, bloating disk space usage
- when proxying npm, Nexus needs additional reverse proxy configuration, since URL encoded slashes aren't typically allowed
- many popular formats, like Composer (or plenty more niche ones) are only community supported https://help.sonatype.com/repomanager3/nexus-repository-administration/formats (nobody will ever cover *all* of the formats you need, unless you limit yourself to very popular stacks)
- many of the tech stacks that have .lock files may also include URLs to the registry/repository from which they're acquired, so some patching might be necessary
- in technologies like Ruby, actually setting up the proxy isn't as easy as running something like "bundle install --registry=..." as it is in npm
- in other technologies, like Java, you get into the whole SNAPSHOT vs RELEASE issue and even setting up publishing your own packages to something like Nexus can be a bit of work; the lack of proper code libraries for reuse and abundance of code being copy-pasted that i've been being a proof of this in my mind
So, for most people the approach that they'll take is to just install stuff from the Internet that other people trust and just hope that the toolchain works as expected, a black box of sorts. I've seen plenty of people just adding packages without auditing 100% of the source code which seems like the inevitable reality when you're just trying to build some software with time/resource constraints.
Sounds like they did a decent job anticipating that use case!
Some of it was a bit awkward to actually use this way in the early days, but those harsh edges have been since sanded off.
Addendum: Until quite recently, this was quite cumbersome. It also meant that all cargo invocations (by the same user) would use that override, always. It meant that compiling someone else's project became quite the hassle. Or compiling a project that mostly uses system dependencies, but some crates.io deps. But the situation is improving.
Cargo binary projects have Cargo.lock by default with checksums of all dependencies used. crates.io doesn't allow changing past releases, and has a policy of not deleting any crates unless legally required (user-accesible "yank" hides, but doesn't delete). Crates.io index is a git repository with full history of all changes to the registry, so you can recreate its state at any point in time (in case you lost your Cargo.lock, you can reliably remake one from the past).
And on top of that there's `cargo vendor` command that makes a local offline copy of everything you use, so you can fully archive a Cargo project and rebuild it any time later.
The answer there is often "use FFI". But if we're all going to use C APIs anyway, then shouldn't our package managers support C?
But C has probably the most awkward build system culture of any language, complicating the job of packaging quite a bit. A lot of language specific package management systems get lift from offloading the ugly C support problems to other layers. See python and "manylinux" for instance.
Not to drag on Rust, but I can see a "global optimization" problem when I try to cargo build a moderate Rust project and have to download hundreds of transitive dependencies, and have a good chance of ending up not building a thing (for reasons that probably include my own incompetence, but still).
There are of course tons of C projects that have similar problems in a way - the number of dependencies might be lower but I have to hunt them manually. But well-engineered projects are at most a handful of clearly identifiable dependencies, and everything will generally build without fuss. This is how it is with the Linux kernel today. The main dependency of the Linux kernel is gcc, and as far as I perceive, communication between Linux and gcc projects is alive.
I think there is a lesson that can be learned in particular from C development: There is value in growing a system vs mainly integrating existing stuff (junk or not). The larger a project gets, the more sense it makes for it to bring its own tools and implementations so it can continue to build and be maintained as a whole without a lot of complications.
This might be the difference between a huge project and an ecosystem, and the question is where are people aiming with Rust support? If it's just about drivers I can see some limited value in the ecosystem approach but it will be a tough sell for any Rust driver that wants its 'M' in the kernel config.
Even in the very worst-case scenario, you can fork the code. That would still leave you far ahead of having to write everything you need from scratch. And my experience is that plenty of libraries are happy to support general use cases when at all possible.
> and have to download hundreds of transitive dependencies, and have a good chance of ending up not building a thing
Can you give an example of what you mean by this? I'm unclear what the concern is regarding the idea of "not building a thing".
> as far as I perceive, communication between Linux and gcc projects is alive
The Rust developers also have open and active lines of communication to the Linux developers.
> The larger a project gets, the more sense it makes for it to bring its own tools and implementations so it can continue to build and be maintained as a whole without a lot of complications.
Certainly, but even very large projects benefit from sharing code in foundational areas.
You can fork the code that you never knew well enough to write yourself.
>> and have to download hundreds of transitive dependencies, and have a good chance of ending up not building a thing
> Can you give an example of what you mean by this? I'm unclear what the concern is.
My concern is of course that as someone who wants, or is supposed to, mess with a project, I can easily get depressed if I have to invest significant to infinite amounts of energy to bring that project to build. And if I get it to build, have a hard time figuring out how it works because what it does is all over the place, hidden in hundreds of libraries.
> Even very large projects benefit from sharing code in foundational areas.
"Other crates that I'd like: anyhow, bincode, byteorder, log, once_cell, pin-project, rand, serde, slab, static_assertions, uuid plus some more esoteric ones."
Certainly. It is hard to envision any circumstance where one is worse off for having the opportunity to stand on the shoulders of giants rather than having to invent the universe from scratch.
> I can easily get depressed if I have to invest significant to infinite amounts of energy to bring that project to build
Can you give an example of a Rust project where you have had this experience? Nearly every Rust project is as simple to build as `git clone && cargo build`. The exceptions are those with C dependencies, where you will need to turn to your C package manager (apt, etc.) to first install the C dependencies, and it's hard to claim this as a weakness of Rust relative to C when it's no worse than what you'd get in a pure C project.
> Other crates that I'd like:
Can you be more specific about which of these crates provides functionality which you do not think qualifies as generally foundational?
Trying to make a better experience I just randomly picked "Weylus" from Github's rust/trending list and wasted more than 1 hour building it on Windows 10 and Debian Bullseye, ultimately quitting both. Yes, it was a lot of C library related issues, but also other executables missing, like tsc, or in case of Windows, make. On both systems, the number of transitive dependencies was about 300, and so I guess it was inevitable that there were going to be some problems.
I've had similar problems before that trying to build other, less ambitious programs than Weylus using cargo.
> it's hard to claim this as a weakness of Rust relative to C when it's no worse than what you'd get in a pure C project.
It seems to me that making it more reliable to depend on other modules (like cargo does) does not lead to easier to use projects, since it doesn't change the amount of bullshit that most developers and users are willing to endure. This is an instance of Parkinson's law and I think it was already a learning from the node.js ecosystem.
As a consequence, with a system like cargo programs end up having far more dependencies (which is empirically true), but the programs aren't easier to build by a random user. For developers, the situation is changed compared to e.g. C, in that it is easier to add more dependencies before the software becomes unmaintainable. This might shift the development situation to a place where the average developer in this ecosystem is more competent in plumbing things than understanding the problems and coming up with reliable solutions that are easy to maintain without a system like cargo.
I'm not sure if this is good or bad - probably it's an evolution that can't be stopped, and which creates new types of developers. What I'm saying is I'm not positive that this attitude is compatible with a central piece of infrastructure, like the Linux kernel is.
> Can you be more specific about which of these crates provides functionality which you do not think qualifies as generally foundational?
I'm not a Rust developer and I don't know any of these, but looking at those names I'm wondering: which is not some arbitrary helper thing, where it would be easy to maintain an alternative in-tree? At least that's better than each developer bringing their own slightly different preference, resulting in more bloat and maintainability problems.
It would be nowhere near enough to make it safe to download untrusted dependencies, but it would at least stop the most trivial forms of attack.
Sandboxed dependencies with user-specified capabilities are a very, very long way away, if they ever happen.
If you want to have more fine grained white listing, like only grant access to a certain directory, this could get really messy quick, trying to solve this at compile time.
When was the last time a major distribution found a backdoor in a popular package?
Packagers not finding a backdoor doesn't mean that there isn't one. How many packagers actively audit the code they support for a given distro? It is not uncommon for distros that support esoteric platforms will claim a given package works for that platform because it compiles, but it reliably segfaults on execution. Who's responsible for that? Packagers have even introduced[1] vulnerabilities by "fixing" code they didn't fully understand at the time.
Packagers have a difficult, thankless task, and we're doing them no favors by being confused at what their job is. They ensure that the package builds, integrates with the rest of the distribution as much as possible and updates/patches swiftly when issues are found upstream.
Nice stramwan there - when the alternative is trusting random strangers on github.
> How many packagers actively audit the code they support for a given distro?
Many, plus large companies do plenty of vetting and indemnification on popular distros.
There are very large contracts involved in this. Do you think the typical bank installs random stuff from the Internet on their payment processors?
> Packagers have even introduced[1] vulnerabilities by "fixing" code they didn't fully understand at the time.
Another strawman. How many vulnerabilties have been prevented or fixed by packagers? Quite a good number.
> we're doing them no favors by being confused at what their job is.
Speak for yourself.
Maybe the rust devs can also learn something worthwhile from working with the kernel community.
Spinlocks are great in the kernel where we can just mask interrupts. Not so fantastic in userspace.
Intrusive linked structures are necessary in the kernel since they let us manipulate collections without allocating. But they are also less convenient. Etc.
However, I'm sure there are lots of things which could be useful in both places. I have from time to time toyed with the idea of porting the kernel "standard library" API into userspace. Maybe something like that already exists?
Why ? (un-opionated "why")
To reuse the migrant melting pot metaphor used at the end of the article, the "kernel has to stand alone" seems to mirror the states wanting a "strong border" (while the idea of borders and states has a beginning and will surely end).
I think in your view in melting pot metaphore 'Kernel has to stand alone' might be a lot like christian countries holding the view "Sunday is a forced day of because it is the day of the lord". Whereas I think it might be more like e.g. the Netherlands having the cultural idea of "we should actually work together to prevent the entire country flooding". Or maybe the Spanish "taking a break during midday is smart, not lazy". Its a cultural preference that arises as an adaption to the specific geographic reality of the country.
There are lots of good arguments against pulling in lots of rust dependencies - it's too much code to audit, maintained by too many different upstream people and teams, lots more tooling would be needed to make sure all the dependencies are safe to use in the kernel, and on and on - but this one about an internet connection is both the most common one I've seen here and the most superficial and frankly silly.
So I agree, this particular criticism is very confusing. Though as I said, that doesn't mean no criticism is warranted!
'rustc' does not, but we are talking about cargo here.
Thank you!
This has to be one of the most insane threads I've read in a while.
10-15 years ago I'd agree with you, now... that would actually surprise me.
(But why wouldn't you have readers for all 3, potentially as peripherals if you don't have a desktop ?)
Linux package maintainers do the vetting. Buildbots build in a clean room environment, without Internet access.
If you mix up the steps supply chain security is gone.
Working offline is needed
I don't know of such a Rust kernel being worked on that is not some hobby side project but something more like Servo was, where you have paid experienced in domain developers.
https://www.linuxfoundation.org/press-release/linux-foundati...
But if Walmart needed a kernel for some reason, and it would provide value to them, they could afford it, it's about a quarter's worth of profit. But, again, Linux would likely do anything they needed, and if it was missing something they could add it.
You do not need to get all Linux features at start, I would focus on the server stuff, support the popular server hardware and popular server work so filesystems and networking. I would hope some competent developers would write a new kernel from scratch then getting 10% of some Linux rustiffied. When you edit shit you need to make sure not break shit, so you are limited, you move slow and you have to implement backwards compatibility. Amazon,Google,Facebook could work on this if they care for security but probably using VMs is cheaper for them.
Not that I think it would make the point of having a bright new kernel easily compete with such a huge and firmly grounded project as Linux though.
All the more, there already are multiple projects using Rust to build OSes out there, if I'm not mistaken.
The exception to this is when you can develop something and have a good case to use it yourself, like Google and Fuchsia. But I doubt Fuchsia will gain wide organic adoption because of the same factors.
This would mean that the only OSes are first created by hobbyists, then fanboys use it then a company adopts it. But there are commercial OSes and commercial software that are created by a company to solve a problem and not by random dudes and companies then adopts it.
IMO is sad that we the tech industry are stuck with old OSes and old architecture while you have this giants that use open source software to make billions and they could find researching and creating of a few new revolutionary OSes like it happened in the past. And I don't mean desktop OS that will defeat Windows.
Vendors aren't going to write drivers for your hobby kernel. No one is using your hobby kernel. Bootstrapping a new kernel without billions of dollars to invest in development time is almost impossible, and anyone who is investing billions of dollars is likely going to have dubious proprietary reasons for doing so.
A successful kernel is Rust is probably the worst thing that could happen to the open source community.
But if you are Google and you believe that this Rust kernel is super safe and fast and has clean code, and parallelism,async and candy ... how many drivers do google Data centers use ?
After you prove the kernel is real good by using it in data centers and devices that need security you can slowly expand, for most devices you could create soem compatibility layer. Who knows if there are soem competent developeres hired to work on it they might use some better architecture, like keep the drivers outside the kernel.
Even if kernel devs are careful to avoid the usual pitfalls, it's just a bad API that encourages use of null-terminated strings elsewhere. If you're using other, better string types elsewhere, you often need to make copies to then make system calls (the article hints at that with the addition of a CString type).
Null terminated strings are a poor choice to converge on for high level application code in particular. It's just an inefficient (expensive substrings, redundant length calculation, unclear ownership) and unsafe default for too many use cases.
I don't mind certain performance sensitive applications using C strings to avoid extra registers, though even that is a premature optimization in certain situations.
[1] See Limitations: https://en.m.wikipedia.org/wiki/Null-terminated_string
As an example: why can a process only have one current working directory? Wouldn't it be nice to be able to have a process maintain pointers to two or more locations in the filesystem at once? Wouldn't it make software more modular if a library could "chdir" into some directory without worrying about breaking the application that depends on it? The filesystem APIs could be extended with a "CWD" handle argument that can be passed around sort of like a file descriptor instead of having one implicit CWD for the whole process.
The same could be done with UIDs. Why not have processes that can use multiple UIDs? Again, you could have UID parameters to API functions that require authorization.
POSIX is pretty good by the standards of 80's computers, but in a lot of ways it's showing its age. We can do better. But it's kind of depressing that OS interface design is treated as a solved problem, and so those interfaces stagnate.
Why would you even want a working directory at that point?
I think the idea of a current working directory is a reasonable one, it's just that the limitation that a process can only have one CWD at a time is kind of arbitrary when you think about it.
I could even see having commands that take multiple CWDs. Like a move operation could take a source and a destination CWD as an alternative to specifying source and destination paths.
The kernel being literally the "bottom" of a system presents a challenge as this space rejects complexity and tooling variety that is intrinsic to "higher-level" languages.
What does "we need approach and definition for operating systems" actually mean? (If you know, that is.)
Also, it's the Linux kernel, not just the kernel. There are more than one, and every operating system has one[0]. You might be interested in looking at the Mach kernel (Apple[1]) or Google's new Zircon kernel (in Fuchsia).
Both of those are microkernels, and as such minimise the amount of work that the kernel does, as well as removing drivers to userspace. Inasmuch as I can extract any meaning from your comment, it seems like your problem might be with the size and scope of Linux's kernel, in which case those might appeal to you.
[0] I'm sure this absolute claim will summon someone to point out some recondite 1980s operating system that doesn't have a kernel.
[1] Yes, yes, used by Apple.
Rust has been a very stable language since Rust 1.0. They have a stellar record of keeping things working - with most code breaking being due to said code invoking UB. The edition system is a brilliant invention that allows evolving the language _without_ causing an ecosystem split. Thanks to this, Rust ends up having a much better stability story than even C++ (for whom you can't really mix and match different C++ versions).
The bootstrapping situation is really not that bad? We have mrustc (A C++ rust->C transpiler) which allows compiling modern versions of rustc (latest supported rustc version being 1.54), which we can then iteratively bootstrap from up to the latest version. And things are getting better, with gccrs[0] in particular promising a rust frontend for GCC, written in C.
As for the "curl something|bash", I suppose you're talking about rustup. You're free to download the script, and review it before installing it. And rust is also distributed many different ways. At least `curl something|bash` does not require root account, unlike `sudo apt install`, which can be very convenient. Like all things: Multiple options are generally better.
Yes, I know it sucks for all of the distribution maintainers who want to bootstrap every package from scratch, and I do feel for them, but that's a very niche thing to do which the vast majority of people will not do.
int concept = 4;
template <class T> concept delicious = true;
These are both valid C++, it's just that they aren't simultaneously valid in any of the half dozen distinct versions of standard C++.
Because C compilers, make, etc. are really stable, existing Linux source tarballs have a habit of building fine for decades after initial release.
If you wrote some Rust 1.0 code back in 2015, put it away in a drawer (maybe on a USB stick) but now get it out today it will still build with current Rust tooling -- except for some very narrow cases where you might have done something inherently unsound and subsequently the compiler was corrected to fix that so it's an error.
Rust's language shifted slightly in those years. However the Editions system - even though it hadn't yet been invented in 2015 - allows for that. Your 2015 code lacks any Edition metadata, so, the modern tools understand that as Rust 2015 edition, and will compile accordingly, while still interoperating correctly with modern Rust.
Suppose your 2015 code named a variable await. That's a keyword in modern Rust. But it isn't a keyword in Rust 2015 edition, so your code compiles just fine. This would not work in C of course, which is why its newer keywords are ugly stuff like _Bool but in Rust it's fine.
But that also means that crates used by the kernel would need to likewise be conservative about updating to new Editions so as to not break expected support surfaces.
But also, all previous versions of published crates are kept indefinitely. If Linux wants serde v1.0.240 then that's fine, even if subsequently serde shipped v1.0.241, v1.1.14 and v2.0.1 the repository holds on to everything.
Is C ecosystems much better?
Since you haven't, it seems a bit ironic that your are complaining about those who didn't.
I see Rust as a replacement for C long term. From this perspective it's absolutely ok to have a transition period (years in fact) where there are multiple toolchains used to build the kernel.
[0] i.e. no standard library ('no_std'), no unwinding panics, no dynamically sized types, &c.
Just as web developers, of all people, are finally, after over a decade, coming to the conclusion that arbitrarily-packaged dependencies are a bad idea for so many obvious reasons, Rust developers are trying to replicate NPM in the kernel.
I was told that Rust in the kernel would be a good thing, and that not much would change. If these are the types of people who are writing Rust for the kernel... can we go back? I really want to go back.
The article states:
> The Rust-for-Linux developers understand this situation and are not envisioning adding the ability to pull in modules with a tool like Cargo
Some? Dependency graphs of well over a hundred packages are ubiquitous. I've long since stopped being surprised when I compile a rust package that makes no network connections and see it (indirectly) pulling in multiple HTTPS libraries.
Tad understated. Popular rust http frameworks with basic functionality are hitting 300+ deps straight off the bat easily.
For the security conscious, move to vendoring or alias cargo to always run in offline mode.
Web developers and Rust developers are the same group
Not that I agree with Rust's adoption at that level, as I think languages with automatic memory management make more sense, but that's me.
But I guess some people just really want the unicode ukrainian flags being appended to every email they send or something and are too lazy to implement it themselves. :P
If someone suggested including a C dependency manager in the kernel, does that mean that the use of the entire C language in kernel should be discredited?
It is fair to say that the developer suggesting it (especially considering some of the proposed crates they want) has some massive misconceptions about the constraints of current Linux kernel development though.
Rust is very slowly winning (but it’s a really hard space and this is impressive). But some people have to come and try snatching defeat from the jaws of victory.
(sigh)
> Other crates that I'd like: anyhow, bincode, byteorder, log, once_cell, pin-project, rand, serde, slab, static_assertions, uuid plus some more esoteric ones.
This makes Rust devs look infantile, with no real understanding of what it means to develop for the kernel. Or what it takes to develop security and performance critical software of any kind, frankly.
I don't understand why it looks bad from a performance or security point of view. These crate are well established and developed by competant devs that also care about security and performance.
Linux has its own UUID infrastructure, so it seems weird to pull in a whole parallel implementation. At the very least you'd want to disable all the generation code, and it's unclear to me if the uuid crate supports that.
rand is large. Does Linux need all the probability distributions it supports? (ripgrepping for "Bernoulli" only turns up the name of a floppy disk system, so probably not.) Doesn't it already have an implementation for the ones it does need?
I would disagree with that assertion: the kernel is a service that end user applications interact with through an API. Using anyhow in a Rust library that Rust applications use is a very bad idea. Using anyhow in the kernel might be a bad idea, but not for that stated reason.
> Linux has its own UUID infrastructure, so it seems weird to pull in a whole parallel implementation. At the very least you'd want to disable all the generation code, and it's unclear to me if the uuid crate supports that.
The uuid crate can be modified to support the kernel infrastructure as its backend. This would allow the same API to be available within the kernel and in the rest of the ecosystem. That's beneficial to everyone involved.
The most far-fetched crate there would be pin-project I think, because a lot of other work would have to be done to make Rust async code work in the kernel.
Pin's usefulness is not limited to async. Any type that contains a pointer relative to itself benefits from being wrapped in it to avoid accidental misuse.
infantile?
i mean, serde in the kernel is overkill, but not every crate is like that. Rust has been designed in the internet era and leverages dependencies for things older languages would ship with (rand!)
Weird justification for missing an extremely central and extraordinarily important feature…
How does rand being a crate make Rust ”internet era“ in any way?
Can you think of any reason why decoupling it from the language itself would be a good idea? Why does it need to be baked in, and what problems would arise if it was? How do these problems trade off against having people just run “cargo add rand”?
Having an boundlessly large and thoroughly unauditable trusted computing base is extremely "Internet" -- or "webscale" to be more precise.
I just compiled the blinky example for a cortex m4 board (the example in the rust hal crate for that board, not using any unsafe), and the striped binary is 420 bytes.
So you probably want to review your stence, because you're about 3 orders of magnitude off.
That's, uh, not a very compelling request.