Committing to Rust for Kernel Code
lwn.net
lwn.net
https://github.com/ClangBuiltLinux/linux/issues is our bug tracker for known issues (a few are tracked in llvm's issue tracker). Bug reporters and future kernel hackers wanted!
As I mentioned on mastodon, there's lots of bugs still to be fixed everywhere, but even if we don't fix them, providing competition in the toolchain space has been worth it to users.
You have to Google everything and then you find out about EXPLICIT_LIBGCC_OPT_IN and you read unofficial responses to issues that describe how to use libunwind and compilert to replace libgcc_s. But all that stuff is out of date fast and never works. Even Rene Rebe who developed T2 ranted about this in 2019 and he couldn't completely figure it out without doing a shitton of workarounds and patches. You need to be fucking Stallmann or Torvalds or whatever computer scientist genius themselves to achieve this. Or Google who did this for Android and Fuchsia. My point is it is rocket science at this point. Fucking make that straightforward already dear LLVM project.
> Fucking make that straightforward already dear LLVM project.
Much of what you are asking for can be enabled by not-rocket-science changes to the driver. Give it a try.
The LLVM project is much more "bazaar" than other high profile Open Source projects, so I think you'd have a relatively easy time convincing folks to accept your changes provided they don't break existing tests.
The parts that make it only "ok" are largely cmake derived problems - yes there's sometimes incomplete docs and so on, but largely the confusing parts come from the multi-project split and cmake reconstitution. I'm particularly excited for a future when llvm-libc has made some more progress, at which point a very hermetic build will get quite a bit easier.
The thing is that if you're cross-building this is really the answer, cross building with the deeply intertwined gcc chains is an equally awful experience. I'll take a static clang tool build I can zip up and give to everyone (except nix, because they decided loader hell should only be played on max difficulty) over a fragile distro intertwined mess any day.
That's a very niche use-cases. Most people don't care about being "completely GCC/binutils/gnu libc independent".
It's fine to care about your niche use case, but to claim a project is not "serious" because this one thing isn't supported is ridiculous.
Then what is the point of llvm ?
Is this another "make" ("we don't use make for our project but use cmake to generate makefiles and make to build") ?
Edit: forgot to mention, ThinLTO from 6.6.2 :)
https://threadreaderapp.com/thread/1577667445719912450.html
At $WORK we are converting a graphics processing application written in C to Rust and the experience has been similarly enlightening.
Learning Rust has been a process, but the dividends are real and exciting.
Biggest takeaways:
Rust's ownership rules enforce "multiple readers, one writer" patterns so plan for such patterns in your design.
Embrace Rust's Result and Option types for error-handling and design. They are elegant and powerful when used with 'match'.
Unit tests right next to the code they test is a huge win. Doc-tests and 'cargo doc' is literate programming (https://en.wikipedia.org/wiki/Literate_programming).
The phf crate (https://docs.rs/phf/latest/phf/) is awesome for global constant data tables.
>> Was there anything that surprised you?
Our Rust code is much more concise than the previous C code. It takes about three times as much C code versus Rust code, but the C code feels much more brittle and less portable.
Not to say I couldn't have gotten the same education from another language, but something about Rust clicked for me that others didn't.
An example of this is generating a JSON and only learning the correct schema at runtime.
Or connecting to an external database with tables of unknown structures that you can’t type check.
You can, of course deal with all these by wrapping the objects in some structure; but it’s just one more thing you need to keep in mind.
In rust you just make a JsonValue enum that represents the values and the data types that you're handling.
In something like Python, you just use the built-in runtime value type to represent the structure.
Either way you still have to plumb up the data ingestation and your codepaths that you want to use the dynamic data in. For example you still have to check if the "name" key's value is a string and not an object, array etc.
Otherwise, you probably could just use some accessors that access the values willy nilly that coerce them to text, return an error or panic, that could also exist in rust (some of these things you would have to wire up manually, some are provided by the language and others are fixed by just annotating structs.)
To summarize, fail to see the point you're trying to make, do you have some specific example you are thinking of that I can take a look at?
Something that I am interested in doing is generating a json schema at runtime, and then parsing some incoming json to see if that matches the target schema. Do you know of any crate already providing this kind of functionality?
When looking for crates lib.rs is an excellent search utility for searching crates that you would need. For example: https://lib.rs/search?q=json
I think it fits the bill. I was doing this on python side and aimed to use Pydantic and this looks close enough.
Some of my python scripts which used to take 24 hours to process data now can finish in under an hour. It’s like magic. Granted theres a development cost so I don’t convert everything but it almost feels like cheating watching it go.
Another thing, it seems that the strictness of the language really cuts down on nasty runtime issues and silent bugs. Rust forces you to state explicitly what you want to do.
Hope to contribute patches in rust to kernel in the near future!
Conversely, Rust never lets you do anything above a mid-level effortlessly. Traits barely get you halfway to typeclasses and you still have to mess with the borrow checker no matter what you’re doing. Add in lifetimes and even simple high-level code gets bogged down with extraneous low-level details.
1. Sticking with C for the rest of eternity for the linux kernel and accepting the baggage which comes with that
2. Migrating to rust and accepting the regressions which will inevitably appear as code is rewritten
Bugs can be fixed.
https://syzkaller.appspot.com/upstream/graph/bugs
So, yes, all bugs can be fixed. And yes, the world produces enough food to feed the entire population. But people are still going hungry.
I don't understand where all these memory bugs are coming from. Are programmers not testing their code? Are they not running sanitizers, fuzzers, and static analysis? Switching programming languages isn't going to fix broken development process.
Even C++ is at least somewhat better, with smart pointers and RAII.
As you suggested though, static analysis has lots of scope for solutions. The best static analyzer around is a little thing called rustc :)
It's true that Rust is currently the only stable language in this space, so it has some advantage there, but "Rust or forever C, there are no other options" is of course a load of nonsense.
> Torvalds answered that, while he used to find problems in the LLVM Clang compiler, now he's more likely to find problems with GCC instead; he now builds with Clang.
> At the conclusion, Torvalds pointed out that there have been problems over the years with GCC changes breaking the kernel; the same will surely happen with Rust, but it will be the same thing in the end.
Based Linus Torvalds.
I actively avoid Rust projects because they take too long to compile.
I wish rustc had an option for disabling monomorphization, a true -Os option.
> I doubt that a hypothetical version of Rust that avoided monomorphization would compile any faster. I remember doing experiments to that effect in the early days and found that monomorphization wasn't really slower. That's because all the runtime bookkeeping necessary to operate on value types generically adds up to a ton of code that has to be optimized away, and it ends up a wash in the end. As a point of comparison, Swift does all this bookkeeping, and it's not appreciably faster to compile than Rust; Swift goes this route for ABI stability reasons, not for compiler performance.
> What you would need to go faster would be not only a non-monomorphizing compiler but also boxed types. That would be a very different language, one higher-level than even Go (which monomorphizes generics).
Edit: Also kernel compile times are a problem for the elite few, while kernel bugs are a problem for everyone.
If you have to restart an embedded system or schedule a job on a cluster to properly test, that can work out to huge savings.
It would still be nice to see them substantially improve. I know there’s been effort to get debug mode to build substantially faster with cranelift, but I forget if that’s in stable now.
Still, I would say Rust is fast enough now that compile time isn't a show stopper.
A host of other issues, which Rust notably doesn't suffer from, are almost "unfixable", even in the long term: think build system, type system, memory safety. Rust is fine in these regards.
Standard library actually shouldn’t increase compile time at all really, since it’s prebuilt and included for most platforms (excepting -Z build-std).
My hardware is an 11th gen Intel i7 (4-core, 8-thread) laptop. I included time to build but did excluded time to fetch dependencies. I measured real (wall) time and user time with time(1). I compiled clean release builds in both cases (I think). Here's what I found:
uutils/coreutils: I fetched dependencies (`cargo fetch`) ahead of time, then just ran `time cargo build --release` to build everything. This took 1m37s wall time, 4m19s user time. It looked like about half of that time was spent compiling the `coreutils` multicall (Busybox-style) binary itself, after compiling all the individual `uu_*` utilities (`uu_cat`, etc.).
GNU coreutils: I fetched dependencies (`git submodule update --init --recursive`) ahead of time, then read `README-hacking`, which indicates a three-step build process:
time ./bootstrap: 2m42s wall time, 4m32s user time
time ./configure: 0m36s wall time, 0m21s user time
time make -j8: 0m21s wall time, 2m24s user time
So, if we're measuring "time for clean release build on a new dev's machine (not including network fetches)", we're looking at ~200s for GNU coreutils and 97s for uutils/coreutils. If we discount ./bootstrap for whatever reason, that goes down to 57s vs. 97s. Either way, it seems within a factor of roughly 2, with the winner going one way or another depending on how you count.
Methodology comments welcome.
Using a different linker (like mold) might change that, if most of the time is spent linking everything together.
$ readelf -p .comment ./target/release/coreutils
String dump of section '.comment':
[ 0] rustc version 1.74.0 (79e9716c9 2023-11-13)
[ 2c] GCC: (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
[ 57] mold 2.3.3 (49066ea329979c3187556091ba62421594799fd1; compatible with GNU ld)
[ a6] GCC: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
(Edited for new timings with mold 2.3.3 instead of 1.0.3.)
Can this power be learned?
If there are any Rust experts around...what am I missing? The way I see it, Rust is still fundamentally designed to work at a higher level of abstraction than C, and is still mostly dependent on C (even C++ really, due to LLVM/Clang). And bare metal or "first compiler" support for bare metal doesn't seem like it's really intended to be a first class use case. At least that's my impression of the ecosystem from the little time I've had to play around with it.
Is this mostly just a thing to get more young people interested in kernel development...allowing them to start out in less important areas and in a language they are passionate about? Or is this a serious proposal about the future of operating systems and other low level infrastructure code? Do you just program everything in unsafe mode? What about runtimes?
It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd. Whereas for myself and most of the C programmers I know, Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does. I could spend hours writing examples of this but it's even apparent in the way the Zig team has handled its relationship with LLVM, where they seem very serious about trying to not accept it as fundamental to their language and eventually even eliminate the dependency on the C++ code...with Rust it doesn't seem like this is even on the minds of most of the users. It's purely a dependency for them and that isn't seen as being fundamentally at odds with the intended use cases. That is totally okay...but it ain't the C culture if we are to accept that such a thing exists.
Rust is not fundamentally designed to work at a higher level of abstraction than C. Rust is designed to work at a range of abstraction levels, from as low as the lowest level C code to something fairly high (but still short of a language like python). Rust is designed to make it easy to quickly build abstractions on top of super low level (rust) code, because usually that's the less error prone way to do things - that doesn't prevent it from working at those levels though, and if anything it makes working on systems where you have to work on super low level code much easier to use.
When working with low level rust you don't "program everything in unsafe mode", you program a few really low level bits in unsafe rust, but quickly make abstractions that allow you to avoid making the vast majority of your code unsafe. This is opposed to languages like C/C++/zig/... where there isn't anything but "unsafe".
Rust doesn't really have a runtime by default (beyond libc for non-baremetal code).
There are a few tiny niches where rust isn't designed to compete with C, but they really are tiny niches. For example where chars aren't 8 bits.
I'm honestly just not following what you're saying about C++. Rust code tends not to depend on C++ code apart from maybe some system libraries that everyone depends on. The rust community tends to be overly-keen on rewriting those libraries in rust, not underly-keen. Obviously some people will make code that manages to depend on the language - but rust doesn't make it easy (like zig does...).
As for the relationship with llvm - rust doesn't actually have a hard dependency on it anymore (with support for a compiler backend called cranelift, written in rust). I also just don't see "the compiler depends on another language" as a huge impediment - as others have mentioned C compilers tend to as well...
Not likely. At the moment you need to do extra work to get Rust working well. It's not exactly beginner friendly and doing work in the kernel, you'll need to dig into C anyway.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
Serious code already exists, so... Yes?
> Do you just program everything in unsafe mode? What about runtimes?
Why would you? You need that only when interfacing with something that can't hold the Rust compiler assumptions. See for example https://github.com/AsahiLinux/linux/blob/gpu/rebase-6.4/driv...
The few places that need direct access / unsafe are almost all single-line areas with an explanation.
> but it ain't the C culture if we are to accept that such a thing exists.
I don't get that whole part of what you wrote. What seems to be the culture / why does it matter in the technical sense? Some parts of the compiler are written in C++ - what's the specific issue here?
Yes Rust helps bring in the next generation of Linux devs. It needs to to survive. But it's not because young people like Rust, it's because C developers like rust too.
Yes, Rust has unsafe and unsafe rust is trickier to get right, but you use it opt in and it's only a small amount if the code. Zig is still having unsoundness bugs in relatively simple code
Speaking as someone going the other way, just started working with C full time.
Yes. Oh Dog, Yes!
Data structures that rely on the memory layout semantics of the C compiler and casting void * is very cool, very 1990s, and does my head in!
And from what I see, Rust can sort of make it easier by making you think about the lifetime of the memory while coding and that's why I really wanted to learn Rust
I attempted to port a low latency trading system from C to Rust last year, but ended up abandoning after 6 month of development.
The overall feeling that I got is that Rust made my life a dream for 80% of the codebase, the _non critical, non high performance part_.
There are tons of well written libraries, cargo is awesome, the performance is on par with what C++ would do, etc.
But the last 20% of the code base, the high performance part, was a nightmare to implement. The Rust static safety basically get in your way constantly for any kind of non obvious memory layout (self reference, etc). Dynamic safety (cells, Rcs, etc) just add too many overhead for the the critical path, and the "escape hatch" of "unsafe {}" ended up being 1000x more error prone that C.
I keep some resentment against the Rust community from this experience to be honest. I felt like instead of understanding the constraints & limitations of a fixed, microsecond time budget that I had, and trying to find solutions and be open about possible improvements, the overall trend was more in trying to defend Rust with whatever it takes.
Even at the surface, if your desired solution is so antithetical to the type of structure that Rust tries to push you towards, you can quite literally just write this component of your code as if it was C. Just using pointers instead of borrows and tag all your functions as unsafe and… pretend you’re writing C? At that point you’ve, I think, more or less disabled 95%+ of the bits that would prohibit you from writing code exactly as you would in C. You don’t get the benefits of Rust for that part, but at least you get them for the remaining 80%.
Or you can very literally just write the “hard part” in C and call it from Rust. You might have to make sure you can’t panic across language boundaries, but other than that the C interop is just about the best I’ve seen from any language.
I also don’t entirely understand how unsafe can be 1,000x more (or even 1.5x more) error prone than C. But I’d love to hear how. The only “trick” to unsafe is that you should aim for your unsafe blocks to be “unit-safe”. Meaning they might do something unsafe inside, but from the outside looking at it as a black box, the unit functionality of it should be safe. I don’t think the docs do a good enough job here of encouraging that style of design. You can violate this guideline, but doing so without sufficient care is quite likely to result in bugs. But of course if you did the same approach in C, you’d have a similar outcome.
The only real way I can reconcile your points is if the performance-critical bits that seriously impact the design of your program are scattered uniformly and don’t have anything resembling clean boundaries. I suppose that’s a real possibility but it does seem very foreign to me.
I sort of imagined that I could get the best of both world, and just "unsafe { <C style code> }" my way out for performance critical things.
But the thing is, the static safety boundaries of Rust allow the compiler to make much, much tighter assumptions than C & C++, especially around aliasing rules, un initialized memory, and moves.
When you relax these boundaries with "unsafe {}", you don't enter "C world", you enter the litteral gates of hell where any innocent temporary cast can throw you in a random load/store reordering bug.
There are still some Rust-specific details you would still need to handle—as you mentioned, uninitialized memory—but for that one specifically I haven’t found MaybeUninit to be particularly cumbersome.
A low latency trading system can't have Rc, or any C equivalent of it, in the critical path, because it does allocation. More generally, your story doesn't ring true to me, because the performance-critical parts of a low latency trading system have to be so simple that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures.
One of my colleagues wrote a low latency trading system in Rust. He had to learn Rust to do it, already knew C, and today thinks this was the right decision.
Hu, so it doesn't ring true because... you feel like it's too complicated from what you imagined? That's a first.
> that there isn't scope for any of the tricky bits you talk about, they have to be braindead simple loops over simple data structures
That is far from the reality of low latency trading systems. Let's just look at the very first building blocks:
1) You will need some form of userspace packet polling, along with its associated multicast A/B line arbitration. That alone is not "simple braindead loops", and we're just talking about getting 1 UDP datagram here.
2) You will need some form of packet queueing before decoding, so that you do not drop packets under bursts. That most likely means some variant of lock free ring buffer in shared memory. Far from "a bunch of brain dead loops" as well.
3) You will need to decode packets and maintain an order book state. That mostly implies a purpose built b+tree. If you've implemented any of those, you will know it's also far from simple.
4) You will have to compute some indicators based on order book state, which most likely will lead you to some form of linear algebra.
5) You will need to disseminate such indicators to perform some simple close formula variant of portfolio optimization. More linear algebra.
6) etc.
I'm happy your colleague that he managed to do all that without struggling, that was not my experience, that's all. But to say that it's "just braindead loops over simple datastructures" is either a sign he didn't do a _real_ low latency system, or he just was somehow spared from the real underlying complexity.
The problem is, the things you needed to do to stay alive in this environment are antipatterns in Rust and you’ve learned the hard way Rust actively opposes being fed Rust antipatterns.
Self referential data structures are some of the hardest problems in CS to get right. There are patterns that work with Rust which replace pointers with eg generational indexes. You’ll find people do the same thing in C++ once they get burnt one time too many. I think you’d have a very different experience if you approached your system with an ECS framework, either off the shelf or roll your own.
IMO this is the biggest issue right now. `unsafe` needs some refinement, certainly.
> the overall trend was more in trying to defend Rust with whatever it takes.
This can sometimes be the case, unfortunately. But fwiw I think a lot of us in the community agree entirely that there's room to improve in the unsafe world.
I'm not following the assertion that compilation with LLVM/Clang introduces a C++ dependency on the language being compiled (Rust here).
> In August 2012, the GCC steering committee announced that GCC now uses C++ as its implementation language.[57] This means that to build GCC from sources, a C++ compiler is required that understands ISO/IEC C++03 standard. > On May 18, 2020, GCC moved away from ISO/IEC C++03 standard to ISO/IEC C++11 standard (i.e. needed to compile, bootstrap, the compiler itself; by default it however compiles later versions of C++).[58]
So for the past >10 years, gcc and clang have both had a C++ dependency. What is unique about LLVM/Clang here?
Alternatively i consider a compiler might only be presented as a binary, with no visibility as to its own source code, so i consider in that case just a Rust program and a binary that compiles it to a native binary linking with binary object files also of unspecified source code languages.
Rust gives you the option of opting out of that `#[no_std]`, which removes some basic tools, such as allocations, panics etc (I haven't used it much, so this may not be fully precise). As such with no_std you do move closer to what you'd consider C development.
That said rust will still have its restrictions, so some of the wild things you can do in C, isn't possible in rust, at least not without really violating the inner workings of it.
But with unsafe you can technically do most things on the level of C, without the overhead, that C++ would provide, via. its abstractions.
I'd expect the Linux Kernel to move in the direction of developing or using their own libraries and data structures, and that becoming its own bespoke part of their use of the rust language. Such that rust only really provides its safety guarantees and ergonomics. Compile times are also quite quick with no_std, as rust only really struggle with compile times, if you have lots of dependencies.
Circling back:
Rust is more like C++ with std library enabled (default) and C with `no_std`
Yes, the Rust language needs a complex optimizing compiler to really be viable. In this respect it is more limiting than ANSI/ISO C. For a lot of contexts, this isn't really a problem though? You aren't going to be able to build the Linux kernel without GCC or Clang anyway…
Yes
That is the point
Those abstractions are available to the programmer and compile down as tight and efficient as C
The available compilers targeting your microcontroller certainly matter, though. You certainly still find lots of options that aren't Rust-compatible, but a non-trivial number microcontrollers are ARM or RISC-V based now, and can be targeted by LLVM/Rust.
What’s the point of high-level languages at all if they’re just going to be implemented on top of a CPU that will happily chase raw pointers and divide by zero if you ask it to?
Because everything else can be expressed safely and more concisely than writing it in C or raw asm. And you have to pay virtually zero abstraction costs for the pleasure: see Rust’s comparative performance in language shootouts. Not infrequently Rust even beats C’s performance due to better visibility for optimizations.
And when you do still have to resort to unsafe low-level bit twiddling, you get to keep all the other benefits of having a modern, powerful language.
Do you genuinely think the Linux kernel is just throwing Rust in to attract kids who just follow the latest programming fads?
It would help if you clarified. I don't want to try to clarify things and end up putting words into your mouth.
Like just as an example - I can write an allocator and toggle register bits etc. All of that requires unsafe code, raw pointers, etc.
But I can then build on top of that in safe Rust, with all the guarantees that brings. I still have to check that the unsafe allocator or whatever work soundly, but Rust checks the stuff on top of it.
Making sure Rust can work in embedded environments is absolutely a design goal, and choices around this are made explicitly.
e.g?
I am curious
How is "the design of async/await." for embedded programming?
This means RTOS-like projects can use it for tasks, and it works well. Embassy is an example of such a project.
(Yet, I should also point out that you don't have to: at work we keep ours synchronous, for Reasons. Rust lets you do what you want.)
I agree. In general I think Zig makes lot more sense as a C replacement for the kernel.
However, Rust is stable, and Zig isn't. And I don't expect it will be for some years. So at this point Zig is basically unusable for kernel work.
That doesn't mean it can't be added in the future though, and perhaps even overtake Rust. See: https://news.ycombinator.com/item?id=26815950
Like?
What other you guys do in C that isn't just integer oriented programming (using ints for everything)?
Oxide Computer has written an embedded bare-metal OS in Rust, to serve their hardware product: https://hubris.oxide.computer/
I've heard good things about Zig but I've never tried it so I can't compare unfortunately.
I think this might touch on part of the problem. Rust can work on higher abstractions but rust doesn't have to. It is a matter of the code base. I've worked on some that felt very similar in abstraction to C code bases. One in particular was a high performance nginx module. Most of the rust code interfaced with nginx directly through zero cost FFI interfaces to the nginx structs.
I've also worked on extremely meta, high level code that was written by a former Haskell dev.
Like many languages that can serve dual roles, the styles tend to adapt to it. You see very different styles of Java when looking at spring web apps or embedded android systems. Every language you have to pick and chose parts of it that, build a coding standard and need to enforce it. C is exactly the same and also has several roles people pick it for. The style of code of embedded systems, high performance networking systems and codebases that prioritize highly portability tend to be different and use different idioms. I think any of of us who have worked in a C code base that falls in one of those camps can name banned idioms that are normal in the others.
>> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.
My career before Rust was very C heavy. I've work from OS-less embedded systems, to device drivers, to Android porting. I've been doing Rust professionally since 2016. This just isn't my impression. A lot of folks I've known from college and internships who's careers went a similar direction ended up in rust independently. There is value in having a language that let's us get the job done that we used to do with C but with more high quality static code analysis. The business case and use case is very clear.
While I haven't done any bare metal development in Rust, in the areas I have worked I've found compiler, community and crate support to be great. I don't know where you are getting this impression. I think if I only read hackernews articles about Rust I would get the impressions that is Haskell that can work on bare metal. I've noticed articles that tend to get upvoted here tend to be either about 1) how quick adoption is 2) higher levels of abstraction like HKT or GATs 3) WASM 4) writing web services. This article is a great example of 1. 2 is catnip for this crowd. The majority of rust devs never need to know what a GAT is but if you used HN as a metric you'd think it was a daily problem. 4 is something that is easy to write, makes good content, approachable but isn't necessarily Rust's best use case. I'm not saying you've only been reading this site about Rust, just saying that it is easy to get a skewed impression of the language unless you really dive in.
Probably more good examples and possibly more ecosystem.
A first big decision is if you're going no-std or not, and if you end up in the no-std world the ecosystem shrinks substantially. If you're building a kernel that's probably not that much of a problem - the same shrinkage occurs for C/C++ - many such projects bootstrap with nearly zero ecosystem anyway.
The examples side is a bigger problem - I've recently been able to watch some of my more curmudgeonly C friends give it a good dive, and after an initial hump they're fairly happy with the core language. They still have regular issues with the ecosystem when they run into, in their words "web dev crap", which comes up even in the stdlib sometimes - a bugbear a while ago coming up down some error handling code paths. They attempted to send patches and hit nebulous arguments against the correctness target, which were largely born of misunderstandings of posix. This kind of thing can come up anywhere, if you take a dependency on some fancy IO abstraction that happens to be written in C, and you take it somewhere "novel" like a BSD, you might well run into the same. The point here is that _examples_ and _exercise_ of these tasks are the things that are going to shake more of this kind of thing out. At the same time though, it's important to reiterate that if you're on the nostd path, then largely you're on your own, which is equivalent to just gcc bare, and this kind of thing generally doesn't come up.
> Or is this a serious proposal about the future of operating systems and other low level infrastructure code?
This is a serious proposal. The outcome is really strong along key axes of correctness and safety. Those of us who've done it (e.g. Fuchsia, where I was) have been able to observe these benefits relative to history with the same teams using other languages (C, C++). We're professional engineers, these statements aren't coming from a place of craziness. The Android team have been writing about their journey: https://security.googleblog.com/search/label/rust
> Do you just program everything in unsafe mode?
Absolutely not. A good amount of bootstrapping effort has been going in across the ecosystem to make it ever easier to avoid unsafe. To take one slice of examples, there's crates that are designed to help you avoid copies while also avoiding dropping to unsafe - they provide tools for automatic structural analysis of the mapping boundary to make it easier to assert the relevant guarantees. Examples: https://docs.rs/zerocopy/latest/zerocopy/ (came out of Fuchsia), https://github.com/serde-rs/serde/releases/tag/v1.0.0 (serde is commonly used, but has more constraints here), https://rkyv.org/rkyv.html (not sure of prominence, but I hear people talk about it).
These kinds of tools get you a long way toward substantially safer code, without needing to think or audit nearly as much. We know that's important, we have plenty of data that demonstrates how important it is, and lately now, we have data that shows how effective it is too (see the aforementioned Android posts).
> What about runtimes?
They're out there, it depends what level of abstraction you're looking for, runtimes means different things to different people. For embedded there's typically a lot more focus on providing libraries rather than a whole runtime framework, so there are crates for a number of soc types out there which are well used, like https://docs.rs/cortex-m/latest/cortex_m/, or it's sibling minimal runtime crate https://docs.rs/cortex-m-rt/latest/cortex_m_rt/. As you get higher level, if you want to do more of the systems level interaction yourself there are a good number of options to choose from along the lines of reactor systems to get you to functional async executors that will build with nostd.
> It seems to me that Rust isn't even really intended to compete with C for the use cases in which C is dominant in 2023. Every indication is that for "serious Rust in production programming" it's mostly a C++ crowd.
The challenge here is that this is an inverted view - it's a C programmers view of C++ being ported over to Rust, and it distorts how the world looks. That doesn't actually apply to the other sides _intent_. Yes, Rust provides a lot more abstraction capabilities than C does, and in that specific regard it has some coarse similarity to C++. It's definitely _possible_ for someone to way off the deep end and produce obscure abstractions around things that a more reductionist bias is going to hate - and you can totally ignore those things and have a great time with the language. There are some really nice things in there which anyone will enjoy if they come at it with an open mind, things like enums and pattern matching, the rich and efficient iterator library, the crates tooling, configuration macros, and so on. There's a lot to love, and they're not things that C++ did well, and comparing it to C++ discards those considerations spuriously.
> Zig has sort of filled that similar space and seems to take the concerns of C programmers more seriously and the team has an attitude more in line with the C culture than the Rust team does
Zig is interesting in it's own right, and on a very surface level it is more similar to C, though this surface level is really "zigs stdlib has terrible and inconsistent short names similar to libc and posix", which isn't really a good measure of anything in particular. It's perfectly functional along this axis and par for the course at the systems programming level.
The toolchain approach has a lot more of a "hacking for hackers" feel, so when you hit bugs and so on it's very typical for folks to be patching their stdlib locally for a while or building their own toolchain to overcome the problems. I spent a little bit of time there recently with building ghostty on windows and was regularly messing with my toolchain and stdlib to make forward progress. Along these lines it's also much less complete - which is largely a function of being much newer, but you can take a ton of rust projects today, particularly things in the GUI space and build them for every major target platform with nearly zero effort. Zig is very very far off that, and there's going to be a need for a lot better platform level abstractions to get there. Rust did a great job with platform abstractions, which sadly was best documented in an anti-go inflammatory articles, but the point stands and if more generalized stands against zig at times too (https://fasterthanli.me/articles/lies-we-tell-ourselves-to-k... and other similar rants he wrote), though not all points port over.
The LLVM removal kind of move is somewhat enabled by this looser approach, which is also helped by the kinds of users the language has, and the smaller ecosystem. Another way of putting that might be "this is the right time to do this", as doing it later might lead to far more user pain and community noise or negativity. It's great for the world as we need more diversity, but it's also not all roses. At my current work we tried out Zig for hermetic cross-builds something that a lot of people tout as a strength. What we found was that the intrinsics that were written in pure Zig were sufficiently far behind libgcc/compiler-rt that it did not have sufficient performance for our use case - literally the binary couldn't handle our production load. Again, this is the kind of thing that can and likely will improve with time, hell if it was a priority I would have done it, but we had other solutions. Point is it's not as simple as a "this vs that" outcome, these moves have long running implications that may or may not affect a particular target - as an example it didn't really harm ghostty at all.
When you talk about culture each of these ecosystems has it's own dominant culture and a wide set of subcultures. How you choose to integrate with those, if you do at all, is up to you. Some might be more attractive for sure, and some might provide a different risk profile for different use cases as well.
Just off the cuff if I was scaling up a team for a professional project with a long lifecycle, I'd probably lean toward rust right now as it has a good balance of stable evolution and production readiness, without being anywhere near as stagnant as C++ despite much effort to move the needle. If I was in a really hacker mood where I just want to twiddle and mess with stuff, I'm not excessively performance sensitive (beyond the general order of magnitude that native compilation and near zero abstractions gets you), and my team is going to remain small and expert "everyone cracks open the source" folks, then I might pick Zig. These days I don't have many good reasons to pick C anymore. If it's patching a pre-existing thing there's no choice of course, but other than that it's mostly going to be "I'm throwing a 30 minute build onto arduino and don't wanna go off the beaten path for this project" kind of thing.