An RFC that adds support for Rust to the Linux kernel
lkml.org
lkml.org
I am pleased that these are all addressable things. We'll see!
It's kind of a funny space; right now Rust handily gives you "no allocations" (this is where I live) or "infallible + fallible allocations" (this is alloc/std by default) but not "only fallible allocations". This sort of thing is basically filling out the quadrant of options.
Rust in the kernel is not a simple thing, but I think both rust and the kernel will benefit.
> So "Result<T, E>" is basically the way to go, and if the standard Rust library alloc() model is based on "panic!" then that kind of model must simply not be used in the kernel.
The most notable way this can go wrong in pure Rust code is that panic-while-panicking aborts. So you have to be careful that your destructor can never panic.
It's pretty hacky, but it works on no_std.
This RFC is accepted and implemented. See the Josh's answer[2].
[1]: https://github.com/rust-lang/rfcs/blob/master/text/2116-allo... [2]: https://lore.kernel.org/lkml/YHdSATy9am21Tj4Z@localhost/
Also, RFC 2116 assumes that it's sufficient to provide (for instance) Vec::try_reserve, and require callers to always call that before calling anything that might expand the Vec. That wouldn't eliminate the runtime panics. It might be necessary to go a step further, and actually provide fallible versions of individual Vec methods. (Or there may be other potential solutions.)
A bunch of us, anticipating a reaction like Linus' have been arguing that we need `try_` versions of everything and* a way to prevent the other ones from being used. (Cargo features?) I sincerely hope we finally get that.
let v = vec![0; 42];
v.try_reserve(1);
frob(&mut v); // I expect this not to expand the array
v.push(1); // fails, because frob took the space
One option for avoiding this would be to pass a mutable reference to a slice, which allows frob to mutate elements of the vec without allowing it to push.
frob(&v[..])
But it's still non-local. one has to manually account for what allocation will be done, and keep the reservation in sync across refactors. This isn't a race but still scares me.
Someone could write
shared_v.lock().try_reserved();
shared_v.lock().something();
This has been anticipated since panic on OOM was introduced.
Adding `try_` version of everything is a horrible solution.
What you want is for the normal APIs to return Result<R,E> where E=! when panic on OOM and some OOM error otherwise.
The largest irony of all time is that, while returning an error on OOM works well on Windows, it makes little sense on Linux because overcommit is often enable by default. Linux and Linux users are directly responsible for the "panic on OOM" that turns out now prevents liballoc from being used in the Linux kernel.
The obvious fix here is for the kernel to use overcommit internally \s
That's a fun idea! I haven't seen that proposed before, but it would work well once we have the ability to configure std and alloc.
It just makes things slower to compile and less portable.
[1]: https://www.kernel.org/doc/html/v4.15/process/stable-kernel-...
See https://en.wikipedia.org/wiki/C0_and_C1_control_codes and https://en.wikipedia.org/wiki/ASCII
In programmer lingo it can means something similar to the original meaning--rejecting a request for faultiness or incompleteness--or it can mean something more like, "no"--answering in the negative.
I actually loved (I mean really loved) reading the response as it not only was encouraging but also highly respectful and almost glowing with a trained sort of nuance! Just wow!
I agree that I found his messages here to be quite pleasant.
Honestly it's really impressive, IMO, that someone could take a look at how they act and think "I should change this," and then actually do it.
It's just that on occasion he would rant and rave to people and call them retards or whatnot. People get a bit of a skewed perception because only those messages make the news and are famous, but that's not really representative of all the messages.
And when he did rant and rave, it was pretty much always towards established contributors who had screwed up somehow; people who in his opinion ought to know better, and most of the time he did have a good point. I don't approve of his style, but it's not like he would randomly call people idiots. It's not as if you ever ran the risk of being scolded by Linus if you were a new contributor sending a patch or anything.
tl;dr: Linus has always been like this, and was just occasionally an asshole.
Quite frankly I consider most of the news to be worse than useless and actively harmful (and news is different from journalism; journalism is great).
(also, I see now I misspelled constructive as "constrictive" in my previous comment do'h facepalm)
Indeed, and I hate it. It's actually very scary, a creepy change in personality. Like the last scene in "one flew over the cuckoo's nest", where the main character is lobotomized and loses the spark on his eyes. Deeply, deeply troubling to behold. I wonder if the real Linus still exists behind all this or he's gone forever.
As with most human beings Linus is a person with different responses to different things and is generally not a complete lemon.
The persona of Linus you get from following LKML has always been different than the persona you'd get from internet gossip.
Also, I think what's really interesting is that with his feedback, could the language be actually be adapted to the kernel?
This would be fascinating because traditionally the code (the kernel in this case) has to adapt to the foibles of C. This could be the reverse, where not only could the language adapt, it might make the kernel technically better, and easier to read and modify.
Hahaha, preemptive offering compromise on tabs vs spaces warms my soul, good stuff.
Having Rust support directly built into GCC means Rust code in kernel will not limit portability and no additional toolchain will be needed.
This is essentially a reimplementation of rustc. I'm not going to say "don't boil the ocean" but it will likely be a while before this is at feature parity with the main compiler. Even with a full-time developer.
There is a project to use gcc as a backend for rustc, which might be a more tractable option to removing the llvm dependency in the short term:
A gcc rustc backend is the way to go, clearly. That company would have been better off paying for that.
This is only the case if you assume their primary goal is to get a reimplementation as fast as possible. In my understanding that is not their primary goal. Their primary goal is to not require a Rust compiler in the build process. This means that the approach you suggest doesn't fit requirements.
Do you mean their goal is not to require `rustc` in the build process? Because I'm not sure how they're going to compile rust code without a rust compiler.
So then this is just a backend?
And a dishonest reimplementation at that. If you aren't enforcing the promises the language makes, you're delivering a lie.
I say "risk" because any such OS would probably be available under a license more permissive than the GPL. The result would almost certainly be that large parts of the OS would become proprietary (NVIDIA has already tried this with Linux's limited module support).
Of course, moving to Rust is also a risk in this respect, since the LLVM toolchain is BSD-licensed, but hopefully Rust support will be added to GCC in the not-too-distant future.
phew, that was bothering me
EDIT: Ah, found lower in the thread: https://github.com/antoyo/rustc_codegen_gcc. The missing c was enough to make it not show up in searches.
That project could use more contributors, if anyone is interested.
It is a hard requirement for Rust in things other than drivers. But that's a ways off. Walk before you run.
I understand developer thoughts on this are skewed (job security, promotions, fun, etc.), but please. Don't.
http://gunkies.org/wiki/4.3_BSD
Thanks to BSD 4.3 and specially BSD 4.4.
I get that new Rust code will be limited to modules, but here's my concern:
I do work for embedded systems based on the PPC architecture. I'm not sure if Rust supports PPC, but let's just assume we're talking about an architecture that's not supported.
Suppose some company creates a PCIe Ethernet adapter and writes their driver in Rust. If Rust doesn't support PPC, then I won't be able to use that hardware even though there's no physical or technical reason why it wouldn't work. If it was written in C like everything else, then there's a good chance it would just work (assuming they didn't make any endianness assumptions).
In fact, I'd bet that the majority of the drivers written for the kernel are really written for X86 and ARM, but just so happen to work for PPC as well for free. I'd be concerned about getting left in the dust, so to speak.
What's really cool is the humility, flexibility, and creativity, of the community who develop Rust.
What's really really cool is Linus is holding a hoop into the kernel
Of course Rust is a technical improvement over substantially older languages. To claim otherwise is basically to claim that the twin fields of programming language theory and language design have discovered and achieved nothing in the last 40 years, which is an strange thing to believe.
I like the security ideas of Rust, but I'm very much against a compiler including a package manager that points at a particular central repository by default, with all the political problems that that entails, not to mention longevity issues.
Does Rust always panic on allocation failure?
No. See more below.
> Can someone expand on the debate about alloc()?
So the question is the one you asked above, but also, more generally, "is it possible to statically disallow panic on OOM?" and related things.
Rust the language itself, that is, the language + the core library, doesn't know anything about allocations at all. There are none. Concept doesn't exist. So that's fine.
Rust also provides the "alloc" library, which can be layered on top of core. This gives you standard APIs for allocation. They also contain several data structures that use allocation internally. These data structures have APIs that may panic on OOM.
So, Linus saw those things, and naturally asked questions about how required they are. The answer is "not required, but it's work to get rid of them, there's a few options, and we didn't want to do that work until we got a higher-level gut check from you."
Does that all make sense?
I remember there was an issue writing a rust-safe wayland compositor on top of wlroots because wlroot’s ownership model was simply not able to be cast in terms of interfaces that rust was able to prove were safe. [1] Is this not an issue in Linux?
[1] http://way-cooler.org/blog/2019/04/29/rewriting-way-cooler-i...
"The intention is to make these as safe as possible so that modules written in Rust require the smallest amount of `unsafe` code possible."
When writing a kernel, some unsafe somewhere is always required, on some level.
You’re saying some unsafe code is always necessary, why should that be the case? I think the big issue with wlroots was its callback-based API which is common with Linux-internal APIs as well. On a theoretical basis I’m not sure what this means for Rust. Is it simply not possible in the abstract to cast these types of APIs in a form that can be statically proven to be safe? Or is this a deficiency in the current design of Rust? Are all “callback-based” APIs inherently unsafe in Rust? Is it always theoretically possible to recast those APIs in a form that Rust can prove is safe? I would just want to understand exactly why it’s possible to write 100% safe Rust programs in user space and not in Linux kernel space.
These are the types of questions I would ask when evaluating whether or not it’s worth investing in and using Rust for my Linux driver.
Userland Rust depends on these guarantees to provide safe code, something has to implement them.
The callback thing is a red herring; it's not the fundamental issue here. Rust code can use callbacks just fine, in the general case. It also wasn't the fundamental issue with that API either.
But honestly at that point, using Rust doesn’t seem to be very much different than using C in terms of safety guarantees. It still requires a programmer capable of competently ensuring required runtime properties.
> But honestly at that point, using Rust doesn’t seem to be very much different than using C in terms of safety guarantees.
The difference is that it's limited in scope, and auditable. Even in a kernel, if you do it right, unsafe is the vast, vast minority of code. Let's be extremely generous and put it at 10% (Redox, an OS in Rust, had about 2% unsafe last I checked). That means that you still have a much, much significantly smaller space in to look for these bugs.
That’s true but it’s also slightly misleading. Any code that uses the unsafe wrappers technically must be checked and all code that uses that code must checked in turn, ad Infinitum. Misuse of the Unsafe wrapper can occur at any level. For instance, if you misuse a DMA command that corrupts memory, it’s not simply the DMA command wrapper that must be checked, the entire sequence of logic that led to the bad command being executed must also be checked.
If that is the case, then your unsafe wrappers are unsound.
Safe functions need to be impossible to use in an unsafe way or else they should marked as unsafe.
That could take the form of a runtime check that the function's invariants are maintained or a proof that the function's invariants are always maintained.
It's not so much "Rust can't represent this ownership model" as "this ownership model is basically orthogonal to Rust's so you have to put much more effort in to write idiomatic Rust code compared to writing it in C". I would love for someone to come along and prove me wrong with a better wlroots wrapper.
Also, Wayland composites are much, much simpler than the Linux kernel, so that train of thought doesn't necessarily scale out.
> I think that the standard Rust API may simply not be acceptable inside the kernel, if it has similar behavior to the (completely broken) C++ "new" operator.
https://lkml.org/lkml/2021/4/14/1131
I'm not a C++ dev BTW, so this could need more details by someone with background for it.
In the particular case of operator new, it's clear from his cumulative statements that Linus is unaware that the programmer provides global ::new, and can make it do whatever the hell he wants it to do. You can also just forbid it and use placement new everywhere.
Also, I agree with what you said early about RAII solving entire classes of issues that C (and C++ without RAII) has.
See also https://github.com/Rust-for-Linux/linux/issues/2#issuecommen...
Many casual users never notice these bugs, but if you put a Linux box under enough pressure that some kmalloc fails, it will fall to pieces because the error paths are full of bugs and little-exercised. C has probably the worst error-path control flow of any of the high-level languages.
The alternatives to goto cleanup; have drawbacks too - like being less readable (cleanup code is before the main function body or destructors do things implicitly).
Basically, we think (and we hope Linus agrees) that Rust has the relevant benefits of C++ without the downsides that have so far kept C++ from being adopted. (It has other great features that C++ doesn't have like compiler-enforced memory safety, too, but I agree with you that RAII is a pretty important and obvious win for the kernel just by itself.)
(And Linus noticed one of the big parts where Rust doesn't live up to this - the idiomatic thing for memory allocation failure is to unwind - and that's a solvable problem.)
If it `can be idiomatic` it means it's not currently idiomatic, and C++ isn't really heading in that direction. There's also a vast suite of C++ programmers out there to who it's very much not idiomatic and downright foreign. So if you're going to fight against how everyone else writes the language, it's best not to use the language.
Disabling exceptions isn’t “fighting against how everyone else writes the language” - it’s pretty common and well-supported.
> "Please note that the Rust support is intended to enable writing drivers and similar "leaf" modules in Rust, at least for the foreseeable future. In particular, we do not intend to rewrite the kernel core nor the major kernel subsystems (e.g. `kernel/`, `mm/`, `sched/`...). Instead, the Rust support is built on top of those."
Half a century ago this living legend wrote an operating system. Can you imagine how much experience this single person holds today? It must be hard for him not to roll his eyes when talking to juniors.
- PDP's Unixen
- I am not sure, but, maybe, QNX
It depends, but it's certainly not the first POSIX implementation in some high level language other than C. For example:
* Real-Time Executive for Multiprocessor Systems (RTEMS) is a real-time operating system (RTOS) designed for embedded systems. While it's often compiled down to as slim-as-possible it does support POSIX, and it's written in Ada. It's been used in a lot of projects, but because it's really an RTOS, RTEMS is typically embedded within dedicated systems and not something you'd normally interact with directly. https://en.wikipedia.org/wiki/RTEMS
* The BiiN system had an operating system written in Ada, and I think it implemented most of POSIX. However, the BiiN hardware never sold well, so it disappeared. I can't even find much about it on the web.
* BeOS implemented a lot of POSIX, and it was mostly C++. Haiku re-implements much of BeOS and is also written mostly in C++.
But I think that information must be old, the current info seems to just say C.
(The Rust brigade is out in force I see :))
Honestly, it is a really weird experience to have around a programming language, and to me is offputting for something that otherwise could have some very interesting merits.
I expect people wont agree with me, and I'm not arguing against the technical merits of the language, I'm just saying that it stands out because of the ideological following it has, and that is cause for concern if ideology is a big reason people try and push to integrate it in something as important as linux.
Rust was more or less specifically designed to appeal to people like us. Throw a rope to a drowning person and they'll grab onto it hard.
So Rust is not a solution.
But it will make those codebases a lot easier to deal with because the Rust compiler enforces correctness in a myriad of ways that other languages don't. I would much, much rather deal with a legacy Rust codebase than a legacy C++ one.
Legacy codebases are smelly not because of a lack of tools for enforcing correctness. They're smelly because a) programming is hard, and b) a boatload of things are more important than correctness, in the real world.
> I would much, much rather deal with a legacy Rust codebase than a legacy C++
Obviously, because right now legacy Rust codebases are only 5 years old, while legacy C++ codebases are 30+ years old.
But in 20 years it will make no difference. The Rust of 2031 that will need to accommodate 30 years of legacy backwards compatibility will be no prettier than C++ today.
There are? Like what?
> But in 20 years it will make no difference. The Rust of 2031 that will need to accommodate 30 years of legacy backwards compatibility will be no prettier than C++ today.
I don't really believe this. There are 20 year old Java codebases, and yes they can be a mess and hell to work with, but they're not nearl as bad as C++ ones. And Rust is stricter than Java.
Exceptions, for example.
> There are 20 year old Java codebases, and yes they can be a mess and hell to work with, but they're not nearl as bad as C++ ones.
That's because Java doesn't attempt to solve difficult problems.
Exceptions are better than magical return values to indicate errors, that's true, but Rust Results achieve the same thing without the downsides of exceptions.
> That's because Java doesn't attempt to solve difficult problems.
Java isn't my favourite language, but that's just silly. GraalVM will do as a counterexample.
Sometimes they are, and sometimes they are not.
> They introduce hidden control flow paths that developers forget about and fail to handle correctly.
The problem that exceptions solve involve control flow paths that aren't supposed to be handled. Exceptions are not for handling recoverable errors, they are for graceful aborts in a complex, layered and modular program. (E.g., any multi-threaded server, for example.)
For control flow paths that are considered irrecoverable (ie. "this can never happen" branches), Rust has panic!(), which defaults to unwinding the stack, calling Drop implementations (destructors) along the way.
panic!() unwinding only kills the thread it occurs in and Rust's thread-related APIs are designed around preventing data that's been left in an inconsistent state from being observable in other threads without explicitly acknowledging that you're dealing with something like a mutex that's set its "poisoned" flag.
For control flow paths that are considered recoverable, Result<T, E> is basically a way to get checked exceptions which work naturally with higher order functions and have a more concise "call the defined conversion to the specified error return type if necessary, and re-throw" syntax.
If you implement the From/Into interface to define how to convert the error type you received into the error type you're returning, the ? operator will do an "unwrap the Ok value or convert and do an early return of the Err value" in a single character.
A lot of people use the thiserror crate to define their custom error types, which has helpers to makeimplementing the From/Into interface trivial... possibly as trivial as annotating an enum variant with #[from], depending on what you want out of it.
Also, this is from 2005 and chose a "considered harmful" title, but this article makes some good points:
Rust code accumulates some cruft over time, but Rust's type system and safety guarantees put a floor on how crappy the code can be. The first Rust code I ever wrote is still part of my project five years later; it's less than perfect, but it's free of undefined behavior now just as it was then. The data structure parsers I wrote four years ago look a bit ugly to me now, but are free of exploitable security bugs, and always were. Etc.
What does that even mean in the absence of a formal standard and competing compiler implementations? (Hint: nothing.)
If you refuse to define anything then you automatically make the "undefinedness" problem go away. (But not the pain it causes.)
> Rust's type system and safety guarantees put a floor on how crappy the code can be
No. The floor on crappiness is defined by the problem domain. Powerful features allow for powerful takes on crappiness.
Unless you want Rust to stay a teaching language forever, you must necessarily introduce abusable features. (See Python for a real-time slow-motion elaboration of this train wreck if you don't believe me.)
It means that when the language designers are asked "what is the behavior of this code (that compiles and doesn't use 'unsafe')?" they never throw up their hands and say "it's undefined behaviour, you must not write that code" (as the C++ definition often says).
They may say "oops, we're not sure what it should do, we need to clarify the language definition and write some tests to ensure the compiler does that". (This happens in C++ too.)
> Powerful features allow for powerful takes on crappiness.
I don't know what this means.
> Unless you want Rust to stay a teaching language forever
A lot of companies big and small are using Rust in production so this is not a compelling premise.
In absence of a standard effectively every language construct is 'undefined behavior'.
Again, look at Python for a vivid example of this. (We were discussing just this is a neighboring thread: https://news.ycombinator.com/item?id=26826158)
I have zero reason to believe that Rust won't meet Python's fate.
The Rust people don't have a standard and don't understand why they need one; in fact, their lack of standards is somehow touted as a benefit. Apparently, people think that if there is no standard for "defined" and "undefined" behavior that everything is "defined" by default.
(They're absolutely wrong, of course; it's actually the opposite.)
For more see: https://blog.regehr.org/archives/213 and https://raphlinus.github.io/programming/rust/2018/08/17/unde... .
> In absence of a standard effectively every language construct is 'implementation defined behaviour'.
But the parent said:
> In absence of a standard effectively every language construct is 'undefined behavior'.
Undefined behaviour != implementation defined behaviour. Both of these things exist in C and are separate. You can rely on implementation defined behaviour giving some consistent result on a given, compiler and hardware. You cannot rely on the behaviour of your program if you hit undefined behaviour.
That's clear enough for programmers to rely on. It is nonsense to argue that this is equivalent to C's "the compiler may do anything it wants" just because the "Rust Book" is not called the "Rust Standard" or not "formal" enough.
Undefined behaviour is stuff that you compiler's optimizers have been promised can never occur, so they are allowed to transform your code based on that assumption.
Here's a post I made on Reddit in 2019 with a list of resources on what undefined behaviour is:
https://www.reddit.com/r/rust/comments/dpxswt/rust_2020_lets...
...and here's the link to an example of what invoking undefined behaviour can do in C++:
https://gcc.godbolt.org/z/0ubnbS
...and here's an explanation of why it does what it does:
https://kristerw.blogspot.com/2017/09/why-undefined-behavior...
(TL;DR: It injects a call to EraseAll because calling Do while it's still null is undefined behaviour, and Do is a static, so the optimizer determines that the only possible answer within the rules it was given is that code outside that compilation unit will have called NeverCalled to set Do = EraseAll before invoking main().)
I think Bryan Cantrill's talk from 2018 encapsulates things pretty well:
https://www.youtube.com/watch?v=LjFM8vw3pbU
Bryan is a hardcore C guy, and can write safe and reliable C code. But he has reached the limits of what C can offer in terms of composability and abstraction.
Like Linus, he wants and needs the kind of high performance and dependable behavior for systems programming that isn't possible with a garbage-collected language.
http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-...
http://dtrace.org/blogs/bmc/2020/10/11/rust-after-the-honeym...
FWIW, I think the memory safety-first ideology of Rust is worth spreading.
Popular GC languages like Java or Go are not able to prevent a class of data race bugs that Rust prevents at compile time. Languages like Haskell of course handle it much better, but are for a multitude of reasons not popularly used in production.
And as the sibling comment points out, GC languages are not applicable everywhere. This discussion is in the context of the first not-C language being added to the Linux kernel. Can you imagine a GC language being similarly considered?
So yes, the borrow checker is amazing. That feeling of satisfaction and confidence when the program finally compiles after a round of serious coding or refactoring is unparalleled among the mainstream languages I've tried so far.
Can you give me an example?
(https://cliffle.com/blog/rust-typestate/ for more on what the typestate pattern is but the TL;DR: is "verifying correct traversal of a state machine at compile time". For example, making it a compile-time error to try to set an HTTP header after you've started streaming the body.)
Before Rust, I'd spent 15 years with Python as my preferred language, and I had experience with TypeScript, CoffeeScript, JavaScript, PHP, Bourne Shell, and the "used very little or very long ago, so I forgot" kind of experience with Lua, XSLT, Perl, C, C++, Visual Basic, QBasic, and DOS/Windows Batch Files.
For me, it's purely about Rust reducing the amount of time I have to spend writing Python unit tests to get the level of confidence I want in my codebases without having to put up with the quirks of a pure functional language like Haskell that doesn't put high value on long-term API stability. (And yes, I do use MyPy and type annotations heavily.)
It's also a big boost to the value proposition that, with no garbage collector of its own, it's easy to integrate Rust modules into my PyQt GUIs or Django+Celery web apps using rust-cpython or PyO3. Trying to hand off objects between multiple garbage collectors in the same address space without something like "serialize the whole thing, hand over ownership of the bag of bytes, then deserialize it" is a recipe for pain.
Also, I'd suggest reading the "What makes Rust work" part of https://boats.gitlab.io/blog/post/notes-on-a-smaller-rust/ for an explanation of why garbage collection doesn't solve all the problems the borrow checker does.
Testing is orthogonal to typing, and I believe that those who claim that the latter can make up for the former simply do not understand how to test software. Correct typing is the foundation for a correct program, but the lack of type errors absolutely do not indicate a lack of bugs. Logic errors that are not caused by type errors are far more common, far more dangerous, and are not caught by type checkers.
> It's also a big boost to the value proposition that, with no garbage collector of its own, it's easy to integrate Rust modules into my PyQt GUIs or Django+Celery web apps using rust-cpython or PyO3. Trying to hand off objects between multiple garbage collectors in the same address space without something like "serialize the whole thing, hand over ownership of the bag of bytes, then deserialize it" is a recipe for pain.
I've written a binding for CPython in Factor. There is some plumbing work for sure, but no, it is not that complicated. You manage objects created by the foreign memory manager using special tokens. The difficult part is callbacks; host language calling CPython which calls back to the host language. rust-cpython's documentation doesn't mention callbacks at all so I guess it doesn't support it.
Your mileage may vary but, on average, I find the time lost to bookkeeping is dwarfed by the time saved on not having to use testing to verify invariants which are upheld by the type system.
Granted, my coding style in Python was already quite similar to what Rust lends itself well to.
...plus, I just find it more relaxing to code in a language where I can delegate more of that to the compiler's type-checker.
> rust-cpython's documentation doesn't mention callbacks at all so I guess it doesn't support it.
First, rust-cpython is the older, less advanced binding. PyO3 forked off from it to explore more advanced API designs that, at the time, required API-unstable features only available in the nightly Rust but PyO3 now runs on stable Rust.
Second, I haven't tried closures with rust-cpython yet but, for functions, the py_fn! macro is how you wrap a Rust function into something you can inject into a namespace in the Python runtime.
As for calling Python functions from Rust in rust-cpython, you call the run or eval methods on the object which indicates that you've taken the GIL.
I suppose, if closures aren't supported, you could work around it by using methods on a py_class!-wrapped Rust object instead.
Third, I avoid C++ these days and limit my modern use of C to retrocomputing projects. It's just not worth the mental effort to write C in my projects, so I stick to binding things in ways where I get a memory-safe, type-safe binding and someone else can be responsible for stressing out over the correctness of the binding generator.
Memory management is orthogonal to type checking. I agree with you that static type checking has advantages but that is not what I meant by bookkeeping. The bookkeeping is Rust's borrow checker, explicit memory management in languages like C, or even weak references in some languages. This is time lost which wouldn't have been lost had the programmer choosen to use a language with tracing garbage collection.
In fact, you can think of automatic garbage collection as the language upholding certain invariants about memory that the programmer otherwise would be forced to ensure themself.
Thus a Java programmer has to think less about memory handling than a Rust programmer. Less to think about means less bugs. You may argue that it is worth it because Rust is faster than Java. I have not seen benchmarks that proves that but, even so, I can count on one hand the number of times Java's gc has caused significant problems even in performance sensitive code.
> Second, I haven't tried closures with rust-cpython yet but, for functions, the py_fn! macro is how you wrap a Rust function into something you can inject into a namespace in the Python runtime.
Right, it's an engineering problem so I'm sure it's solvable in Rust. After all, it was solvable in Factor which is a gc:ed language. My point was that I doubt Rust's lack of gc makes it easier to embed CPython.
Java? Pain in the ass with all those non-inferred type signatures and comparatively poor POSIX API integration. Also, I've yet to encounter a Java GUI, AWT, Swing, SWT, or otherwise, that wasn't buggy and sluggish under X11, and the startup time is, in my experience, even worse than the Python-based CLI utilities I'm often migrating to Rust to get improved startup time. (Also, checked exceptions don't compose well with higher-order functions. Monadic error handling does.)
C#? Not as bad as Java, but still doesn't have a value proposition that would make me switch away from Python.
C or C++? No. I use Rust for getting a stronger type system on top of something that's still memory-safe, not its performance. (Aside from the aforementioned "If Rust is offering, sure I'll take faster startup for my CLI tools AND monadic error handling AND explicit nullability".)
Vala? I forgot to mention that I played around with it and it's got all the ills you'd expect of a niche compile-to-C language.
TypeScript on Node.js? Worse than Python+MyPy in pretty much every way that matters to me except for having native sum types.
Haskell? Sorry. You'd have to pay me to code in a pure functional language, even without my dislike for its syntax and the ecosystem's philosophy of not being afraid to break APIs to advance the state of the art.
etc. etc. etc.
Still, as I've said before, I tended to already use Python for stuff that's well-suited to Rust. For example, so far, I've yet to need an Rc or Arc aside from the ones actix-web embeds in its data containers.
...and if I did, I certainly would want something along the lines of the borrow checker double-checking that I'm not introducing data races in threaded code, and a system akin to Rust's for compiler safety checks on non-memory resources managed through RAII.
(See also https://boats.gitlab.io/blog/post/notes-on-a-smaller-rust/ )
...plus, it's shamefully rare to find things that match Serde for declarative serialization and deserialization, let alone exceed it.
That said, I'm a "right tool for the job" kind of guy and I still use Python for anything that involves SQL for want of a library like Django ORM or SQLAlchemy+Alembic which abstracts over the difference between SQLite and PostgreSQL DDL, doesn't use the database or a raw SQL file as the authoritative source of truth for the schema, and has a migrations system which auto-generates draft migrations by diffing the authoritative schema against the database.
https://thenewstack.io/microsoft-rust-is-the-industrys-best-...
https://medium.com/the-innovation/how-microsoft-is-adopting-...
https://www.infoworld.com/article/3605551/microsoft-forms-ru...
https://docs.microsoft.com/en-us/windows/dev-environment/rus...
The stakes are low and nobody cares if your little team writes code targeting a Brainfuck compiler. (Yes, this is a real thing; there's a Brainfuck compiler developed somewhere in the vast guts of Google. No, I don't have a link to share with you, sorry.)
Not in core parts of Windows they don't. It's C++, perhaps some C, and now Rust. It's a pretty big endorsement.
It's not really an endorsement because the bar to getting into a Google or Microsoft or Facebook codebase is as low as it gets in this industry.
Even Rust isn't all that easy. That it cleared the hurdle at all on so many teams already (and Windows especially!) is extremely impressive for a piece of tech so new.
This is the lowest of the low bars w.r.t. technology choice.
Compare, for example, to the vetting and QA the Linux kernel does.
With gccrs, the portability issues of rustc will go away and no extra toolchain will be needed for the Rust code in the kernel either.
> LLVM does not target all of the architectures that Linux supports and just because a target is supported in LLVM does not mean that the kernel will build or work without any issues.
Obviously there is still significantly more diversity at the low end of the embedded world, but you wouldn’t run Linux on e.g. an MSP430, even if you somehow had a compiler that would support that.
Not to mention plain ol' inertia. The vast majority of people compiling the kernel are using GCC. Switching to a different toolchain or maintaining two toolchains (assuming there are no subtle ABI issues between a GCC kernel and LLVM driver) isn't necessarily trivial, and requiring that just to build a single driver might rub some people the wrong way. That's not a _problem_ with LLVM, but it is an inconvenience.
This is why the idea is to start using Rust in the kernel for drivers.
Is GCC really any smaller?
Perl 6 or bust!
> At the present time, we require certain nightly features. That is, features that are not available in the stable compiler. Nevertheless, we aim to remove this restriction within a year by either `rustc` landing the features in stable or removing our usage of them otherwise.
Using unstable features in something like the Linux kernel just seems like a bad idea. Why not wait until Rust is a little more matured, and doesn't need to use nightly features?
One is going to be stable in Rust 1.52.0, the next release of Rust. This will happen before the next kernel release, so it barely counts. It's also a documentation tooling feature.
One is related to symbol mangling, and so in theory could be worked around I'd imagine, but is landing soon enough they don't think it's worth it.
The final three are related to each other; two of them look like they're being stable pretty soon, and the third, while it's less clear on timeline, is one pretty small thing.
So, the answer is basically that the exposure here is pretty small. How much that matters is a social question as much as a technical one. While "unstable" is a binary designation, within it is a spectrum of unstability. "This feature has no path to stabilization" to "this will be stable in the next Rust" and all things in between. Maybe allowing some unstable things that are closer to the latter is acceptable, maybe not.
I know that many think that Rust is the be-all and end-all of programming languages, so much so that there's this effort to put Rust into Linux despite the requirement of nightly Rust versions and Rust still having inadequate support for various architectures. But the hype regarding Rust always seemed misguided to me, Rust is a language built upon the (interesting) idea of ensuring memory safety without a garbage collector, which is cool, but I get the feeling that most Rust fans forget that the actual goal is correctness, not memory safety.
On another note, using C++ in the Linux kernel could give a similar kind of improvement, but with much lesser costs than with Rust.
Unclear if this is a hard requirement, even if it is, it's stated to be feasible on the timescale of "less than a year".
> support for all relevant architectures)
This is already fulfilled; this is for drivers only, which are inherently platform-specific, and so only drivers that are usable on the supported platforms would be written.
> there could well be a dozen different languages available that improve greatly on Rust
As mentioned above, I doubt that this is true within a year, but beyond that, any new language is also going to have all of these same growing pains, but be a decade behind. When they're ready, they should be considered, but "don't do a good thing now because maybe, some unknown amazing thing might appear" isn't usually a good way to make decisions.
> As mentioned above, I doubt that this is true within a year, but beyond that, any new language is also going to have all of these same growing pains, but be a decade behind. When they're ready, they should be considered, but "don't do a good thing now because maybe, some unknown amazing thing might appear" isn't usually a good way to make decisions.
This is true, but Linux is a long-term project, we don't know what the future will bring, but I'd be surprised if it wasn't still a mainstream kernel in 20 or even 40 years.
It seems to me it would be wise to look at the future as well, and perhaps things are the wrong way around; rather than asking "what existing languages could work for the kernel?" the question should be "what would the ideal language for the kernel look like?"
It always seemed to me that Rust is a bit of a strange choice for the Linux kernel; I can see why it has some appeal, but it's also a fairly large and complicated language.
How "complex" or "minimal" a language should be is an old debate that we've all done at least a few times, so don't want to repeat it here; for my part, I feel comfterable with both approaches; I'm just as happy programming in Ruby as in Go for example (even though they're polar opposites in many ways), but both approaches come with their own set of advantages and downsides.
I'm not so sure that a large and complex language is really the best fit for this particular purpose. For example writing a C compiler for a new architecture is comparatively easy whereas porting Rust to something new will be much harder.
Given Linux's long-term prospects it might be wise to invest in an "ideal" language rather than a "it'll work" language.
Then again, I don't work on Linux so what do I know :-)
We're really trying two things here - getting Linux to use some Rust, and getting Linux to use some other language, with an independent compiler. That second part is pretty significant by itself (we've had discussions about build systems, compiler ABI bugs, etc.), and even if we were just trying to add C Except Different, it would be a major project on its own.
Once we've figured out how to make them interoperate and what the ground rules are (architecture compatibility, where new development goes, whether the new language can be mandatory, release timeframes, etc.), a third language will be on much better-explored territory.
(I do actually think that Rust is pretty close to ideal for the Linux kernel as it currently exists, though! As I argue in https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf / https://www.youtube.com/watch?v=RyY01fRyGhM , a lot of Rust constructs happen to match idioms and abstractions that the kernel uses with C. But yes, there's a lot of research on both good kernels and good programming languages yet to be done.)
It's good this is finally been considered; there have been projects like this for decades but they never made much headway. I remember seeing this presentation about a memory safe driver framework over 10 years ago, and while it was functional (according to the authors) it never seemed to see much implementation (IIRC it generated C code, I can't recall the name of the project). There's also stuff like Cyclone[1], and I never understood why that never got any traction as it embodies more or less the same ideas as Rust (the homepage even recommends Rust now), and there's of course D, although the lack of adoption there can probably be explained by the licensing (D can run without a GC).
It's that Rust combined a whole lot of attractive features in a single package, of which, memory safety without GC is only one.
No one thinks Rust is the end-all-be-all. They simply think that Rust is better than C, quite easily at that. And Rust does focus on correctness... If people wanted memory safety, almost any not-C/C++ language would be good enough. Rust's correctness story is wonderful, which is why I myself don't focus on the garbage-collector thing when talking about Rust.
And C++ has been thoroughly rebuked by Linus, and his statements have only gotten stronger with time.
Rust seems to be much more opinionated on how to do certain tasks, which might make it an easier to use for the kernel because there will be one way to do certain tasks. That's my read on things at least.
This comment getting downvoted is everything wrong with HN.