I want to be able to actually compile my software if I wish so. This is becoming increasingly difficult. Rust is adding to the problem.
https://seclists.org/oss-sec/2022/q4/23
This is a catastrophic bug, that (after some work on developing an actual RCE) lets anybody within wifi range to get root on your laptop (or phone, or access point). And all it took is this one line:
https://git.kernel.org/pub/scm/linux/kernel/git/wireless/wir...
We've all been repeating the "1000 eyes - all bugs are shallow" mantra for far too long. This one was in the mainline for more than 3 years, and nobody noticed. How many more are lurking there?
memcpy(pos, mbssid + cpy_len, ((ie + ielen) - (mbssid + cpy_len)));You can turn on runtime overflow checking with the Rust compiler, but you can do that with gcc compiling C too.
I guess the answer is that you wouldn't do the memcpy like that in idiomatic Rust - you'd use some higher level construct that gives the compiler more chance to catch your errors. Could anyone comment on how this works in a case like this?
edit: Can't reply to repliers. Defined overflow doesn't help here. A checked_add() function could be used in the Linux kernel in C, just as easily as in Rust. Forcing checked_add() to be used in the Rust compiler would help, but would also have performance and readability impact, which is presumably why it is still being debated.
In Rust the u8 type wraps in default release builds, but panics on overflow in debug builds, but this is kernel code so it will definitely be built in release mode with wrapping.
Yes, that's the trick: it will be caught not in the u8 overflow, but either in the memcpy equivalent (which is "dst.copy_from_slice(src)"), or in the slice manipulation before it. What happens is that slices in Rust are represented by a "fat pointer", a pair of the starting address and the length, and both the "copy_from_slice" method and the index/index_mut operator check the bounds before doing the operation.
(You could do things using "unsafe" and raw pointers, which have only the starting address without the length, but in idiomatic Rust you'd use slices most of the time.)
DoS is the only publicly known exploit, right now. There is understanding that RCE is possible with additional specialist work.
Local packet injection is an implementation detail of the first public PoC. There are other implementations of the same exploit, that deliver packets over the air instead. Including one that runs on ESP32, and attacks unmodified Linux nearby:
Just leave a small board, a battery, and a solar panel on a roof or tree near your office. If RCE is possible with this (people think it is), this would be very valuable.
On a tree near my office, you'd gain access to our network, Microsoft's and a European central bank. Not bad at all.
- The Linux kernel and issuance .so files lack a "view source" button on compiled binaries. And even checking out the matching source, building a replacement binary, and diffing your local changes from the matching source is an arduous progress to setup per program/library from a tarball/Git tag, wait for the computer to finish, install dependency .so files globally, ensure symbols are present, ensure you can breakpoint static functions...
- Dynamic dispatch and generic code might help maintainers and code extensibility but (in my experience) definitely impede external eyeballs from understanding code.
To a smaller extent, netbsd has pkgsrc, and most BSD friends have something like that too.
While I agree that bug is serious, that "some" is doing pretty heavy lifting here. Is there an RCE for this bug?
* Overflow with the default operators is not UB in Rust, it will either panic or two's compliment wrap, depending on various things (including things you can set to choose this global behavior). This already prevents various issues.
* You can also explicitly choose to do various operations with whatever overflow semantics you want, as you mention with saturated_add and friends.
* Because indexing is bounds checked, where in some languages an overflowed integer would lead to incorrect indexing and therefore possible memory problems, you'll either get a panic or a logic bug, not a memory bug.
* If your integer isn't being used for indexing, you'll end up with some sort of error, but again, at worst a logical error, not a memory error.
We should aspire to make it as fast but in the short term a slower compiler in exchange for less CVEs and random buffer overrun crashes seems like a reasonable trade off to me.
Distributions such as Fedora offer build infrastructure[1] that you can use to compile packages to use in your system for testing if you feel your local hardware isn't powerful enough.
(Same reason why C with typedefs is slower to compile than plain C, why C++ is slower, etc)
(that and cargo dependencies, etc - also C compilers have some +30yrs of optimizations)
A full Rust kernel might be neat. A compromised Rust driver inside of a C kernel is never going to be the default choice.
The actual driver code is pretty readable, see https://lwn.net/Articles/863459/.
May I interest you in Redox? https://www.redox-os.org/
Obviously very different than Linux.
There's been multiple passes at making the Rust compiler faster. It'll happen. There's a GCC-based Rust implementation in the works too.
> The main difference between Rust and other languages is that it does some more (but not all [2]) safety checks at compile time rather than at runtime. It also allows you to avoid GC, but does not provide you any memory safety over GC. Rust's borrow checker allows you to statically prove that references are live [3]; a GC simply avoids deallocating any memory that has a live reference to it (on the other hand, a GC can ensure that references remain live even where this is hard or impossible to prove statically). The end result is the same with respect to memory safety (the reason some people want to avoid GC is for performance reasons, not memory safety).
> The main difference between ARC and Nim GCs is that ARC is fully deterministic - the compiler automatically injects destructors when it deems that some variable (a string, sequence, reference, or something else) is no longer needed. In this sense, it’s similar to C++ with its destructors (RAII). To illustrate, we can use Nim’s expandArc introspection (will be available in Nim 1.4).
> This shows one of the main ARC features: scope-based memory management. A scope is a separate region of code in the program. Scope-based MM means that the compiler will automatically insert destructor calls for any variables which need a destructor after the scope ends. Many Nim constructs introduce new scopes: procs, funcs, converters, methods, block statements and expressions, for and while loops, etc.
> ARC also has so-called hooks - special procedures that can be defined for types to override the default compiler behaviour when destroying/moving/copying the variable. These are particularly useful when you want to make custom semantics for your types, deal with low-level operations involving pointers, or do FFI.
You can always make it your own personal default with older versions (e.g. nim-1.6), by editing your $HOME/.config/nim/nim.cfg to say so or doing similar on per project/file basis.
Rust also uses the same mechanisms to get compile time thread safety with full memory sharing between threads. Does any other language that doesn't have a global lock, throwing away most of the advantage, have that? There are actual new and interesting advantages to the compile time ownership model.
A minor advantage on the age of microservices and OS IPC to shared external resources.
Sendable doesn't apply when those threads are accessing shared external resources.
If you want fully lock-free it might be possible to prove that with extensions like Liquid Haskell, via Linear types, and is definitely easier to prove with a theorem prover than it is, generally, for C++ code. Not sure about Rust though I realize quite a lot of its moving parts have already been formalized which is super cool.
There are reasons for using Rust but it's not the only game in town. And GC doesn't automatically mean pessimistic performance.
Rust is a huge, huge win for that reason. It allows for code written in a Lisp/OCaml/Nim style that's memory safe without any additional overhead.
Rust compilation is not intrinsically slow; better IR and more incremental compilation can and will improve it. Incremental compilation (only changed files, not crates) is 7 years in making though.
[1]: https://prev.rust-lang.org/en-US/faq.html#why-is-rustc-slow
Other reasons:
- suboptimal LLVM IR code, as you mentioned, and other tech debt
- the preferred strategy of monomorphising generics is fast at runtime, but slow to compile, using trait objects is faster
- the complex type system of course also plays a part
- LLVM itself is of course not optimized for Rust
I'm not sure it was a "bad decision" if it was something that can be addressed after the fact and allowed them to ship working software more quickly.
The vast majority of the popular projects I am aware of have this kind of technical debt, likely due to survivorship bias. Project teams that refuse to take on technical debt are rarely successful enough to become popular.
Maybe we are giving Rust developers computers that are too fast. ;-)
OTOH, it's always wise to design for the future, so it also makes sense to give Rust developers beefy server grade hardware so they can play with SIMD pipelines and all those extra cores and threads, because that's what workstations will be a couple years from now.
It is only a matter of time, until GCC and clang finally catch up.
Shouldn't this be enough? Or does it require explicitly including the precompiled header whereas VC++ has magic to make that transparent?
Ada, Delphi, OCaml, C#/F# (.NET Native / Native AOT), D, Nim,...
For everything else regarding concurrent access to shared data out of process, they are on the same foot.
Ada didn't took off, because of several reasons, price of compilers, on UNIX it was an additional SKU on top of the respective developers SDK that already offered C and C++ in the box, 1980's mainstream hardware wasn't able to cope with it, most OS vendors outside UNIX decide to migrate from their toolchains into C and C++, so again additional money on top of the OS SDK.
Early Ada compilers had expensive licensing and required expensive hardware to run.
By the time GNAT was added to GCC, C++ had already taken over most of the spaces that were not Ada exclusive (meaning safety critical / defense / aerospace niches where either Ada was once mandated or has thrived in despite the original high costs).
I believe the point the author is making is that other languages provide better safety than C and have faster compile times than Rust, therefore Rust should be able to improve its compile times.
Ada with SPARK 2014 formal proofs, for your other part of the remark.
What you may want to try to avoid is complex and non-deterministic GC, which makes it harder to reason about.
Except most of C# ecosystem relies on classes and GC. A lot of these problems are caused by overuse of GC-ed classes, and their ease of use.
C and C++ aren't immune to heap abuse as well.
You do on C# just like on them, think about data structures, and avoid heap during render loop
They still ran into limitations where basically "This would be a lot easier if we didn't have GC".
tr;dl — I’d really prefer the option to determine when I’d like GC, as opposed to dodging the collector to avoid performance hits, hot code can default to manual. Probably not a realistic ask, but it could work.
Fair enough. If SS14 used D maybe they wouldn't have these problems.
D and early Rust (pre 0.2, like alpha alpha) had that. Problem is you split your community in two. You get a version of "What GC-color is your function/lib?".
It's a tradeoff for some domains - allow no GC bypass you're going to run into nigh insurmountable performance cliff.
Allow GC as opt-in and you run into issue of splitting your APIs in two.
"This would be a lot easier if we didn't have malloc()/free()", basically.
In any case Sony, Nintendo and Microsoft have been doing graphics stuff with C# for their platforms, although they could have kept being pure C++.
Also we shouldn't silo ourselves into only C# as discussion point, Go, D, Nim, Swift, Eiffel are also possible examples.
Well, not really. It's just some libraries like YamlDotNet copy waaaay more than needed and it shows in serialization. They mostly minimized calling YamlDotNet, but true solution would be a zero copy parser.
Other issue was HashSet operation like Clear had huge impact. Think they replaced those with arrays.
Third issue was something about flecs and archetype ECS. I don't know if it was a jest, but they mentioned changing GC layout or rewriting GC.
Points is, they now face a steep performance cliff. The only way out of it is through sheer effort.
So no, it's not anti-GC religion. Some domains and GC really badly mix.
- the Linux project is very likely to stick to simple, fast elements of Rust (based on the excellent approach of the Linux/Rust devs thus far)
- the more Rust is used, the more work will be done to improve its performance
- you can still build a kernel on a low-powered device... i've built kernels that took > 12 hours on, for example, PA-RISC boxes that were once regarded as beefy :-)
- most people don't (and shouldn't) compile their kernel, and by most I mean more than 99%
All I really want is to be able to compile my stuff without waiting overnight (or more).
Not only I woundn't say that most people shouldn't compile their kernel, I would say that most linux users* should do it at least once, so they can understand the power they have compared to closed-source operaring systems.
*with linux users I mean users that use linux as their main operating system, not people that do ssh once in a while or rarely boots their linux partition
a modular kernel with a custom inird (generated by the distro) is small enough for most.
so if you are into adventures or you are in the business of kernel development yes roll your own. anybody else is better served standing on the shoulders of a maintained binary distribution.
- everyone has better things to do than compile software they didn't write
- a good distro has probably tested it on a bunch of hardware, and hopefully signed it (or at least the packaging), so you know it was securely acquired and built
- you won't learn much at all about the Linux kernel by compiling it... you may learn a tiny introductory about about it by configuring it, but that's still not very much at all, really (it may seem like a lot when you don't know how to measure what you're learning)
- what you should learn from configuring and compiling a Linux kernel is that you don't ever want to be in a situation where you have to do it again (without a really spectacular reason, or being paid)
- if you're compiling a kernel because your boot partition is small... make it bigger, or don't have one at all. come on.
With every year, abandoning OSS and looking for non-computing hobbies gets more attractive.
So you can of course compile your Rust compiler. If you are used to compile clang or gcc, it's not that much of a hassle. And the benefits have already been shown. If you only want to compile Rust code, and not develop it, mrustc might also be a good choice for you (it doesn't implement borrowck, just what's needed for codegen).
Finally, if you don't want to use Rust drivers, you can simply configure them out and don't need to build Rust. It'll be quite a long while until Rust will arrive in the kernel outside of drivers (which tend to benefit most from Rust anyway).
Majority of the time is spent in LLVM, because rustc throws a ton of code at it to clean up. This is being addressed by MIR optimizations (rustc's built-in optimizer working on higher-level code) to remove costly abstractions before they become a pile of low-level code to eliminate.
Came across this page, on the motivations of Gentoo, a few years later:
> Installing a working Linux box used to require over 550 man hours, learning a Nordic language, sacrificing a goat, wading through hundreds of pages of (purposely) inscrutable help files...Old-school Linux users were desperate to find a new way to feel superior.
It turns out that compiler speed makes development faster, keeps people interested in the language, and ultimately allows more iterations before release (which can be better for speed and safety than throwing in a bunch of extra compile steps).
Rust offers a better trade-off between compilation time and other parameters, than C++.
And let's admit even though Rust compilation speed could use some improvement, it's not terrible.
And yes, a substantial time is also spent for things like the borrow checker but no all of it.
An no, it is not a niche requirement. Short compile times are absolutely critical for developer productivity. One main reason Golang exists and got popular is that people got fed up with how slow C++ is to compile. Not to mention that most people on this earth are not as privileged as to have a beefy machine.
Can you back this up? I'm not a programming language expert, so if there's some common knowledge you're referring to, I'm not aware of it.
> most people on this earth are not as privileged
There's very little "privilege" you need to compile Rust. In my spare time, I develop mainly in Haskell on my ThinkPad X270 and it goes just fine. Building libraries takes time, yes, but you need to do it just once. And while it's building you can think with a piece of paper more — also privilege in a way.
So what you're saying is that you're opposed to millions of people having more secure software, and perhaps millions of dollars spared from breaches, because it makes your own occasional singular personal experience of compiling the software faster?
Or did you mean something else?
Fast compute at this point is quite literally the least expensive part of the equation. Machines will get faster. Compilers will get optimized.
We've spent decades optimizing the developer experience (compile times) at the expense of the rigor, robustness, and quality of our resulting product. I've been doing this for 30 years, and I can categorically say that I've spent FAR more time chasing NPE, OBO, and race condition bugs than I would have ever added to my build time with a slightly slower compiler.
itripn&
The idea here would be to compile in a way that just assumes everything is correct and either crashes catastrophically or produces invalid output otherwise. But in doing so, it should allow at least slightly faster compilation.
You could even remove the need for this "fast and loose" compiler to do any type inference by shipping pre-processed source with all types resolved. However I don't know if this addition would fit your needs if, e.g., your goal is to be able to compile from any given commit rather than only official releases.
At very least it could be an interesting experiment to discover what tradeoffs are possible.
It seems, there is a wide group of developers which thinks that Rust is the best candidate as a kernel development language. If you see issues with that choice, now would be the time to propose an alternative and try to find momentum in the developer community supporting that alternative. While I also lack practical experience there, by all what I heard, ADA could be one. But I don't know how it exactly compares to Rust and what the trade offs are. But so far, no one has pushed for ADA as a possible kernel implementation language.
So to bootstrap rust this way, you'd need to go GCC 4.2 -> GCC 13+ -> Rust
I followed the https://rustc-dev-guide.rust-lang.org/building/how-to-build-...
Then ran the build step "time ./x.py build -j 8" ... Build completed successfully in 0:36:53
real 36m53.398s user 254m52.720s sys 12m48.289s
Seems pretty reasonable considering it's not a particularly high end desktop from 2015. Seems like a cheap price to pay for increased reliability and security.
If I had a choice between 50% CVEs and 2h per compile or 5' compiles, I'd take the former in a blink.
I know this because I've managed to add a form of RAII and borrow checking to portable C11, and C is known for being faster than Rust to compile. Imagine what happened if we made a language with that stuff built-in.
The funny thing is that C is also slower to compile than it could be because of headers.
I haven't checked the times, but if the borrow checker is really the slowest part, maybe making rust skip it is a valid approach for end users. Sounds like an interesting experiment.
Regardless of how fast one's machine is, I think having some compile caching infra like sscache should help improve time to compile.
[0] https://perf.rust-lang.org/compare.html?start=9be2f35a4c1ed1...
Relevant username?...
In my experience when compilation takes longer the actual delays are in the last steps, long after the compiler is done checking and printing warnings.
On an unrelated note, I also noticed a large amount of static string literals in the code can slow down compilation to a surprising degree.
Now, I wonder what is the most reasonable option: writting a naive and simple 'c11' compiler or a naive and simple rust compiler.
I wonder if somebody has done a "syntax complexity diff" between 'c11' and rust.
I know that linux is written in "gcc C", not 'c11'... so...
On the other end of the software stack, we have servo, mozilla web engine written in rust. What's up there? Still a drop of rust in a ocean of c++? (SDK included).
Because after years, if it is still impossible to run servo without c++ code, this is bad omens for kernel rust.