Lessons learned from a successful Rust rewrite
gaultier.github.io
gaultier.github.io
fn main() {
let mut x = 1;
unsafe {
let a = &raw mut x;
let b = &raw mut x;
*a = 2;
*b = 3;
}
}
That said, this doesn't mean that the pain is invalid; unsafe Rust is tricky. But at least in this case, the fix isn't too bad.
(All right, i don't actually know if this is old or new. But you get my point?
> you get my point?
I don't think Rust adds keywords very often. But I can acknowledge this is a subjective point. Additionally, in this case, it is literally a one sentence explanation: `&raw mut` is how you can create a `*mut T`, and `&raw` is how you can create a `*const T`. That's it. You can safely ignore this whole thing until you're writing some unsafe code. It doesn't feel like a large burden to me, though of course I am biased.
Joke aside, my main issue with Rust is that is is already more complicated than C++. And it is still growing.
I agree that it seems like the team has an appetite for change that's larger than I personally would agree is appropriate. We'll see what they end up shipping.
Rust on the other hand has a pretty huge base part that everyone needs to learn. Because certain normally simple things can get very complex in Rust and you start approaching the dark corners of the language very very quickly.
So yes, I belive Rust is more complicated that C++. And it is still growing
That being said they do definitely make it hard to do certain types of things (data structures with cyclic references, etc). But that’s not quite complexity, that’s more of a constraint on expressing complexity
To me complexity comes from implicit lifetimes, operator overloading, turing-complete templates, unhygienic macros, lack of a blessed package manager/build system, many different types of implicit constructors/destructors used in different contexts, easy to hit language footguns, etc.
IMO lifetimes (and downstream interactions with things like async) are the most complex bit of Rust.
But for a fairly significant number of places in Rust projects you can avoid them. The same isn’t true for the complexities of C++ I listed above.
I mainly program in C++, and the past 5 years or so of writing Rust has drastically changed the way I write code (in general). Sometimes, if the problem is particularly hairy, I'll even write the code in a similar way in Rust first and then port it back to C++ after the compiler has helped me iron out my mistakes.
It's definitely a big part of it, but I think sometimes it is a Rust thing. Of course, motivated by the essential complexity, but accidental complexity is still complexity. I don't think or hope it's an existential issue for Rust, which is good. The sharp edges around pointer-reference nuances in unsafe, for instance, are being addressed by things like raw references (mentioned upthread). I think languages that explore this problem domain more might provide interesting alternative designs for what Rust handles through unsafe.
What kinds of projects or C++ features are requiring such high usage of unsafe? I'm not implying that this is bad or unnecessary--I'm genuinely curious as to what requires unsafe to be used so frequently. Since by all accounts unsafe Rust can be harder to use than C++, this may help inform me as to whether I attempt using Rust in future rewrites.
> This project is a library that exposes a C API but the implementation is C++, and it vendors C libraries (e.g. mbedtls) which we build from source. The final artifacts are a `libfoo.a` static library and a `libfoo.h` C header.
In this case, this project is doing a lot of FFI, both exposing C, as well as calling into C libraries. That's unsafe. Which is a good example of a project that may use unsafe more than the average Rust project.
And for the C libraries they vendor in, assuming none of them are exposed directly in their public API, it's likely they can replace them with Rust libraries with equivalent behavior. mbedtls seems like a good example of that; certainly it wouldn't be a small effort to switch to rustls, but it might be worth it to do so. And even if they didn't choose to do that, I just did a quick search on crates.io, and it looks like there are safe wrappers for mbedtls.
There is no mention of how much of the codebase is even in safe Rust after all this work so no clear value to the migration?
Frequently when people get their code ported they then begin a process of reducing the unsafe surface area but not here.
The author seems to have little or no value on safe Rust? It doesn't seem evident from reading/skimming his 4 articles on the process.
Interesting mechanical bits to read for sure though so it' still a useful read more broadly.
It's unsurprising that the author would go use Zig next time since they didn't seem to have any value alignment with Rust's core safety guarantees.
That is not at all an obvious axiom.
I am willing to concede that "Rust" is safer than "C".
However, in "unsafe Rust" it is super easy to violate a Rust API precondition that the compiler takes advantage of. Even the Rust intelligentsia have pointed out that writing correct "unsafe Rust" is significantly harder than writing correct C.
HN discussion: https://news.ycombinator.com/item?id=41944121
While we at it, someone argued that Zig is nicer than unsafe Rust: https://zackoverflow.dev/writing/unsafe-rust-vs-zig/
That's how the project ended up in such a bad state by the end. Instead of having rust-rust linkages that the compiler could check, they designed every function boundary to be an uncheckable rust-* linkage. This would be like porting a C library to C++, but only moving one function at a time, such that every single function had to comply with extern C.
Here is an important warning:
The difficulty of setting up boundaries to unsafe languages are the hidden reason for people wanting to rewrite every C library in Rust. Do not choose a design pattern that requires more than one of these boundaries to exist within your own code!
I don't think that's true. The end application talks to smart cards. See also one of the links: https://gaultier.github.io/blog/how_to_rewrite_a_cpp_codebas...
However, the codebase has the possibility to use multiple memory allocators, and Rust is simply actively bad when faced with that.
It just seems like the codebase has a set of idioms that really lean into the areas that Zig is actively good at and where Rust is weak. Memory allocators, lexical (not RAII) defer, C interop, C++ interop, and cross-compilation are Zig's raison d'être, after all.
The one thing I disagree with is the complaint about "repr(C)". Sorry, but I've become convinced that if we want our compiler and languages to work well on modern hardware, we're going to have to allow the compilers to do lots of struct-of-array to array-of-struct (and vice versa) transformations depending upon the actual access patterns. That means that a struct or an array will not be locked to a specific memory layout unless you specifically request as such.
It _is_ valid to have multiple mutable pointers, just as C and C++ allow. It's when you have multiple live, mutable references (including pointers created from live mutable references) that you end up in UB territory.
Interfacing with the C world, both as caller and calllee, happens a lot in real world code. All the C bugs come right back at that point.
What people mistakenly take for the C ABI, is in reality the OS ABI when written in C.
Two C binary libraries might fail to link, or reveal strange behaviours/crashes, when compiled with different C compilers, for anything that isn't clearly defined as part of the OS ABI.
> I am still chasing Undefined Behavior. Doing an incremental rewrite from C/C++ to Rust, we had to use a lot of raw pointers and unsafe{} blocks. And even when segregating these to the entry point of the library, they proved to be a big pain in the neck.
These sound like an artifact of the rewrite itself, and I suspect many of these unsafe blocks can be rewritten safely now that there is no C++ code left.
> I am talking about code that Miri cannot run, period: [some code that calls OpenSSL (mbedtls?) directly]
This should be replaced by a safe OpenSSL (mbedtls?) wrapper, or if it wouldn't change the behavior of their library in incompatible ways, rustls.
> I am still chasing memory leaks. Our library offers a C API, something like this: [init()/release() C memory management pattern]
Not sure what this has to do with Rust, though. Yes, if you're going to test your library using the exposed C API interface, your tests may have memory leaks. And yes, if your users are expected to use the library using the C API, they will have to be just as careful about memory as they were before.
The benefit of this rewrite in Rust would be about them not misusing memory internally inside the library. If that benefit isn't useful enough, then they shouldn't have done this rewrite.
> Cross-compilation does not always work
I've certainly run into issues with cross-compilation with Rust, but it is always so much easier than with C/C++.
> Cbindgen does not always work. [...] Every time, I thought of dumping cbindgen and writing all of the C prototypes by hand. I think it would have been simpler in the end.
I'm skeptical of the idea that an automated tool is going to generate something that you'll want to use as your public API. I would probably use cbindgen to get a first draft of the API, modify and clean up the output, and use that as the first version, and then manually add/change things from there as the API changes.
I don't want to silently, accidentally change the API (or worse, ABI) of my library because a code generator changed behavior in a subtle way based on either me upgrading it, or me changing my code in a seemingly-innocuous way.
> Unstable ABI
This is a bummer, but consider that they are not exposing a Rust API to their customers: they're exposing a C API. Why would the expect to be able to expose Rust types through the API?
And they actually can do this: while it is correct that standard Rust types could have a different layout depending on what version of rustc is used to build it, that doesn't actually matter for a pre-built, distributed binary, as long as access to those types from the outside code (that is, through the C API) is done only through accessors/functions and never through direct struct member access. Sure, that requires some overhead, but I would argue that you should never expose struct/object internals in your public API anyway.
I'm also not totally convinced that rewrite from scratch is always the wrong thing. For small projects the total work rewriting can be much less than dealing with this kind of FFI.
So we just have to do our best.