AutoCXX: Safely call C++ from Rust with auto generated bindings
github.com
github.com
[0]: https://news.ycombinator.com/item?id=24211691 [1]: https://github.com/dtolnay/cxx [2]: https://www.chromium.org/Home/chromium-security/memory-safet...
Not to mention as far as I know std::expected does not actually exist. It was first proposed 7 years ago but is still a proposal, now hoping for inclusion in C++23 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p032...)
Look I hate to be that girl with the strong opinions on the orange website, but "goto is not an appropriate feature for an ffi" is a hill I'm ready to die on.
Also the exception does not leak into rust-space. Like it or not, any serious c++ ffi must accommodate C++ exceptions, since that is a standard feature (not to mention idiomatic error signaling mechanism) in C++.
They are not isomorphic. Return always hands control back to the caller; you can emerge from an exception inside any arbitrarily higher scope, and the proliferation of weird edge cases (what happens when I throw an uncaught exception in a constructor or destructor, the exception safety guarantee hierarchy) are good evidence for why this behavior is too complicated for its own good.
> Like it or not, any serious c++ ffi must accommodate C++ exceptions
Obviously this is the case and I'm not trying to say otherwise. I already said so upthread, but it seems to me that the least complicated way to do so is to wrap wrap foreign C++ function return values in a Result<T> (unless they can be statically proven not to throw), and report errors to C++ through some kind of Either-ish box. Then the C++ caller can decide whether they want to handle the error then and there, or throw an "idiomatic" exception.
> They are not isomorphic.
It was never claimed that “return” and “throw” were isomorphic. Only that they are both examples of non-local exits.
> good evidence for why this behavior is too complicated for its own good.
That’s a nice opinion but I wouldn’t consider that evidence except under the loosest definitions of the word. By the same reasoning I could claim that being able to invoke “return” at any arbitrary point in the function is evidence that return is a complicated feature.
Just like “return” exceptions in C++ have a statically defined set of areas they can branch to and the invocation of throw is well defined w.r.t. to object cleanup. Admittedly those landing sites are more numerous than “return” but not different in their more defining properties.
Not really, because a return statement (like a go-to) only has one place to go. An exception has an unlimited number of places to come from. That put it into its own special circle of hell.
> An exception has an unlimited number of places to come from
False. An exception can only come from a throw statement, which is both lexical/statically defined and finite in number within every codebase.
Obviously.
> ...which is both lexical/statically defined and finite in number within every codebase.
Not in every case. Any time you have a library that calls into client code the set is dynamic and uncountable from the perspective of the library author.
Exceptions are probably the worst thing in computer science.
This is a hill I'm happy to join AprilArcus on.
False. exceptions can only come from throw statements, which are both static and finite within every codebase.
> Exceptions are probably the worst thing in computer science.
Opinion, irrelevant.
Not between libraries and library customers. From the perspective of the library author, the set is dynamic and uncountable, and similarly from the perspective of the library customer, any exceptions thrown are dynamic and uncountable.
> Opinion, irrelevant.
It's almost like opinions come from some basis worth exploring instead of outright rejecting without consideration. In my (obviously irrelevant) opinion, you may grow as a person if you learn to explore the opinions of others :)
On the flipside, any serious gamedev C++ FFI must accomodate exceptions-disabled environments, since that's common in gamedev codebases - and even the default settings on at least one relatively modern gamedev platform.
Any C++ ffi that accommodates exception also accommodates exception-disabled environments by virtue of accommodating function calls. Certainly AutoCXX accommodates codebases that use exceptions as well as those that disable exceptions by convention.
From the documentation, it sounds like that's only true if you eschew Result, the absolute backbone of not-automatically-unwinding Rust error handling.
You could presumably, technically, roll your own Rust struct/enum - "NonStdResult" - and expose that to C++ however cxx deigns to expose Rust enums to C++. But that seems... very awkward at best, to completely counter to the entire point of AutoCXX - in trying to eliminate the need to write a bunch of special case conversion boilerplate - at worst. If you're suggesting that as a superior design choice, we're seriously going to have to agree to disagree.
Even setting aside personal taste, some gamedev platforms - even in $(CURRENTYEAR) - still default to C++ exceptions being disabled. And may throw linker errors if you try to enable exceptions, while linking any closed source third party libraries that were built with default exceptionless build settings. I've rewritten my share of exception-based error handling - just to sanely port across platforms - as a result. It's just as well - I've fixed enough bugs where exceptions propigate across a C ABI to consider them UB-bait.
There's a reason Rust lets you configure panic="abort", and a reason it gives you Result s without magic unwinding semantics, and a reason why a lot of C++ codebases have ended up with some kind of custom Result-alike.
To do that, it would essentially need to literally prove that the C++ code is thread and memory safe, which is an open research problem at Best, and probably impossible since it requires solving the halting problem.
If it can do that, then the actual binding generation would be the most uninteresting part of this work.
Basically, burden of proof of using unsafe correctly is on the programmer.
A function like memmove would be completely unsafe in Rust for instance but C++ could expose such an interface without having to do anything special.
Not a problem. Rust checks are applied to the Rust code you write or import, and everything else is just ingested as is.
No. C++ programs are safe under certain conditions. If they aren't then they need to be changed in C++ or in Rust interfacing code.
Even if something is unsafe, it can still be used safely.
Publishing safe Rust APIs that are _intentionally_ unsound should at best be warned about to any other crate that depends on it. Ideally, those authors and their crate would simply be banned.
Either you change the C++ lib to not behave unsafely for inputs.
Or you manually change the binding to keep the invariant.
The assumption behind linking to C++ code is that it doesn't contain unsoundness. If it does the library was fucked way before.
But this discussion is about a tool that:
* does not change the C++ code to behave correctly for all inputs
* does not (and technically cannot) generate safe Rust bindings that keep the invariant (the bindings are automatically generated under the assumption that all C++ code is safe for all inputs).
I think this is already invalidated by countless code generation and build scripts.
Even regular safe Rust assumes you use unsafe in a proper manner.
This build tool assumes your C++ code is safe enough to wrap around unsafe trivially.
This build tool, as first proposed, knowingly exposed macros that can cause memory unsafety with the wrong inputs, without making you write unsafe. If a build tool makes assumptions like "you probably know what you're doing with that C++ code" without asking you to write unsafe, it is just as wrong as an implementation of unchecked indexing that wasn't marked unsafe (because "you'll probably know not to use this unless you're really in bounds."). It is clear that you consider this a trivial matter, but many people very heavily involved in the Rust community (including the people working on its soundness proofs) do not.
This was done to deal with remarks Chromium devs had against Rust C++ binding.
In lieu of that it's liberal treatment of unsafe is understandable.
I have a hard time believing those Chromium devs know what they want.
Their request allows safe Rust to introduce undefined behavior, making the unsafe Rust keyword meaningless (since now you can't even grep for the places in the code where this UB is introduced), and eliminating the actual reason they want to use Rust in the first place (to avoid undefined behavior, segfaults, CVEs, etc.).
The only thing complying with the request will achieve is hurt the language. Once the chromium devs start hitting the same CVEs and segfaults in Rust that they were hitting in C++, they will just tell the whole world that Rust is as bad as C++ and using it is not worth it.
Rust std wasn't yet verified to not contain unsafe behavior. In fact, some modelling discovered and fixed unsafe behavior in Rust std libs.
Same happens here. If you discover that for inputs you have unsafe behavior calling ffi, you uncapsulate it in way that prevents misuses or fix in source ffi lib.
I entirely agree with this. My point is that these conditions are not always (and often aren't) encoded in the C++ syntax itself, and are instead part of the documentation or some softer coding convention.
As such an automatic code converter can't be aware of these details and can't decide if a call is safe or not. C++ doesn't have the concept of safety built in, it's the programmer that's in charge of keeping track of the soundness of it all. It's not enforced by the compiler like Rust does.
I'm really surprised that there's so much debate around this in this thread honestly (including by people who know Rust very well). That's blindingly obvious to me that it's a very bad idea. Clearly that means that I'm missing one side of this issue.
Given that C++ has no explicit concept of safety it seems like it would be hard to do that automatically as soon as pointers or references are involved.
As a quick example: if you have the following signature in C++:
int *foo(int &a);
As a human doing the same task I'd have to ask myself at the very least:
- Can the return value be NULL?
- Can the return value alias a?
- What's the lifetime of the returned value and who owns it?
You really don't. Rust checks are applied to the code written in Rust. If a developer intentionally marked a dependency as safe then you get exactly what you've asked for.
Sometimes it's unquestionably better to have a working system than not having one just because a pedantic compiler complains about stuff that you can't do nothing about.
Yes this may mean you sometimes can't create a safe interface to a particular foreign function. However all this means is that calling the function requires an explicit unsafe block and extra care. But that 'unsafe' marker is valuable! It's something that screams "here be dragons".
If you erroneously tag unsafe interfaces as safe then you basically throw all of Rust's safety out of the window. You can end up with multiple mutable borrows, or borrows with a bogus lifetime or whatever.
Raw bindings to OpenGL in Rust is actually a good example of FFI that wouldn't be safe to call for about half a trillion reasons. There are so many implicit constraints in there that even manually designing a safe Rust interface is a rather complicated endeavor.
If this is not the case, then this whole library thing is _unsound_. We really need an "intentially-broken" tag in crates.io to identify crates that lie about their soundness intentionally.
No need to write 'unsafe' because .cpp files are already known to need carefull review.
That's a misunderstanding of safety, and ub, and `unsafe`.
The C++ code could be unsafe when called with certain values which it is not normally called with. This is common. This is also not allowed in Rust, it'd be unsound.
Furthermore C++ has different notions of safety than Rust. C++ allows dangling and null pointers (whether raw or smart), it doesn't allow calling them. Rust does not allow dangling or null pointers unless they're raw. You can have a null unique_ptr, you can not have an empty Box.
The cxx crate and the autocxx tool should make sure that the exposed C++ functions only take arguments types which have well defined semantics.
In your example, a rust Box<T> maps to a rust::Box<T> in C++, which cannot be null. And a unique_ptr from C++ maps to a cxx::UniquePtr in rust which can be empty.
If somehow the C++ code puts a dangling or null pointer into a rust::Box, that is clearly a bug in the C++ code.
The real problem is, however, that C++ lacks an "unsafe" keyword, so functions like:
/// # Unsafe
///
/// Must call `bar` after a sequence of calls to `foo`
unsafe fn foo();
fn bar();
/// note: must call bar after a sequence of calls to foo
void foo();
void bar();
To generate Rust from C++, it does not suffice to just "look at the types" like cxx and autocxx do. One also _at least_ need to read all the API documentation comments, check if there are any invariants that must be preserved, and act accordingly.
If the APIs are ok and can be wrapped mechanically, the actual wrapping can be made trivial with tools, but there is no tool today that will tell you whether this is the case.
That is, at the end of the day, if you need to expose 10k C++ APIs from Rust, you will still need to manually inspect those 10k C++ APIs, and _think_ about whether they are safe or not.
That's the time consuming part, and you actually want to only do this once, and write down why an API is safe or not, so that other programmers don't have to repeat this work every time you hit an FFI issue.
So IMO while cxx and autocxx are "ergonomic", they spare you only the easy lest time consuming portion of the work. autocxx also makes it easy for you to either not check, or not write down the result of the check, and this could end up creating a lot more work down the road.
---
Note that this is something one wants to do even when one trusts that the C++ code is correct. In the example above, the C++ APIs can be correct, but one can still UB by using them incorrectly.
* int foo(); which returns an uninitialized int is OK according to C++ rules, but would need a MaybeUninit<c_int> according to Rust rules.
* int foo(); could throw an exception, causing UB in Rust, since Rust assumes FFI declarations not to throw according to the spec. Rust can only export `noexcept(true)` C++ FFI declarations, or C functions (since C cannot throw). Apparently, autocxx and the cxx crate ignore this and treat all C++ functions as if they never throw, giving them a safe API. That's unsound. (One can fix that on nightly Rust though).
Unsafety can also be introduced through ABI incompatibilities, but IIUC autocxx usage of rust-bindgen deals with that.
Safe bindings to unsafe code need to enforce the invariants to make the calls safe, otherwise they are not safe.
Consider this code:
fn int_to_string(e: &mut u32) -> &mut String {
unsafe {
&mut*(e as *mut u32 as *mut String)
}
}
fn main() {
let mut i = 42u32;
let s = int_to_string(&mut i);
s.push_str("Ayyy");
println!("{}", s);
}
If you just start tagging random, potentially unsound interfaces as safe, what's even the point?
And if you agree that the code in this example is bad and "int_to_string" should definitely not be considered safe, why would that change if I rewrote it to make "int_to_string" a C++ function called through FFI instead?
unsafe { ... }
People lying about having proved safety in their crates accidentally is bad. People doing this intentionally is extremely bad and I wish there was a way to automatically reject being able to depend on intentionally unsound crates in crates.io (or ban their authors are their crates from pushing anything to crates.io since they cannot be trusted).
Code generators that automatically generate thousands of broken soundness proof en masse and by design are IMO the ultimate evil. They completely defeat Rust's purpose. It makes absolutely no sense to interface Rust and C++ in this way, and the people doing this would be better off by just sticking to C++ instead of trying to make safe Rust unsound.
If safe Rust cannot be trusted, Rust value proposition is _dead_ (you cannot hack without feat anymore, refactor without fear, avoid segfaults, ...). This people are writing tools to automatically generate massive amounts of broken Rust code. If crates.io does not protect Rust users from them, we need a different crate repository that does.
I think this debate is really, really interesting.
Just haven't found time to make the compiler PR to allow exposing that syntax to macros yet, but I will.
(i still think its a soundness bug in Rust that `extern { }` does not require unsafe since one can use it to trigger UB in safe Rust code)
Frankly Rust values being a usable language too much to be 100% sound.
> Frankly Rust values being a usable language too much to be 100% sound.
Rust has a really good track record of identifying, prioritizing, and fixing soundness bugs (e.g. it took 4 years of work to fix one floating-point soundness bug!). Many people continuously work on this at the academic, toolchain, and backend (llvm, crane lift) levels. There is also a lot of people continuously working on making sure that new language features like async/await, const generics, specialization, GATs, ... are sound.
TBH i'm surprised to learn that not all Rust core members consider soundness to be a Rust core value. Maybe it isn't a Rust core value? (it is for me; without it, Rust makes no sense as a language to me)
The unsafe in `unsafe mod ffi { ... }` is literally the proof that all APIs exposed in the block are sound to call from safe Rust.
It would only need to go hand in hand with a comment explaining why each API in the block is sound to call safely, and it allows users to not list unsound APIs in there, but wrap them manually when needed. Along with code that checks the C++ lib version, etc. to make sure that the proof are kept in sync with each version of the lib.
That's completely different from not requiring any unsafe in the Rust side, and doing this en masse via bindgen.
How is it not a compromise? Party A wanted (and implemented) something that didn't require `unsafe`. Party B wanted `unsafe` to be used. After much discussion, Party A concedes with allowing people to put `unsafe` in some of the code, and Party B concedes that putting it there is sufficient instead of requiring it to be strewn all over the place.
Sounds like a (reasonable) compromise to me.
Party A wanting safe Rust to have undefined behavior was wrong. Party A now adds unsafe to their API, so that undefined behavior only happens in unsafe Rust.
I don't see the compromise anywhere. Party B told party A that they were wrong, and party A acknowledge it and fixed their crate.
cxx generates code that uses `unsafe` blocks/functions. It then generates safe wrappers around those, which are what it exposes to users. It's no different then someone doing this:
pub fn safe_fn() {
extern { fn unsafe_fn(); }
unsafe { unsafe_fn(); }
}
Statements like "Party A wanting safe Rust to have undefined behavior was wrong" are just straight up incorrect.
cxx is currently unsound, and the author (dtolnay) has chimed in above with a fix they are considering that fixes it (https://news.ycombinator.com/item?id=24244121)
The API of the cxx crate is safe, and it can cause UB, therefore it is unsound.
It doesn't matter that the cxx crate API is a macro. Yes, this macro expands to unsafe code like you mention, but the problem is that this unsafe code is often "broken" (unsound).
The Rust spec defines "unsound" as "introducing undefined behavior in safe Rust". The API of the cxx crate allows a safe Rust program to have undefined behavior, and it is therefore unsound.
If you don't want to have unsafe everywhere it seems to me that the reasonable path (taken by many Rust libraries) is simply to wrap the raw C/C++ interface around a Rust interface that enforces the safety invariants. That's why you can write safe GTK or OpenGL code in Rust, using Rust libraries that expose an actually sound interface.
I mean, FFI code is tagged as unsafe because it is unsafe to call. I don't see how "but I don't like having to write unsafe everywhere in my FFI code :(" is in any way a reasonable technical argument here.
I intentionally avoided taking a side in this debate. My response was focused on the claim that there was no compromise between the parties, when I think it's pretty clear cut that there is a compromise.
Personally, I'm sympathetic to both parties. In my own personal FFI-binding library I chose to let the programmer decide per-method whether it's safe or unsafe.
I think there might be a misunderstanding here. I interpreted the `unsafe mod ffi { ... }` to be like `unsafe fn foo()`, declaring the module as unsafe, not an unsafe block where we're telling the compiler we will maintain the invariants ourselves.
It is somewhat unfortunate both the proof obligation and proof 'declaration' use the same token.
There have been some RFCs open to improve this situation (e.g. unsafe blocks in unsafe functions comes to mind).
One of the last ones was the actix debacle, which was contained and accidental and ended up badly.
This is IMO infinitely worse.
Rust code author knowingly exposes unsound safe Rust APIs.
The details are different. Actix did so in their own project and were willing to fix it (the debacle was mostly social, and in some sense even accidental), while here exposing unsound safe Rust APIs is the whole raison d'etre of the project.
AutoCXX doesn't even check that the underlying C++ code does not change. So even if an autogenerated safe Rust API happens to be "accidentally safe", that can change any time without Rust users knowing.
This is where we disagree. I 100% agree that knowingly exposing an unsound API to safe Rust is a bad thing. However, what I see is the user of cxx/autocxx saying "I am asserting that this API is sound."
> AutoCXX doesn't even check that the underlying C++ code does not change
This is no different than any other Rust code calling into C++ code. The author has to declare that they believe it is safe either way.
And since the API of this library allows this assertion to be performed in safe Rust code, when it fails, safe Rust code has UB, which according to the Rust language reference makes the API of this crate unsound.
Unsound safe Rust APIs are broken Rust code. The Rust language spec and toolchain make no guarantees about what the behavior of these APIs is.
So that's what I see here. Just another safe Rust API that is broken by design. Maybe with the twist that this crate is actually a factory to generates thousands of those broken safe Rust APIs in masse, which kind of makes it worse than your usual "broken API" bug (which is what these are, these are soundness bugs in Rust libraries).
This assertion is always performed in safe Rust code. That's how you turn unsafe code into safe code: you write "unsafe { }" in safe code. That this does this in the body of a macro is not material.
(I know you disagree and don't think we're going to get anywhere, and frankly, find your aggressiveness really offputting.)
Safe Rust code performs these assertions by using the "unsafe" keyword. This library API allows safe Rust to perform these assertions _without_ writing "unsafe".
That's the problem.
If so, why? How is that materially different from calling a function with unsafe in its body? If not, why is this particular macro different?
No. Exported macros (as opposed to private ones) are only sound if they do not allow safe Rust to introduce undefined behavior.
Whether these macros use "unsafe" internally or not is irrelevant. For example, `pin_mut!` uses unsafe internally, but it does not allow safe Rust calling it to introduce UB.
A macro that allows safe Rust to introduce undefined behavior is unsound. An example of such an unsound macro would be `offset_of!`.
---
That is, I do not differentiate Rust abstractions when it comes to soundness. Whether its a function, a trait method, a macro, or a function pointer, it does not matter. If an abstraction its safe to use it shall not introduce UB. If it does, it is an unsound abstraction.
There is no debate about this.
You didn't really answer my question. I'm also not really gonna continue this argument.
Of course there is, for example, the offset_of! macro was unsound for a long time:
* https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-o...
* https://github.com/rust-lang/unsafe-code-guidelines/issues/1...
* https://github.com/rust-lang/unsafe-code-guidelines/issues/2...
* https://github.com/rust-lang/rust-memory-model/issues/35
> You didn't really answer my question.
I literally wrote "No.". To your follow up questions starting with "If so ..." i did not reply, because the assumption these questions were based on (that I would reply to the previous question with "yes") did not hold.
To expand on this. Rust does not have "unsafe macros", so all exported macros must be sound. Whether a macro is sound or not is orthogonal to whether the macro itself uses `unsafe` (and your original claim was whether I thought that macros containing unsafe should be rejected by the compiler, to which I replied "No.", `pin_mut!` is an example of a macro that uses `unsafe` and is sound).
Whoever invokes the macro asserts that doing so is sound / cannot cause UB, and having "unsafe" in the macro name draws attention to that in the same way that the unsafe keyword itself does.
Right now that's very implicit, and I bet there are users of the library that don't think they're asserting that. They expect you to follow the documented rules about invariants.
Requiring people to write 'unsafe' at the import site makes that a lot clearer, but if it's mandatory then people are probably going to write it whether the API is sound or not.
I'd be more comfortable if you could either declare an API sound when you import it or write 'unsafe' everywhere you call it.
TL;DR: I think this is just regular old composition of Rust features, and that using the library itself is the proof obligation. I can see why others don't like it. I think dtolnay has come to a good compromise.
(I will point out, however, that I still seriously doubt that every single one of these autogenerated functions is safe, and will probably avoid libraries that use CXX for that reason. But at least now someone who hasn't heard of CXX can come to that conclusion independently :)).
I kinda feel like the original sin in this discussion is that "unsafe {}" should really have been called "safe {}" or "sound {}" since it's really the programmer telling the compiler "hey, this block is safe to run as-is, I've checked, trust me on that". An automatic C++ binding library is in no position to make this assertion.
It's weird to quote the Holy Scripture back at you of all people but here it goes (emphasis mine):
https://doc.rust-lang.org/beta/reference/behavior-considered...
> Rust code is incorrect if it exhibits any of the behaviors in the following list. This includes code within unsafe blocks and unsafe functions. unsafe only means that avoiding undefined behavior is on the programmer; it does not change anything about the fact that Rust programs must never cause undefined behavior.
> It is the programmer's responsibility when writing unsafe code to ensure that any safe code interacting with the unsafe code cannot trigger these behaviors. unsafe code that satisfies this property for any safe client is called sound; if unsafe code can be misused by safe code to exhibit undefined behavior, it is unsound.
This library generates safe bindings to unsafe FFI calls without actually checking that said code is safe to call that way. It's therefore totally unsound until proven otherwise.
So what's the argument here? That the C++ code might be safe to call that way if we're lucky? I mean, sure, but how is that reasonable or practical? I definitely don't want a FFI library to tag functions as safe for me before I had the time to validate it, it seems to easy to let something bad slip unnoticed. If the interface is unsafe it means that I need to review it and manually mark it as safe once I've taken due diligence and I can vouch that the interface is actually sound. And I definitely don't want a third-party crate to expose a safe-but-unsound Rust interface with a big disclaimer that "basically it's just raw calls to C++, make sure you use it right otherwise it's segfault time!".
Again, it's mind boggling to me that this is controversial. It's not just bullshit language lawyering either, I think it's a terrible idea both in theory and in practice. I can already see the Rust crates leaking unsound """safe""" interfaces left and right because they're just a thin automatically-generated wrapper around C++ header files.
> So what's the argument here? That the C++ code might be safe to call that way if we're lucky?
No, the argument is that we've checked that the C++ code is safe to call, just like any other FFI. We've reduced the boilerplate of doing so with a macro, just like any other boilerplate might be.
> I definitely don't want a FFI library to tag functions as safe for me before I had the time to validate it,
Do you validate every single unsafe code in every single library you use? Even the standard library? Every time?
> And I definitely don't want a third-party crate to expose a safe-but-unsound Rust interface with a big disclaimer that "basically it's just raw calls to C++, make sure you use it right otherwise it's segfault time!".
I do not either, but CXX doesn't change this in any meaningful way. This could still happen before.
Nope. But if I use a rust library, the compiler will do that for me. If it's Rust. And not CXX.
I shouldn't have to in Rust. That's... the central premise here. Usage of unsafe code should say that it's unsafe.
We don't need to? Rust's main soundness theorem says that if a safe Rust API is sound, then safe Rust code using it is sound.
So the only thing we need to check is that the unsafe code that _we_ write is sound, and we 100% do this all the way for the standard library. In the standard library _every_ unsafe block has a comment explaining why the "user-provided soundness proof" that they represent is correct. We even have a linter that actually rejects PRs that add unsafe blocks to the standard library without such comments.
> Do you validate every single unsafe code in every single library you use? Even the standard library? Every time?
Soundness is IMO Rust's main feature and its main core value, and is more important than other Rust core values like, e.g., zero-overhead abstractions, which are also extremely important. Soundness is what enables "Hack without fear", "no segfaults", "refactor without fear", cargo ("large scale software without fear"), crates.io ("using other people's libraries without fear")... and it's what sets Rust apart from unsound-by-design languages like D, Nim, Zig, C, C++, etc.
I haven't read the book in a long time, but the documentation I do read (nomicon, spec, unsafe code guidelines, issue tracker, internals) makes it very clear that making sure that unsafe Rust code is correct is critical for the ecosystem and for users to benefit from the main advantages Rust has to offer.
1) C++ may be unsafe
2) The rust bindings are safe
3) But the rust bindings call potentially unsafe C++
Yet... it's controversial to want unsafe{}? unsafe{} is there SO THAT YOU KNOW that the rust compiler can't assert soundness.
Absolutely bananas.
2. many C/C++ library bindings in Rust ship the C code with them, so no need to install the library, only the compiler.
Not if the library comes with your system. One can use this macro to generate a more "ergonomic" "safe libc" crate, that just introduces UB in safe Rust.
It is your responsibility to vet your dependencies. For anything serious, you better setup a layered review process.
If you want a "vetted crates.io", then propose that. I’d be in favor. I certainly never liked crates.io to be the next NPM.
But telling people their crates are "evil" and trying to get them banned is breaking the CoC. The people uploading "broken" code, no matter how much they upload, aren’t the ones breaking it.
Rewrite everything in Rust in one go? Not going to happen.
The last teams I worked on all maintained a giant pile of C and C++ code. Which obviously sometimes is buggy, and sometimes might even exhibit memory safety issues. I’m looking for every opportunity to make things better and using Rust for to improve safety and reliability. However due to realities this isn’t going to happen at once.
So now we have 2 choices:
Option 1: use bindings and replace things step by step, knowing that we have still unsafe code (but less so!)
Option 2: Stick with C and C++
I think everyone agrees that option 1 is better than 2. But reading about it as „the ultimate evil“ makes me sound very uncomfortable.
No? Just do what every good Rust programmer does today instead?
If I have a C++ function like this:
// If a == nullptr the behavior is undefined
void foo(int* a);
mod ffi { extern "C" { fn foo(a: *mut c_int); } }
fn foo(a: *mut c_int) {
assert!(!a.is_null());
unsafe { ffi::foo(a) }
}
mod ffi { extern "C" { fn foo(a: *mut c_int); } }
fn foo(a: *mut c_int) {
unsafe { ffi::foo(a) }
}
If this is what you want, you are better off just sticking with C++ instead.
What Firefox, Servo, and any other good C++ project that cannot/does not want to review all C++ APIs does is to just write:
extern "C" { fn foo(a: *mut c_int); }
Firefox and Servo heavily use `rust-bindgen`, which will automatically generate all these FFI wrappers with a correct ABIs for you. The main difference is that rust-bindgen does not attempt to falsely convey the idea that these APIs are safe to call.
Consider the (completely and utterly standard) practice of writing a safe Rust library wrapping a C++ library. To be sound, this crate has two obligations: First, its `extern {}` function signatures must match the C++ library's. Second, its public API must enforce any additional soundness invariants from the C++ library.
Ensuring that signatures match is tedious and entirely mechanical- automating it increases confidence that a program is sound! Crucially, there is more to signature matching than mere arity and type layout- C++ vocabulary types express ownership information that bindgen alone does not capture!
Cxx isolates all the auditing work related to C++ vocabulary types, like `std::vector` or `std::unique_ptr`. A single audit of cxx covers this aspect of all cxx-using crates. This is an enormous win if your goal is to integrate Rust with a large C++ codebase like Gecko or Chromium!
The kernel of truth to your argument lies only in that second obligation. As Ralf concludes in that same issue thread, the problem of generated `unsafe` blocks is not new to cxx, but a near-universal aspect of Rust/C++ bindings. The solution is not, and never was, to give up and hand-write a bunch of `unsafe` blocks.
Instead, the solution is simply to ensure that auditors can easily locate generated `unsafe` blocks and the soundness assertions they represent. A typical solution here (also mentioned in that thread) is to add "unsafe" to the macro name or input. (Dtolnay already plans to do this for cxx.) Personally, I would argue that this is not even cxx's failing, but a tooling issue- if we're going to use macros to generate bindings, then our auditing tools should see through those macros.
If I can write an unsound Rust program using autocxx or cxx safe Rust APIs, then these APIs are unsound.
Whether these APIs are macros or function calls does not matter. There is no distinction at the API level about whether incorrect unsafe code is expanded, or called into. The API is safe, and for these crates, the currently-safe API introduces undefined behavior.
cxx will be fixed to require unsafe, but AFAICT autocxx usecase is to satisfy Chromiums requirements which require completely avoiding unsafe.
Putting "unsafe" in the name or input of cxx/autocxx/etc. is a great tweak that improves local readability, but it's not a fundamental change. The actual generated wrapper functions would still be safe, after all. If you're generating `unsafe` blocks, good tooling will point them out to you just the same as if they were hand-written. Nobody is trying to sweep that responsibility under the rug- not cxx, not Chromium, not autocxx.
That is, Chromium's requirement is notably not "completely avoiding unsafe." (This is another piece of hyperbole that derails productive discussion.) Their actual requirement is perfectly reasonable: restrict manual per-function unsafe to exceptional scenarios that deserve extra attention, above and beyond the usual check that a function is always safe to call with arguments that match its type.
Updating autocxx to require surface-level `unsafe` in the same way as cxx does not conflict with this requirement. However, it's also relevant that autocxx is a) rather new, and b) not (yet?) an official Google project: https://github.com/google/autocxx#license-and-usage-notes. It would be unwise to blindly infer any sort of Chromium policy from it at this point, especially when they have just published a rather detailed document that contradicts your inference.
Overall, the problem we should be trying to solve here is not simply "how do we prevent people from generating unsound code," but "how can we enable people to correctly bind to large C++ APIs?"
From all the soundness issues that have been filled on macros that were not safe in the past, the impression I have is that all exported Rust macros must be safe.
> but "how can we enable people to correctly bind to large C++ APIs?"
This is a solved problem: rust-bindgen, it correctly generates unsafe Rust FFI wrappers for all C++ code.
What the Chromium devs actually want is what I pointed out in my root comment: to automatically generate correct soundness proofs for their C++ code / APIs, so that they can automatically generate safe Rust FFI wrappers for them.
This is an unsolved problem, which is probably impossible to solve.
So I think this is an absurd requirement from the part of the Chromium project, that deserves the absurd solution proposed by the `autocxx` crate, which is to just assume that these proofs exist. The disclaimer "You can only use `autocxx` on those APIs for which those proofs actually exist" isn't very reassuring TBH. They don't appear interested in finding out whether those proofs actually exist, so what's the point really.
They will just start using Rust, continue getting CVEs because "oh, I guess we shouldn't have blidnly wrapped those thousands of APIs", and conclude that Rust isn't more secure than C++.
What Chromium wants is, again, not a way to generate soundness proofs, but a way to consolidate large numbers of similar proofs into a single place (the invocation of the binding macro), so that the remaining proofs can stand out.
Please, do us all a favor and stop misattributing absurdities to Chromium and autocxx. This is a useful part of the design space, ill-served by bindgen alone. Chromium is not the first project to hit it, nor will it be the last. For example, here is a Servo/Gecko dev's take on the subject: https://www.reddit.com/r/rust/comments/ielvxu/the_cxx_debate...
Chromium does not want to use `unsafe` on every call. The solution is simple, write a safe wrapper, that uses `unsafe` once.
> but a way to consolidate large numbers of similar proofs into a single place
That wasn't my read of their comment. If that's what they want, then the patched `cxx` crate which requires `unsafe` would give them exactly this.
The reddit user mentions:
> The key issue that people get worked up about, is that all C++ is unsafe,
This is not what we are talking about here. Lot of C++ code is safe, and writing a safe wrapper over C++ that's obviously safe is a one liner. What's problematic is doing so without checking _and_ without writing unsafe, by assuming that all C++ code is safe, which is what a library that asks you for a path to a headerfile and that will automatically generate safe wrappers in safe rust without asking even if the header file silently changes seems to encourage.
I also worked on one of the largest FFI Rust projects, and what one does is automatically generate thousands of unsafe C bindings, and have a crate exposing safe wrappers. Every time you need to call one FFI you either use the safe wrapper, or you add one if there isn't one. Very often, safe C wrappers weren't trivial, because what the FFI bindings were doing was inherently unsafe, and the amount of abstraction required to make that safe was prohibitive (and many many projects have run into this, Vulkan, wayland, ...). Then you either spend a lot of time into a safe abstraction, or you just use the unsafe API. Automatically generating safe wrappers instead just sounds like a bad idea, one would just be pushing all those issues onto safe Rust, which is not where they belong.
This is exactly what the first reply to that reddit user mentions, and here is your reddit user's reply to that, in which they explain that they were mostly referring to FFI when it comes to splicing Rust into C++: https://www.reddit.com/r/rust/comments/ielvxu/the_cxx_debate...
Calling Rust from C++ is a _very_ different use case from just calling C++ from Rust, which is what the Chromium devs mention is their primary concern:
> we are primarily concerned with the ability for new Rust code to call into existing C++ code
It's so different that one can in fact easily generate "safe" C++ bindings to safe Rust.
This is the thing I'm calling a cop-out. Chromium wants to avoid the error-prone boilerplate of writing out all those safe wrappers, by asserting in a single place that a whole collection of C++ functions are always safe to call with any arguments that match their (generated Rust-side) type.
> That wasn't my read of their comment. If that's what they want, then the patched `cxx` crate which requires `unsafe` would give them exactly this.
Yes, this is what I have been telling you. The Chromium document also says this at the end of point #1: "This particular property is satisfied by dtolnay’s marvellous cxx library already."
> > The key issue that people get worked up about, is that all C++ is unsafe,
>
> This is not what we are talking about here. Lot of C++ code is safe, and writing a safe wrapper over C++ that's obviously safe is a one liner.
We need to be careful with terminology here. All C++ code is unsafe in the sense that its soundness is unchecked by the compiler. Some C++ is also unsafe in the sense of an `unsafe fn`, placing additional expectations on its callers which are not captured by the type signature.
I'm repeating myself now, but it is absolutely possible to take a C++ header file, review the functions it declares, and decide "yes, the wrappers generated by cxx for these functions would all be sound." This is the case where autocxx helps- once you have such a header file, generating a bunch of safe wrappers is much less error prone than writing and maintaining them by hand.
You are correct that Vulkan/Wayland/etc. are usually not conducive to this. But that is why I linked that Reddit comment- the Chromium devs invoking cxx also control the APIs they're feeding it! The reply you linked is not talking about calling Rust from C++, but about this approach of calling C++ which you also own from Rust.
Once you control both sides of the language boundary, changes to the header are a totally different matter. All the thought that goes into preserving those functions' contracts, or updating their callers when they change, must "simply" take the Rust code (and its generated wrappers) into account as well. This is what "all C++ code is unsafe" means- regardless of whether or how Rust is involved, changes to C++ functions must consider their impact on soundness.
My only complain about autocxx remains that it does not require `unsafe`, but if they adopt the same fix that the `cxx` crate wants to adopt, that complain is gone. I hope the Chromium devs would be "ok" with such a small amount of unsafe code that condenses the proofs of all APIs being wrapped.
We kind of tossed aside the discussion of unsafe vs safe macros, but I'd understand some people see that differently (macros expand Rust code, but IMO they are not that different from generics, and we do have safe and unsafe generics; right now, AFAICT, we only have safe macros).
Like, say we wanted to make cxx or autocxx macros unsafe. We'd need to allow at item scope writing this:
mod foo {
unsafe { my_unsafe_macro!() }
}
I don't know. Seems like a lot of contortion for something that could be solved with just a rename like other have proposed, e.g., unsafe_autocxx_include!(...);
Might be worth opening an internal threads about this. Maybe Centril could shed more like on how hard this would actually be.
Most (all?) macros that I regularly use are sound (the derives, println!, debug!, offset_of!, pin_mut!...). So it might also be worth looking into how common of a problem this is. I've seen soundness bugs in macros before (offset_of!), but these were always bugs. This is the first time I recall a macro that's unsound by design.
Like, I get that you want the loophole to not exist, but does there exist a possible world in which Rust closed the loophole and you would actually be happy about it?
The fact that you can write bytes to random memory addresses if you intentionally set out to do so is not what Rust's safety system is about. Rust's safety system is a tool for the programmer to build working programs. It is not an end goal in itself - hence why unsafe exists and why so much of the standard library and so many third party crates use safe. It's a tool you have to use correctly, sure, but it's only a tool. We don't avoid memory corruption and data races because they are inherently bad - we avoid them because they cause bad things to happen, like getting the wrong results out of a computation or allowing attackers to subvert program control flow.
If you want to 100% guarantee that a program does not bad things, you're looking for a sandbox. Compile your code to wasm, or run it in a VM, or something, and then even if you have memory corruption or data races, it can't escape the sandbox and read the rest of your files. Or if you want to 100% guarantee that your program has no internal bugs, write it in Coq or ATS or something. Rust has a specific goal, and it does not involve being either a sandbox or a theorem prover.
AFAIK, it is actually possible to prevent /proc/$pid/mem from being updated from an individual process, and Rust's runtime could in fact ensure this on startup, at least on Linux. I was also pretty sure (but would have to look this up--it's a really crappy interface) that in general other processes cannot invoke this interface under ordinary privileges, which I think is a fairly reasonable compromise in terms of safety.
Not really sure what the point of the rest of your post is. I'm quite confident I have never argued that libraries should not use unsafe, or that Rust should prevent all bugs. I do think that Rust's standard library should not expose tools that can lead directly to memory unsafety, in the same process, when only safe code is used, under some "reasonable" model of the environment in which it runs (which might include, for example, a working MMU, and process isolation boundaries). That seems like a pretty reasonable standard for a language calling itself safe, and even if we can't quite reach it, we should strive for it. I could even buy the argument that exposing an interface like /proc/$pid/mem/ is not "reasonable" and thus discount this unsafety (well, I could if it weren't the default on all Linux distributions I'm aware of).
To instead use cases like this as an argument for "pragmatically" giving up memory safety in situations where it isn't actually forced upon us by the underlying execution environment, isn't at all convincing to me.
- Use the debugger interface (ptrace on Linux, but most OSes have some form of debugger). Should Rust programs block themselves from being debugged, iTunes-style?
- By editing the binary files on disk that are mapped into the process space. Should Rust binaries refuse to run if they are writable by the current user?
- By loading a kernel module. Should Rust binaries drop CAP_SYS_MODULE from the bounding set, just in case someone wants to load kernel modules from a subprocess?
And so forth. Now, yes, these cases are getting increasingly silly, but I'm having trouble figuring out what the precise distinction is between "It is bad to release a crate that permits you, in safe code, to call a function that might be buggy" and "It is bad to release a crate that permits you, in safe code, to run SSH, because you could SSH to your own hypervisor and modify the current process's memory."
The best distinction I can think of is that a programmer who thinks they are writing code to do task X should not unintentionally also be writing code to cause problem Y. This is the fundamental problem with unsafe interfaces - it's not that, say, gets() doesn't work, it's that it's literally impossible to use gets() without also allowing whoever's providing input to overflow a buffer. If you set out to modify your own memory, well, you're intending to do so, so I don't think that the safe/unsafe distinction makes much sense there. I suspect an argument could be made that AutoCXX's bindings should be marked unsafe, but I think that argument needs to be more complicated than "producing any sort of automated safe bindings is equivalent to the halting problem".
> Rust's standard library should not expose tools that can lead directly to memory unsafety when only safe code is used, under some "reasonable" model of the environment in which it runs (which might include, for example, a working MMU, and process isolation boundaries)
I do agree with this, I think - I just think that "directly" / "reasonable" excludes stuff like /proc/self/mem.
The idea that unsafe is supposed to only cover "deliberate" invocations is kind of silly, IMO. That is not what unsafe means. A function called "get_unchecked_index_this_requires_a_bounds_check_be_careful" still needs to be marked unsafe. But even if somehow this distinction could be made meaningful, this turns an accidental bug that lets you manipulate the file path into full blown arbitrary code execution.
(note: the bug was proposed to review by the libs team, but this review never happened, some prominent community members claim there that this is indeed a soundness bug, just one that might not be worth fixing, or would be really hard to fix, but that should be tracked as such).
To me it looks like one should just link the bug to the unsafe code guidelines, so that they can clarify the definition of soundness (and the limits of the rust abstract machine in which soundness is meaningful) to take this into account.
See the linked "mitosis crate" issue at the end for a similar issue.
E.g. "This code and its transitive dependencies has been carefully validated to be correct", "I had to do this to make it compile", and "This is a FFI that is inherently potentially unsafe".
Then the unsafty of any function would be defined as the logical OR of it and its dependencies... and you could filter what you're willing to use -- maybe with exceptions programmed for specific unsafe blocks, so you could e.g. except specific FFI interfaces without taking random ones people sneak in while they're adding malware to deep dependencies in crates.io to get silently embedded into your software.
(which is great, when the topic is a little complicated like this)
It needs to ensure that whatever preconditions those libraries have which are not reflected in their API because the languages they use don't allow for it are never broken. So let's say a libgit function takes a pointer (for an array) and an index, the rust bindings must ensure that the pointer is valid and the index is within the array.
Will there be bugs and things which will be missed? Likely, after all we've seen that in pure-rust unsafe code, including the standard library.
But the library "can't" just yolo and expose the entire thing as-is through a safe interface. As in technically it can do that just fine, but that's completely unsound even if it's effectively never called incorrectly.
I agree that the program should not automatically generate "safe" bindings that take pointers, because in Rust, creating and passing around raw pointers is safe but actually using them is safe. But if, let's say, you have an API that consists entirely of integers, bools, std::strings, and structs and classes thereof, what additional things would you need to check to be confident calling the binding safe?
(Sure, there are weird cases here like "this function takes a long, casts it to a pointer, and dereferences it," but I assume those are uncommon enough that you'll see if you're about to create or use auto-generated bindings to such a function. I suppose there could be "this function takes an std::string and an index, and the index must be less than the length or it's UB" - are those common enough that they make this endeavor questionable?)
My understanding is that this is the idea behind cxx: high-level C++ types are considered "safe", and so functions which use these are assumed to be safe.
Whether this is sufficient is the debate you see upthread.
> But if, let's say, you have an API that consists entirely of integers, bools, std::strings, and structs and classes thereof, what additional things would you need to check to be confident calling the binding safe?
The entire codebase, because even with no parameters at all, C++ can do thing which Rust considers wildly unsafe. Like manipulating global state, possibly across functions (there are entire libraries which work on that principle).
Put an other way, a "safe rust" function is reentrant, thread-safe and does not cause UB given any input.
By this definition, it is not possible to know that a C(++) function is safe without going through its source with a microscope.
(I mean, it would be really cool if we rigorously audited and fuzzed all these C libraries, so I'm not completely opposed to this!)