Undefined vs. Unsafe in Rust | Hacker News Reader

Undefined vs. Unsafe in Rust

manishearth.github.io

160 pointsingve47 comments

Unsafe code might result in undefined behaviour. Code in an `unsafe` block, however, is code that is not compiler checked to be safe, but hopefully was hand checked to be safe.

The `unsafeXYZ` idiom is very common in the functional programming world. For individual functions, prefixing them with `unsafe` generally means, this function could result in undefined behaviour. However, there is the other idea that we mark something as `unsafe` to ask the compiler not to check it and just to trust us (`unsafePerformIO` is often used this way). Maybe having a second term would make this clearer.

It is in fact checked by both the compiler and the borrow checker. The only difference between safe and unsafe code is that the latter allows dereference of pointers and direct manipulation of memory, which may result in segfaults, memory leaks and data races.

The Rust compiler specifically aims to make it impossible to have segfaults, memory leaks, and data races — so to say unsafe code “is in fact checked” is at least a little misleading. The user still needs to do additional checking for memory errors in unsafe code.

Rust doesn't aim to make it impossible to have memory leaks: https://users.rust-lang.org/t/memory-leaks-are-memory-safe/5...

It at least eliminates what I'd estimate to be the most common cause of memory leaks (in C): missing a free. Which is the same guarantee that GCs provide.

The story is similar to C++. Almost all allocations are done by types that'll automatically call free in their destructors. But because there's no global GC, it's still possible to leak accidentally by creating a cycle in reference-counted smart pointers.

There is still the other most common source of memory leaks: putting data into long living (e.g. 'static) hash tables and having no plan for pruning it.

Maybe there is some checking but there is clearly somewhat less. This is in keeping with the general idiom (shared with Haskell and Swift, among others).

To be clear: unsafe Rust is a superset Of safe Rust. Every single safety check is still turned on in an unsafe block.

However, that superset contains things that are not checked. This has no effect on any other code though; safe code with a superfluous unsafe block acts 100% exactly the same.

One nasty class of errors in C and C++ is due to environment mismatches in separate compilation. For example, a struct may conditionally add a field according to some compilation flag, say, whether ASAN is enabled. If compilation units disagree on that flag, they can disagree on the size of the struct, leading to UB.

Does Rust have a way to prevent this class of errors? What ensures that all units have the same view of a type?

Multiple units don't have their own copies of type definitions to begin with. The compiler sees them all at once instead, to the point that different versions of a crate produce different incompatible types.

At a lower level, symbols are mangled to include a crate identification hash. This is more of a way to allow multiple versions to coexist though, as the problem is already solved by that point.

Thanks for the reply. What do you mean by "different versions of a crate?" Here's a scenario:

1. Compile a library libA that references type T in CrateX.

2. Add a field to T, and recompile CrateX.

3. Compile an executable a.out that passes a T to libA.

Nobody bumped CrateX's version or recompiled libA. Won't this crash? Or does this produce a "different version" of CrateX and so it will refuse to link?

It produces a crate with incompatible symbol names (a "different version"). There's a "strict version hash" derived by hashing a representation of the structure of a crate's entire external interface. That hash value is appended to every symbol name as part of the name mangling process.

Is that actually true anymore? I just tried compiling a simple crate:

    pub struct S {
        a: usize,
    }
    pub fn foo(s: &S) -> usize {
        s.a
    }

…as a dylib, then added another field to S and recompiled. I expected it to change the symbol names (as shown by nm), but it didn't. The metadata might be different (I don't know how to display it), but the dynamic linker doesn't know about that.

That changed with incremental recompilation, to allow reuse across changes.

However, Cargo will still ensure different versions of the same crate have different symbols, by passing its own hashes to rustc via -C metadata.

Ah here we go again! What does "different version" mean? Is it explicit version metadata or something computed from the interface?

Computed from the interface. The metadata does not understand semver or any of the higher level versioning tools Cargo exposes to users.

The metadata just contains info on all the types, and their hashes (or something like that), so if stuff doesn't match you'll know.

This is generally visible from the "expected type Foo but found type Foo" error, which will often mention you have two versions of the same crate.

Worth mentioning that unlike C++ or C Rust doesn't have a global name mangling scheme; so it is totally ok for two crates to have a toplevel struct Foo (unlike in C++ where you are forced to namespace them with uniquely-named namespaces). This has the side effect of it being totally ok to link two versions of the same crate together; and Rust will just complain if you try to mix the types.

Are you sure the metadata hash that Cargo computes (and passes with -C metadata) is based on the AST? I don’t think Cargo tries to parse source files…

No, that's a metadata hash it creates (probably from semver info, but also perhaps file contents) so that it can ask rustc to mangle symbol names.

No file contents, that would defeat incremental recompilation.

You sure you meant to reply to the follow-up to my comment? None of the hashes you mentioned is actually used by the compiler, except to detect changes wrt incremental recompilation. (EDIT: and to catch recompiled crates, see: https://news.ycombinator.com/item?id=16004250)

Distinguishing between crates is done solely through their name and -C metadata values provided by Cargo.

Once crates are loaded by the compiler based on either their explicit path (via --extern) or by being a dependency of another dependency (and there the name and -C metadata prevent collisions), "two versions of the same crate" appears no different than "two different crates".

The Rust compiler tends to "index" information (e.g. turning strings into various IDs) as quickly as possible, so a lot more semantics are "by identity" than "by syntax", and that helps when multiple identities may share a name.

That includes compiling against already compiled crates, instead of header files you have serialized semantic types and functions, which all use proper identities to "name" anything they use in turn - you never have to be looking for a definition, or risk using the wrong one.

In that specific context, crates that have the same name within a Cargo crate graph but aren't the same exact crate - this is pretty much always about semver versions, when incompatible ones are required from different parts of the crate graph, you can end up with e.g. serde-1.0 and serde-0.9, and they get compiled with different -C metadata values, into which "1.0" and "0.9" were factored in.

Okay I just properly took a look at this, and some of the sibling responses are only partially correct. It's not symbol names that change, but the crate does appear to have a hash used to sanity-check against this.

I used these sources:

    // a.rs
    #![crate_type = "lib"]
    pub struct Foo { pub x: i32 }

    // b.rs
    #![crate_type = "lib"]
    extern crate a;
    pub fn foo() -> a::Foo { a::Foo {} }

    // c.rs
    extern crate b;
    fn main() {
        println!("{}", b::foo().x);
    }

The field `x` from crate `a` was originally not there, and added after compiling crate `b`. The error when trying to compile crate `c` was:

    error[E0460]: found possibly newer version of crate `a` which `b` depends on
     --> c.rs:1:1
      |
    1 | extern crate b;
      | ^^^^^^^^^^^^^^^
      |
      = note: perhaps that crate needs to be recompiled?
      = note: crate `a` path #1: liba.rlib
      = note: crate `b` path #1: libb.rlib

If you interface with C, then Rust doesn't have anything to prevent this class of errors. If you interface Rust with Rust, then Rust weasels out of solving this problem by declaring it doesn't have a stable ABI :)

In practice you compile everything with Cargo and it ensures all affected crates are recompiled when interfaces or settings change, so everything matches.

it kind of sucks that rust doesn't provide an easier way for Julia to solve her problem and she has to resort to unsafe. it sounds like she just wants to cast some bytes to a structure. if rust had c-style serialization from bytes->structure then this could be done without any unsafe [assuming all the c types were rust 'safe' which they should be]. but i guess this is a pretty rare use case.

i guess the reason behind using the c-structs is to avoid manually calculating the offsets for N different ruby versions you want to support. i guess the alternative to casting/serialization would be to extract the offsets from the generated rust bindings which is apparently unsafe in rust as well. heh

Casting bytes to a structure is tricky! First of all, unlike C, Rust doesn't define struct layout. We change the internal layout every so often to add optimizations, and we can only do this thanks to that guarantee. We do have a way to say "use C's layout", if it's a struct you're defining. Also, you have things like endianness to worry about, portability is tough here!

Depending on your sitaution, you don't have to use unsafe yourself; for example, the byteorder crate can help here, or any of the serialization libraries that can serialize to a binary format.

Tl;dr;

Code in an unsafe block may generate undefined behavior if it is used incorrectly, but, if used correctly, will not.

Rust code which is not in an unsafe block cannot generate undefined behavior (aside from bugs in the compiler/underlying libraries).

Your tl;dr is wrong.

> Code in an unsafe block may generate undefined behavior if it is used incorrectly, but, if used correctly, will not.

The point of this was that code in an unsafe block should not be able to generate undefined behavior no matter how it is used from safe code, otherwise that unsafe block is unsafe, not safe.

> Rust code which is not in an unsafe block cannot generate undefined behavior

That's false, code outside of an unsafe block can generate undefined behavior by calling unsafe code that was not written safely.

> tl;dr

Please don't post misleading summaries on nuanced topics.. especially when the original post is short.

> That's false, code outside of an unsafe block can generate undefined behavior by calling unsafe code that was not written safely.

Is there any way to write safe code in Rust then?

Yes.

To guarantee complete rust-safety, modulo LLVM bugs, do not use any Rust code that has unsafe blocks in it (this includes any code that has memory allocations, such as Vec).

Less strictly one can refuse to use code with unsafe blocks that has not been thoroughly vetted by experts. Things like Vec may therefore be used depending on how strict one's definition of "vetted by experts" is, but rolling one's own "unsafe" block code is not allowed.

I write rust-safe code all the time using the second standard, even though under the covers it's littered with unsafe.

Thanks for all the replies!

So, while it's possible to manually check the use of unsafe, there is no way I can currently do it automatically, right? I noticed there is a #![forbid(unsafe_code)], which I can add to my source file, but this does not check that whatever functions I am importing or calling do not underlyingly use unsafe, right?

Moreover, I admit I am being lazy here, but any idea whether there are sub-communities in Rust that are keen on creating these kinds of safe programs?

At some point, in the context of developing on commodity hardware using mainstream operating systems, you need to trust something and use unsafe. There's no way around it.

People have varying tolerances on the use of unsafe code. I'm not aware of any coherent sub-community with a fundamentalist take on it though.

My own personal tolerance is "when using it, carefully justify it." Generally, this takes the form of making the code run faster. Otherwise, I'll sacrifice almost everything else to avoid unsafe. Thankfully, said sacrifices are neither common nor arduous (in my experience).

To the GP, also note that there is a distinction between what Rust is able to confirm is safe with the compiler and what is or is not necessarily safe. An unsafe block carefully vetted to be implemented correctly with a formally proven safe algorithm may still be safe in the general sense, if not save in the Rust compiler enforced sense. Indeed, that's one of the major reasons for using unsafe blocks, so foregoing them entirely may be unduly hamstringing yourself for no reason.

This tldr-counter-tldr discussion is circling around this subtle issue: Unsafe code has a "contaminating" effect on the module it's in. For example, consider the length of a Vec. That's just an integer. Any method on Vec could change that integer without needing an unsafe block. However, increasing the length incorrectly (for example, past the allocated capacity) will totally cause UB, because unsafe code in other methods of Vec assumes the length is correct. Vec is able to expose a safe API, because the length is a private member, and callers can't set it willy-nilly. But code inside the Vec module needs to be very careful, even when it's not explicitly using unsafe blocks.

Is it possibly to model that efficiently with a total programming language - or do we need a turing-complete language to write a feature-rich vector (or array, matrix, `Vector a N`, etc.) library?

I sometimes wonder if we should maybe invest more in non-turing-complete language research. Obviously Haskell (and Rust) are improvements to the status-quo (critical code can be guarded explicitely via unsafe attributes and written by experts), but maybe this does not go far enough.

I'm not sure if this is directly related to what you're thinking about or not, but I think some of Rust's stdlib has been formally verified. It might be that more work in formal verification can get us the guarantees we want, without needing to change the language itself.

This is well-said in the article:

If the 'unsafe' blocks contain their unsafety correctly (thus being 'safe' 'unsafe' blocks using the two different meanings of safe the article discusses), then using them is fine. The rust stdlib has a promise that all of their 'unsafe' blocks not marked as unsafe are 'safe', so using the stdlib you're writing 'safe' code (though it's possible it's unsafe if there are stdlib/compiler bugs).

As the article explains, it's very nounced and, while imperfect, still a damn sight better than C++.

This kinda misses the point; this is correct, but not what the article is about. The article is about the concept of "unsafe" (distinct from "code in an unsafe block") which is "cannot generate UB no matter how you use it"

This all seems rather obvious.

It all depends on the audience. To someone that's been following rust, it's rather obvious (even if the distinction in meanings of unsafe may be a bit clearer), but for someone that only vaguely knows that it "allows you write safe code" it may have been very useful in quickly getting to the crux of the matter.

Rust actually has undefined behavior that is triggered by code outside of unsafe blocks, depending on how a result is used:

https://github.com/rust-lang/rust/issues/33813

This basically exposes LLVM's poison semantics (a kind of deferred UB) to Rust code:

https://llvm.org/docs/LangRef.html#poisonvalues

Code in another module that doesn't even know it is calling an unsafe function could trigger UB by using this.

Can you provide a specific example or playground link?

It seems like you'd still have to dereference it, which is unsafe, yes?

If you read the linked LLVM documentation, you'll see that any LLVM instruction with side effects that depends on poison can trigger UB, e.g. a volatile store where the poison value is the stored value, not the address.

I haven't tried very hard to write a seemingly safe Rust program using ptr:offset that LLVM will optimize into a crash. It's probably possible with a bit of work; this is basically C's signed overflow UB in a different guise, and programmer misunderstanding of that has caused serious bugs and security issues over the years.

ptr::offset is unsafe.

I think you misinterpreted the issue report you linked, which goes along with what Manishearth was saying about there being two meanings of “unsafe”. In this case, ptr::offset is declared as an unsafe fn - meaning only unsafe code can call it - because of the UB issue you mentioned. The reporter was asking to remove “unsafe”, arguing that Rust generally allows safe code to create arbitrary raw pointer values (but not deference them). Which is true, but only because that’s designed not to cause UB. ptr::offset can cause UB, so it has to remain an unsafe fn - lest it become “unsafe” in the sense of “potentially dangerous when called from safe code” - and the report was closed.

Edit: There are some other known ways of causing UB from safe Rust code, but they’re considered bugs - and some longstanding ones are on the way to being fixed in the next few months, thanks to MIR borrow checking and saturating float->int casts.

Segfault:

    #[link_section = ".data"] fn main() { }

There are also ones which are considered "well, whatcanyado", like writing to /proc/self/mem.

(link_section might be counted as one of these. or just tweaking your link flags)

Reply on news.ycombinator.com