A surprising enum size optimization in the Rust compiler
jpfennell.com
jpfennell.com
If you manage to construct an invalid char from an invalid string or any other way, you can defeat the niche optimization code and accidentally create yourself an unsound transmute, which is game over for soundness.
What does "undefined behavior" mean without a spec? Wouldn't the behavior rustc produces today be de-facto defined behavior? It seems like the contention is violating some transmute constraint, but does this not result in reproducible runtime behavior? In what context are you framing "soundness"?
EDIT: I'm honestly befuddled why anyone would downvote this. I certainly don't think this is detracting from the conversation at all—how can you understand the semantics of the above comment without understanding what the intended meaning of "undefined behavior" or "soundness" is?
While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).
The compiler may exploit "lack of UB" for optimizations, e.g., using a known-invalid value as a niche, optimizing away safety checks, etc.
> Wouldn't the behavior rustc produces today be de-facto defined behavior?
Absolutely not. Bugs are fixed and the behaviour changes. Not often, but it happens.
This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html
> While not as formalized as C/C++, Rust's "spec" exists in the reference, nomicon, RFCs and documentation. I believe that there is a desire for a spec, but enough resources exist that the community can continue without one with no major negative side-effects (unless you want to re-implement the compiler from scratch, I suppose).
Thank you, I was unaware that this is a thing.
> This post probably answers a lot of your reply as well: https://jacko.io/safety_and_soundness.html
This appears to also rely on "undefined behavior" as a meaningful term.
I assure you it is a meaningful term:
Consider a hypothetical non-LLVM full reimplementation of the compiler. If it optimizes and there are invalid assumptions then there is likely UB. LLVM isn't involved in that case though.
It's the distinguishing from bugs that concerns me.
No, not at all. UB can still produce correct and expected results for the entire input domain.
Just like segfault or logic bug, it’s a class of bugs. Why is special though is that in most bugs you just hit an invalid state. In UB you can end up executing code that never existed or not executing code that does exist. Or any number of other things can happen because the compiler applies an optimization assuming a runtime state you promised it would never occur but did.
It’s slightly different from being a strict subset because UB is actually exploited to perform optimizations - UB is not allowed so the compiler is able to emit more efficient code is taught to exploit that and the language allows for it (eg the niche optimization the blog describes)
The ferroscene language spec was recently donated to the rust foundation.
[1] https://doc.rust-lang.org/reference/behavior-considered-unde... [2] https://spec.ferrocene.dev/undefined-behavior.html
You don't need a full language spec to declare something UB. And, arguably, from the compiler correctness perspective, there is no fundamental difference between walls of prose in the C/C++ "spec" and the "informal spec" currently used by Rust. (Well, there is the CompCert exception, but it's quite far from the mainstream compilers in many regards)
Incorrect with respect to an assumption furnished where? Your sibling comment mentions RFCs—is this behavior tied to some kind of documented expectation?
> A simpler example is `Option<NonZeroU8>`, the compiler assumes that `NonZeroU8` can never contain 0, thus it can use 0 as value for `None`. Now, if you take a reference to the inner `NonZeroU8` stored in `Some` and write 0 to it, you changed `Some` to `None`, while other optimizations may rely on the assumption that references to the content of `Some` can not flip the enum variant to `None`.
That seems to be the intended behavior, unless I'm reading incorrectly. Why else would you write a 0 to it? Also, does this not require using the `unsafe` keyword? So is tricking the compiler into producing the behavior you described not the expected and intended behavior?
In the definition of the `NonZeroU8` type. Or in a more practical terms, in LLVM, when we generate LLVM IR we communicate this property to LLVM and it in turn uses it to apply optimizations to our code.
>Also, does this not require using the `unsafe` keyword?
Yes, it requires `unsafe` and the point is that writing 0 to `NonZeroU8` is UB since it breaks the locality principle critical for correctness of optimizations. Applying just one incorrect (because of the broken assumption) optimization together with numerous other (correct) optimizations can easily lead to very surprising results, which are practically impossible to predict and debug. This is why it's considered such anathema to have UB in code, since having UB in one place may completely break code somewhere far away.
It might be what that programmer intended and expected, but they should not expect it. E.g. the current compiler might check for 0, and a future more optimized compiler might optimize out that check (because it knows the Option is not None) and then e.g. perform an out-of-bounds array access (if you were using that NonZeroU8 as an index into some kind of 1-based array).
Whether something is a bug or not is sometimes hard to pin down because there's no formal spec. Most of the time it's pretty clear though. Most software doesn't have a formal spec and manages to categorize bugs anyway.
This is even more vague. The language is getting blamed regardless. This makes no sense.
To try to characterise what any compiler, hypothetical or not, does if you nonetheless produce one (again, via means that aren't valid) isn't meaningful.
I think there's two parts to this. First, there's a bit of a history of people making disingenious jabs at Rust for not having an "ISO C++" style spec. Typically people would try to suggest that Rust can't be ready for production or shouldn't receive support in other ecosystems without being certified by some manner of international committee. Second, Rust by now has an extensive tradition of people discussing memory safety invariants, what soundness means, formal models of what is a valid memory access, desirable optimizations, etc, etc, so your question what undefined behavior means could be taken to be, like, polemically reductive or dismissive.
In context I don't think it's what you're doing, but I would also not be surprised if a lot of people reading Rust-related HN discussions are just super tired of anything that even slightly looks like an effort to re-litigate undefined behavior from first principles, because it tends to derail more specific discussions.
If you break an invariant the compiler is relying on for optimization then you can't say for sure what the effect after all optimisation passes or in future versions of the compiler will be. It's just "undefined"
How are you supposed to be specific about what the possible damage might entail for corrupted memory? If you have a function with an "if" or a "while" or a "switch" in it, and you break the variable being evaluated, you might cause the program to skip over the choices and run whatever happens to be next in memory. What's the non-lazy listing of possible outcomes at that point?
Rust is still lacking a definitive formal model of "soundness" in unsafe code. I'm not sure why you're suggesting that this is not a valid criticism or remark, it's just a fact.
Note that there are non-zero integer types that can also be used in this way, like NonZeroU8 https://doc.rust-lang.org/std/num/type.NonZeroU8.html. The NULL pointer is also used as a niche, and you can create your own as well, as documented in https://www.0xatticus.com/posts/understanding_rust_niche/
(And I think for much the same reason, the niche optimization. Option<bool> is 1 B.)
(And for the non-Rustaceans, the only way to get a bool to be not false or true, i.e., not 0 or 1, would be unsafe {} code. Or put differently, not having a bool be "2" is an invariant unsafe code must not violate. (IIRC, at all times, even in unsafe code.))
But yes, there are the NonZero integers, and you can make your own NonBlah integer using the "XOR trick" for a relatively tiny performance overhead, as well as you can make enums which is how the current CompactString works.
The link you gave mentions that Rust does this for other types, but in particular OwnedFd is often useful on Unix systems. Option<OwnedFd> has the same implementation as a C file descriptor, but the same ergonomics as a fancy high level data structure, that's the sort of optimisation we're here for.
Alas the Windows equivalent can't do this because different parts of Microsoft use all zeroes and -1 to mean different things, so both are potentially valid.
There is no generic way to re-validate structs in a bounded address space. You'd need something akin to a garbage collector that traces references at fixed offsets including type knowledge. This is not completely infeasible since Rust has a lot of information at compile time to avoid checks, but the extreme cases where people are writing to complicated graph like structures inside unsafe {} can realistically only be dealt with through tracing all safe references that lie inside the bounded address space.
In practice it will also be a struggle to sandbox C code into a small enough CHERI style address space so that you don't have to check literally your entire computer's memory after an FFI call.
It's not the enums that are the problem. unsafe can break anything if you are determined enough.
Edit: In retrospect, the optimization doesn't actually cause any security or safety problems, because unsafe code can break any invariant, including an enum with a separated tag and value. The particular memory layout of the enum is irrelevant.
The author seems to have a limited understanding of niche optimization.
In any case, the definition of what is exactly niche optimization is besides the point. The point of the post is: the literature gives you the impression that there's one limited form of enum size optimization in the Rust compiler, but in fact there are other optimizations too. And knowing this is useful!
It's useful if you got that specific impression.
But I didn't. So I was disappointed and unenlightened because I was promised a non-niche optimization and I didn't get one.
What's cool about this in general is nested match statements can be flattened into a jumplist (idk if rustc does this, but it's possible in theory).
But in the case of Options they tend to be both write-once and short-lived, so that removes a lot of necessity. Options are going to stay mostly on the stack and in async callbacks, unless they go into caches.
But for other data structures where multiples fields need a tag, I suspect Rust could use some bitfields for representing them. You’d need a fairly big win to make it worth implementing however.
CPUs nowadays support double the largest general-purpose register width. Unofficially, some CPUs can also do twice that: https://rigtorp.se/isatomic/
Yes, Rust suppresses the niche optimization for values wrapped in an `UnsafeCell` (which is how you signal to the compiler that “atomically writing two values at once” might happen). https://github.com/rust-lang/rust/pull/68491
It is. One easy way to see this is with an Option<Option<bool>> [0]: if both options are Some, it takes the value 0 or 1 depending on the boolean; if the inner Option is None, it takes the value 2; and if the outer Option is None, it takes the value 3. And as you keep adding more Options, they take values 4, 5, 6, etc. to represent None, while still only taking up 1 byte.
[0] https://play.rust-lang.org/?version=stable&mode=debug&editio...
(Granted, in the None variant, the byte used for the u8 is not usable, but if we're already using a separate discriminant byte, 256 variants should be plenty.)
The mental model is, an enum payload will have some number of integer/pointer values with niches in their representation. Niches don't work by counting bits, they work as numeric ranges. E.g., a char is just a u32 with values 0 through 0x10ffff, a bool is a u8 with values 0 and 1, a reference is just a pointer with any value except 0, etc., and the niche is precisely the negation of this range.
Sometimes the niche corresponds to bits used in a valid value (e.g., the 0 value of a NonZeroU8), and sometimes it corresponds to other bits (e.g., values 2 through 255 of a bool): the compiler only cares about the ranges, not the bits. If there is no large-enough niche, then the discriminant is placed in a separate byte.
An outer enum can't use sometimes-valid payloads in an inner enum to represent its discriminant, if that's what you're trying to say. Multiple discriminants can be 'flattened' into a single continuous range of niche values, but they can't be 'flattened' into inner enums' payloads. That would cause a weird inversion where you need to read the inner discriminant just to know whether the outer discriminant is valid.
(The compiler does have a few tricks up its sleeve to make the most of niches. E.g., in a Result<(u8, bool), u8>, an Err(42) becomes [42, 2], but in a Result<(bool, u8), u8>, an Err(42) becomes [2, 42] (https://play.rust-lang.org/?version=stable&mode=debug&editio...). The 42 is repositioned to keep the niche intact.)
- Some(Some(x)) for x in [0, 255]
- Some(None)
- None
No matter how you slice it, you can't fit them all into 8 bits, hence why it needs an extra byte.
https://rust-lang.zulipchat.com/#narrow/channel/131828-t-com...
The "but do it automatically" part seems rather problematic. Requiring global analysis is a big ask indeed.
Say, if you change a variant's discriminant value, does that count as a semver breaking change? Probably. Would it be an ABI breaking change? Probably.
Also, how do I get some code to do the memory layout vizualizer, perhaps one that is a bit more advanced and knows what pointers are?
One thing it can't use is padding in structs, because references to individual fields must remain valid, and they don't guarantee that padding will be preserved.