Hacker News Reader

Comment by josephg | Hacker News Reader

> 'Constructing a non-UTF-8 string slice is not immediate undefined behavior

Rust’s string constructors (for both str and String) check that the data contains strictly valid UTF8.

You can opt out of this check in performance sensitive code - but doing so is considered unsafe. In safe rust it’s impossible to construct a str / String which contains invalid utf8.

Eg: https://doc.rust-lang.org/std/str/fn.from_utf8.html

View on HN

You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8.

[0]: https://crates.io/crates/bstr

The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that."

This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.

It changed, it used to be immediately considered UB, but isn’t any longer, as your grandparent says, that’s what is documented now. It still leads to UB, but isn’t inherently UB on its own. There’s a term for this but I’m forgetting at the moment and search is failing me.

I agree that bstr is what you want unless you specifically understand this detail and know why it is the way it is.

The term is "library UB" versus "language UB". To be clear, you should still avoid it most of the times.

Ah yes, thank you.

Would you say library undefined behavior is just a violation of a library invariant? Like `&str` and `String` containing valid UTF-8?

I admit that a library invariant is allowed to be violated temporarily under some strict preconditions. One example are bank account money transfers, where the credit and debit don't happen exactly simultaneously. Something similar could happen in a library.

But for language UB this doesn't work. Why? Is it because of compile-time optimizations possibly deleting code restoring the invariant?

> You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8.

Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.

good to know! I just copied that quote directly from the Rust docs[0] which were not clear it is only possible in unsafe code.

[0] https://doc.rust-lang.org/std/primitive.str.html#invariant

Reply on news.ycombinator.com