Comment by emidoots | Hacker News Reader

Comment by emidoots | Hacker News Reader

Parent Full thread

In fact, Zig has pretty much the same support for strings as both Rust and Go. The primary difference which people seem to complain about, that "[]const u8" is not written as "string", can be solved by writing `const string = []const u8;` at the top of your program if you like.

Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior, but any function called on a string slice may assume that it is valid UTF-8, which means that a non-UTF-8 string slice can lead to undefined behavior down the road.' - also `String` and `OsString`.

edit: apparently Rust enforces that &str cannot be created with invalid UTF8, at the additional performance cost of runtime checks, so it has stronger guarantees than other languages like Go/Java/Zig/etc (as usual)

Go: `string`, 'in effect a read-only slice of bytes', 'a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format. As far as the content of a string is concerned, it is exactly equivalent to a slice of bytes.'

Zig: `[]const u8`, 'some bytes, and a length', 'Zig has no concept of strings', 'by convention parameters that are "strings" are expected to be UTF-8 encoded slices of u8.', 'Generally, you can use UTF-8 and not worry about whether something is a string.' - and the stdlib has functions for working with unicode and strings.

> 'Constructing a non-UTF-8 string slice is not immediate undefined behavior

Rust’s string constructors (for both str and String) check that the data contains strictly valid UTF8.

You can opt out of this check in performance sensitive code - but doing so is considered unsafe. In safe rust it’s impossible to construct a str / String which contains invalid utf8.

Eg: https://doc.rust-lang.org/std/str/fn.from_utf8.html

good to know! I just copied that quote directly from the Rust docs[0] which were not clear it is only possible in unsafe code.

[0] https://doc.rust-lang.org/std/primitive.str.html#invariant

You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8. Doing that violates assumptions some optimizations rely on. I even would contend it's undefined behavior. At least I am convinced that you should never have invalid UTF-8 in `&str` or `String`. If you need that, there's the [bstr][0] crate for byte strings which may or may not be valid UTF-8. Essentially these are just byte slices `&[u8]` and the owned version byte vectors `Vec<u8>` and traits and methods around them. Very useful for text data which is not reliably UTF-8.

[0]: https://crates.io/crates/bstr

The idea of using unsafety here is to have a way to tell the Rust compiler: "Trust me, I promise that I only fill the strings with valid UTF-8 data. You don't need to insert code to check UTF-8 validity." And the Rust compiler answers: "Okay. I accept that but if something goes wrong I can't catch that."

This allows avoiding repeated validations of a byte slice for UTF-8 validity, for example.

It changed, it used to be immediately considered UB, but isn’t any longer, as your grandparent says, that’s what is documented now. It still leads to UB, but isn’t inherently UB on its own. There’s a term for this but I’m forgetting at the moment and search is failing me.

I agree that bstr is what you want unless you specifically understand this detail and know why it is the way it is.

The term is "library UB" versus "language UB". To be clear, you should still avoid it most of the times.

Ah yes, thank you.

Would you say library undefined behavior is just a violation of a library invariant? Like `&str` and `String` containing valid UTF-8?

I admit that a library invariant is allowed to be violated temporarily under some strict preconditions. One example are bank account money transfers, where the credit and debit don't happen exactly simultaneously. Something similar could happen in a library.

But for language UB this doesn't work. Why? Is it because of compile-time optimizations possibly deleting code restoring the invariant?

> You shouldn't misuse unsafe Rust to create `&str` or `String` with invalid UTF-8.

Sorry - I think my comment wasn't clear. The point of from_utf8_unchecked isn't to create strings with invalid UTF8. The point is that if you know that your string has already been validated, you can skip re-validating it.

String is utf8, OsString is not (necessarily). Having two separate types mean you can guarantee utf8 when you want to, but don't have to when you can't, such as when dealing with “strings” that an OS produces, which are basically just sequences of bytes (perhaps with zero bytes disallowed). When you have one string type that has to do both jobs, you lose out on the guarantees of the more restrictive type.

I am definitely not a rust expert, but as far as I can tell this is just not true? The quote you copied is from the documentation is in the context of creating a string in an unsafe manner. If you're going to include unsafe behavior, you may as well say that nothing in rust means anything at all.

> Rust: `&str`, 'some bytes, and a length', 'Constructing a non-UTF-8 string slice is not immediate undefined behavior

If you're constructing invalid `str` you already have severely fucked up. And dog is eating your HDD.

> If you're constructing invalid `str` you already have severely fucked up.

Presumably it'd be better to know this immediately on initialization of the variable rather than at some point in the program that actually expects valid utf8—god forbid you transmit or persist the data before this happens.

That's why str::from_utf8 returns Result<&str, FromUtf8Error>. And if you know the string is UTF-8 encoded and don't want to pay the cost, there's `from_utf8_unchecked`, which is marked unsafe.

Ah, I thought you were referring to the benefits of such a system as compared to zig and blithely passing the responsibility of input validation on to the programmer. Rust of course demonstrates the benefit of checking validity at initialization.

> and the stdlib has functions for working with unicode and strings.

Last time I looked I don't think it did. It looks like there is a unicode namespace now, but it is very primative. It doesn't have a simple way to iterate over codepoints, much less graphemes. Nor does it have anything to inspect the category a codepoint belongs to.

`std.unicode` supports validation, codepoint iteration, conversion from UTF8 <-> other encodings, etc.

It doesn't have graphemes (neither does Go or Rust), or other advanced unicode support. Instead that is provided by ziglyph[0]

[0] https://codeberg.org/dude_the_builder/ziglyph

> also `String` and `OsString`

Because these are different types with different semantics.

Why should there be one string type? Languages have to interface with the real world, where string data could be anything, but programs still need to be written for the common cases... hence, &str and String.

Reply on news.ycombinator.com