Hacker News Reader

Comment by CJefferson | Hacker News Reader

One thing I've often wondered about Rust strings. I often hear that &str is 'a string slice'. But, Rust has a notation for slices -- &[T]. Why are strings the only thing (that I know of) that don't use the same slice notation as everything else?

View on HN

So, there's like, a few things here. First is, technically they both do use the same notation, &T, where T=str and T=[u8]. This is the whole "unsized types" thing. &Path is another example of this, String : &str :: PathBuf : &Path.

Beyond that though, &[T] implies a slice of Ts, that is, multiple Ts in a row. But a &str is a slice of a single string. So &[str] would feel wrong; that is, a &str is a slice of a String or another &str or something else, but isn't like, a list of multiple things. It's String, not Vec<str>.

Basically, Strings are just weird.

In particular: strings aren’t actually simple arrays of characters in Rust like they are in C, but there is an underlying array on the heap, and the notion of “slicing” that array of characters still makes sense semantically.

> strings aren’t actually simple arrays of characters in Rust

Are you talking about them being a container with a pointer to the actual array, and also a size and etc?

char is usually some single byte of data. Characters can be multiple bytes. Slicing a string on character boundaries is more coarse grained than slicing on byte boundaries.

char in Rust is 32bits, so it has a 1:1 mapping to Unicode glyphs. You might also want to care about grapheme clusters, but those are not part of the stdlib.

Unicode codepoints; glyphs may take multiple codepoints.

You are, of course, correct.

Right - the point is that &str isn't syntactic sugar / alias / etc for &[u8] and it would be confusing to have a notation that suggested otherwise.

In C, strings can hold invalid unicode. However, in Rust a str is guaranteed to be valid utf-8.

For added confusion, Rust has a `char` type which is actually 32-bits. You can create arrays of them, but the resulting string would be in utf-32 and thus incompatible with the normal `str` type.

    let s = "hello world".to_string();
    for ch in s.chars() {
        print!({},ch);
    }

will iterate through a string character by character. That's the most common use of the "char" type - one at a a time, not arrays of them.

Although the proper grapheme form is:

    use unicode_segmentation::UnicodeSegmentation; // 1.7.1
    let s = "hello world".to_string();
    for gr in UnicodeSegmentation::graphemes(s.as_str(), true) {
        print!("{}", gr)
    }

This will handle accented characters and emoji modifiers. A line break in the middle of a grapheme will mess up output.

By the way, open season for proposing new emoji starts tomorrow.[1]

[1] http://blog.unicode.org/2020/09/emoji-150-submissions-re-ope...

Thanks for the code snippet, however (kind of a Rust newbie here)

  $ cargo install unicode_segmentation (chokes)

  $ cargo install unicode-segmentation (seems to work)

  and in Cargo.toml
  [dependencies]
  unicode-segmentation = "1.7.1" (seems to work)

  yet in the code, its:
  use unicode_segmentation::UnicodeSegmentation;

  Why couldn't they be consistent using a dash vs. an underscore ?

You don’t cargo install libraries; the cargo.toml line is the right thing.

You can’t use -s in Rust identifiers, so they need to be normalized to _ to be referred to in code.

Identifiers in Rust can't have dashes (rustc will never rely on whitespace for proper tokenization). It is arguable that because of that crates should not have dashes in their name. cargo was changed to automatically convert dashes to underscores for convenience because some dislike underscores in names. I personally think this is an unnecessary source of confusion and now I would make all tools (mainly cargo) treat them interchangeably.

> rustc will never rely on whitespace for proper tokenization

Huh? That seems clearly untrue, eg `fnfoo` vs `fn foo` or `x && y` vs `x & &y` or `x<<shift_or_type > ::foo` vs `x < <shift_or_type>::foo`? Presumably for some or all of those, one version ends up being a error (eg bitwise and with a pointer from `x & &y` probably doesn't work), but that's not at the level of tokenization.

Given the context, I think they simply mean that a-b is interpreted the same as a - b.

It can't be interpreted as a single identifier.

this is way too common in Python world too - packages (crates) use dashes (they look nicer?), but the packaged library is named with underscores, because `-` is math operation and can't be in a name

Slicing strings is a difficult concept when dealing with Unicode because slice indices and lengths can come in multiple flavors, but the only flavor that makes sense (grapheme or grapheme cluster) is difficult to count.

>strings aren’t actually simple arrays of characters in Rust like they are in C, but there is an underlying array on the heap

Not a Rust expert, but to my understanding, 'string' is an array of characters, not necessarily living in the heap.

The object 'String' (with capital S) might be, but with &str I can have a constant array of characters that is not in the heap and not even in the stack: it's in the code (.text code segment[1]?, or EEPROM or flash, for embedded folks).

If I understood it correctly, &str is a slice with a pointer to that piece of code. Of course being const, it cannot be changed. I can copy it into a String and manipulate it (append, insert, etc.), and that's where the heap is used.

&str, as an hypothetical struct, it may be in the stack, initialized in-scope with (probably) a pointer to somewhere in .text plus some bytes for length or other information.

[1] https://en.wikipedia.org/wiki/Code_segment

&str is not necessarily a static string literal; it's a slice of any string, living anywhere (including on the heap inside a String). Technically speaking, it can't be mutated because it's a non-mutable reference (like any other non-mutable reference). &mut str is a thing that can exist, and is mutable, it's just that you can't get one of those for a static (literal) string that lives in the code.

Thanks. Yes, of course it can point to something in the heap. My comment was related to GP's "but there is an underlying array on the heap" and I wanted to say that &str is not always necessarily in the heap. I missed that part.

Yea but i don't think that's what GP was asking, imo. Rather than `[str]`, i think they were asking why it's `str` and not `[char]`, no?

Just as `[u8]` is to `Vec<u8>`, `[char]` is hypothetically to `Vec<char>`.. and `Vec<char>` is basically a `String`, no?

_edit_: Though looking at the docs, `char` is 4 always bytes, so i guess that's where the breakdown would be? `char` would need to be unsized i guess, but then it would be an awkward `[unsized_char]`, which is like two unsized types.... hence `str` maybe?

`char` is a (internally) a `u32` because it represents any single unicode character. `str` is not a `[char]`, because rust doesn't store strings as utf-32 (system APIs don't accept utf-32, and it tends to waste space in many cases)

`str`'s data layout happens to be `[u8]`, but it's type provides additional guarantees about the structure of the data within it's internal `[u8]` (for example, forbidding sequences of u8 that don't encode valid utf-8).

To be extra pedantic, char represents a single Unicode Scalar Value.

This is a big deal because adding an accent mark to a letter (often) means a single char can no longer store it. APIs should not orient around isolated scalar values or codepoints because most devs will misuse them, not being experts on combining and normalization.

Well yea, i wasn't saying `[char]` _is_ a `str`, rather i was positing that the GP comment was asking why it's a `str` than some hypothetical `[unsized_char]`.

I think `char` _would_ work, if it was similarly unsized like a single piece of `str`. The Problem is .. as i see it, that `[unsized_char]` seems odd.

`str` is not `[char]`, there is no datatype `char` possible or which this would hold, and the name is already taken.

`str` is not a slice; this is itself already a wrong statement. A slice is a dynamically sized type, a region of memory that contains any nonnegative number of elements of another type.

A `str` is dynamically sized, but is not guaranteed to contain a succession of any particular type. It's simply a dynamically sized sequence of bytes guaranteed to be Utf8.

`strs` aren't slices; all they have in common with them is that they are both dynamically sized types.

`Vec<char>` is also not the same as `String`; a string is not a vector of `char`, which is already a type that has a size of 4 bytes.

This all results from that Utf8 is a variable width encoding, and since slices are homogeneous, all elements have the same size.

Yes, the reason you can't use char here is that a char is always 4 bytes, so a &[char] is a type that already exists, and that type uses four bytes per character.

I wish they had been named &path and Path instead, it would feel more consistent than String, &str, PathBuf, &Path

I joke the only way that I'll sign off on Rust 2.0 is if we get to do the exact opposite, rename String to StrBuf, haha! (Same reason though, for consistency.)

I'd actually love something like that, mostly because differentiating between "str" and "String" in, uh, speech can be pretty annoying. They both sound the same! :)

I've taken to calling "String" "owned string" but that's a little unwieldly and not quite suited for complete Rust beginners who don't know the concept of ownership yet. Same with "string reference" for "str".

Related question (if you don't mind another).

Why is 'str' a "primitive type"? What about 'str' means it has to be primitive, instead of being a light-weight wrapper around a '&[u8]' (that obviously enforces UTF8 requirements as approriate).

You can't just index into a string.

    let hello: &[u8] = "Héllo".as_u8_slice();
    // hello[1] != "é", I believe it would be the bottom half-bytes of "H".
    // hello[2] != "é", I believe it would be the top half-bytes of "é".

To use &[u8] would be very non-ergonomic.

One could, in theory, make an ExtendedGraphemeCluster type, and make str a slice of ExtendedGraphemeClusters. So &[ExtendedGraphemeCluster] could be indexed into without having things not make sense. Of course that's much more complicated than most other primitives, and most people don't have any idea what an Extended Grapheme Cluster even is. But since they're the Unicode notion that most naturally maps to a "character" you could just call the type Character or char, and confuse the hell out of the C programmers by having a variable-width char type.

Sure - but iterating by extended graphemes isn’t the only thing you want to do with strings. Sometimes you want to treat them as a bunch of UTF-8 bytes. Sometimes you want to iterate / index by Unicode code points (eg for CRDTs). And sometimes you want to render them, however the system fonts group them.

It makes sense to have a special type because it can supports all of this stuff through separate methods on the type. (Or through nearby APIs). It’s confusing, but I think it’s the right choice.

Although, I think the most confusing thing about rust strings isn’t that &str isn’t &[u8]. It’s that &str isn’t just &String or something like that.

Well, &String is ALSO a valid Rust type, it's a reference to a (heap-allocated) String.

Really, there's no single, good option that covers all use cases. Which is why Rust ended up with multiple string types, along with ways to convert between them.

> Well, &String is ALSO a valid Rust type

Yeah I know. This is really confusing when learning rust. Coming from another language where there's a single rust type its really tricky to figure out when to use what!

For reference, I now use String for large strings, string builders, and when the string is owned - like in a struct. And &str when taking in a string as an argument to a function. Or when iterating or something like that where you don't want to move the string. And then for small owned strings (like labels or IDs) I reach for the InlinableString crate. That handles small strings inline and puts large strings on the heap, like what Apple does for Swift and Objective-C.

I understand why its this way; but its a pity the best answer rust has is so complex and confusing.

It's worth noting that you can use &String anywhere that &str is accepted. The distinction is subtle and I agree it's sort of confusing, but in practice the former gets automatically coerced to the latter as needed

The other issue is that Rust pretty strongly expects slices to be laid out in memory as a flat array of some fixed-size element type. If something isn’t laid out that way, it can’t support all the operations a slice is supposed to support, ranging from the high-level operation of O(1) indexing, to the low-level operation of getting a raw pointer to the slice contents. In principle you could make &[ExtendedGraphemeCluster] special, but it would hamper the ability to write generic code that works on &[T] for any T. Besides, Rust likes to be explicit. Things that are different should look different.

In addition, even if you gave up on O(1) indexing by grapheme cluster index, anything faster than O(n) indexing would require some sort of tree data structure to translate grapheme cluster indices to byte indices. Such a structure would be expensive to maintain and not needed the vast majority of the time; in a language that prioritizes performance, it simply couldn’t be part of the default string type no matter what the syntax.

The bstr crate is designed to make handling of ASCII-compatible (but not necessarily valid UTF-8) encodings a bit more ergonomic: https://docs.rs/bstr

Well, &str is the type of string literals, so on some level, it has to be special.

There was a PR implementing it as a wrapper around &[u8], but it didn't really provide any actual advantages, so it was decided to not do that.

https://github.com/rust-lang/rust/pull/19612

Are there potentially other situations where `&[T + !Sized]` makes sense?

The majority of the functions on `&str` seem to make sense for all `&[T + !Sized]` where `type str = [unsized_char]`.

I don't really know what unsized_char would even mean, chars have a size, and str is not a sequence of chars.

That said, I'm also not sure in the general case.

unsized_char would be some phantom/opaque type that is not actually useable because it represents an object of variable length.

e.g. Think of it as an enumeration like :

    unsized_enum unsized_char {
        Char1(u8),
        Char2(u8, u8),
        Char3(u8, u8, u8),
    }

    let c: unsized_char = ...; // is a complied error: “cannot assign a variable stack size.” or something.

    let parsed: &[unsized_char] = unsafe { /* parse some bytes */}; // this complies because the implementation of some hypothetical &[T + !Sized] appropriately abstracts the allocated memory region.

Unlike a normal enum it doesn’t always take 3 bytes of memory, it’s variable memory such that [unsized_char] might only take 1 byte if there is only 1 unsized_char::Char1 inside.

Of course this means that you can’t index into the slice because the boundary is not clear.

What I’m describing is not different than the internals of str. I’m just trying to armchair design a generic implementation such that str and slice are more unified.

You’re right that what you’re suggesting could work as a way to treat &str as a type of slice, without breaking the type system. But there’s a fatal flaw. If you have `s: &[unsized_char]`, what should slicing, say, `&s[2..4]` do?

With the actual `&str`, this treats 2 and 4 as byte indices. But if you’re going to treat str as a slice of a specific type, the indices really should be in units of that type. `&s[2..4]` should return the second and third grapheme clusters, or codepoints, or whatever you want to define `unsized_char` to be. The same applies to `len()` and all the other methods that return or accept indices.

But if it’s in units of `unsized_char`, how do you actually implement indexing? You could do an O(n) scan from the beginning of the string, but that’s clearly unacceptable. Or you could set up some kind of acceleration data structure behind the scenes, but that would be expensive to maintain, and would conflict with Rust’s goals of explicitness and cheap FFI.

If on the other hand you keep indexing with byte offsets, you end up with almost no operations that work, and do the same thing, for both &[unsized_char] and normal slices. You wouldn’t be able to write any useful generic code that operates on &[T] for T: ?Sized, at least not without branching on which kind of slice it is. And even from a human user’s perspective, the syntactic similarity between two things that work differently could confuse more than it helps. Again, a matter of explicitness.

Are they though? I've long wondered why the Rust team hasn't imitated C# or other languages ease of use with strings while also retaining the existing functionality for lower-level use cases. I suppose it's a kind of gauntlet that a Rust dev would have to go through which could be a good thing but personally hitting walls with strings really turned me off on Rust the first time I tried it simply because my expectations were diametrically opposed to the reality of strings in rust.

Different languages trade off different things. I don't know enough about C#'s string handling to reply effectively, but Rust does the best it can given the things it cares about. Other languages don't care about the same things, and therefore, can make different tradeoffs.

`str` contractually guarantees UTF-8 contents, so because of multi-byte codepoints it cannot be sliced at arbitrary indexes like a `[u8]` can be.

As a side note, it is possible to define your own "unsized" slice type which wraps `[u8]`. This can be useful for binary serialization formats which can be subdivided / sliced into smaller data units.

> `str` contractually guarantees UTF-8 contents

I don’t think any other languages do that. Instead, most of them are implementing as much as they can while viewing the storage as a blob of UTF8/UTF16 bytes/words, and throw exceptions from the methods which interpret the data as codepoints.

Strings are used a lot in all kinds of APIs. For instance, strings are used for file and directory names. The OS kernels don’t require these strings to be valid UTF-8 (Linux) or UTF-16 (Windows).

To address the use case, Rust standard library needs yet another string type, OsString. This contributes to complexity and learning curve.

The number of things that you have to learn remains constant. I could even make the argument that the number of things you need to learn up-front is lowered when only talking about the distinction between String and OsString/CString. The difference is that rustc will be pedantic and complain about all of these cases, asking you to specify exactly what you wanted, while other languages will fail at runtime.

> rustc will be pedantic and complain about all of these cases, asking you to specify exactly what you wanted

So, they offloading complexity to programmers. Being such a programmer, I don’t like their attitude.

> other languages will fail at runtime

In practice, other languages usually printing squares, or sometimes character escape codes after backslash, for encoding errors in their strings. That’s not always the best thing to do, but I think that’s what people want in majority of use cases.

The complexity already exists regardless of what the language does.

The only choice is whether it's explicit and managed by the language, or hidden, and you need knowledge and experience to handle it yourself without language's help. If you want "squares" for broken encoding, Rust has `to_string_lossy()` for you. It's explicit, so you won't get that error by accident.

Avoiding "mojibake" in other languages is usually a major pain. For example, PHP is completely hands-off when it comes to string encodings. To actually encode characters properly you need to know which ini settings to tweak, remember to use mb_ functions when appropriate, and don't lose track of which string has what encoding. There's internal encoding, filesystem encoding, output encoding, etc. They may be incompatible, but PHP doesn't care and won't help you.

> It's explicit, so you won't get that error by accident

I would want it to be implicit.

Ideally, for rare 20% of cases when I care about UTF encoding errors, I’d want a compiler switch or something similar to re-introduce these checks, but I can live without that.

> For example, PHP

When you compare Rust with PHP it’s no surprise Rust is better, many people think PHP is notoriously bad language: https://eev.ee/blog/2012/04/09/php-a-fractal-of-bad-design/

I like C# strings the best, but I also have lots of experiencer with C++, and some experience with Java, Objective-C, Python, and a few others. None of them have Rust’s amount of various string types exposed to programmers, many higher-level languages have exactly 1 string type.

Interestingly, some dynamic languages like swift use similar stuff internally, but they don’t expose the complexity to programmers, they manage to provide a higher-level abstraction over the memory layout. Compared to Rust, improves usability a lot.

> I would want it to be implicit.

Then simply use the `lossy` function. Or create an alias to it if you don't like the name.

Because there's no possible choice for T. You can't use u8, because then you would allow non-utf8 data. You can't use char, because a &[char] uses four bytes per character, whereas a &str stores the characters in utf-8, which is a variable-width encoding.

A &str is really a different kind of thing from other slices. In any other slice, each element in the slice takes up a constant number of bytes, but this is not the case for a &str.

That's a great question. I don't have a complete answer, but I do know that &str has lots of string-specific functionality that's really helpful. The .chars() method for example gives you an iterator over actual unicode chars, as opposed to bytes, because the former can have variable byte-widths. There may be other reasons; I'm not sure.

In Rust a slice `&[T]` is a fixed length sequence of `T`s laid out linearly in memory. Every `T` is required to be of the same size.

Strings in Rust are (normally) represented as UTF-8. Both `String` and `str` represent data that is guaranteed to be valid UTF-8.

This means that if Rusts UTF-8 strings were represented as normal slices, they would have to be slices of UTF-8 code-units.

Rust wants to provide a safe and correct String data type, and therefore, indexing a string on a byte (code-unit) level would be incorrect behavior.

Having a custom type `String` and `str` instead of just a `Vec<u8>` enables you to have more correct behavior implemented on top of the data type that doesn't implement normal slice indexing and such.

---

As a note, even though you probably don't want to normally, you can quite easily access the backing data of your string using `String::as_bytes`

The regular slice type &[T] lets you access and manipulate individual elements. But Rust strings enforce the invariant that they are valid Unicode, which puts restrictions on element-wise operations.

Calling &str a "string slice" is really more about the contrast with String, and how the relationship there mirrors the relationship between &[T] and Vec<T>. It's more of an analogy than a concrete description of the interface.

What would they be, &[u8]? That's already a thing: it's an arbitrary byte sequence. &str is specifically UTF-8 data.

&OsStr and &Path are the same way.

Indeed &str has as_bytes which returns itself as &[u8]

But, str is a subset of &[u8], the type's contract is that it is unsafe to have non valid UTF-8 data, hence https://doc.rust-lang.org/std/str/fn.from_utf8.html can error, offering the unsafe variant https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm...

This is all very different than &[char] which would be an array of 4 byte characters (or, a UCS4 string)

> But, Rust has a notation for slices -- &[T]

&[T] is an “array-slice”, even if it's called just “slice”.

See this example ([..] is the syntax to create a slice of something): https://play.rust-lang.org/?version=stable&mode=debug&editio...

Long story short, Rust slices are contiguous collections of identically-sized elements. Strings, by being UTF-8, are not guaranteed to be identically-sized. So they require a distinct type that lets you work with them in spite of this detail.

Another: String is inmutable AND whole. &[T] is parts, and when declared as &mut [T] is mutable.

Because unicode, &[T] make easy to write wrong code (that asume here T = Char).

It CAN'T be Char, because char is larger than u8:

https://doc.rust-lang.org/std/primitive.char.html

and it mean unicode point.

In other words: Rust is using types to PREVENT the wrong behavior.

Fun fact: `&mut str` exists. You don't get random access, but in controlled scenarios it's fine to mutate str in-place, e.g. `make_ascii_lowercase`

https://doc.rust-lang.org/stable/std/primitive.str.html#meth...

"string slice" is just the name of the one thing, and "slice" the name of the other, and they are different things with similar names and related features.

Reply on news.ycombinator.com