Hacker News Reader

Comment by latch | Hacker News Reader

I've started to learn rust for the 2nd time. The first time was a really frustrating. In my defense though, I'm pretty sure the compiler inferred a lot less back then (maybe 5-6 years ago now?)

This time, it's going much better and I've managed to build something non-trivial. I quickly ran into the self-referencing issue. A simple structure to hold an [normalized] phrase and a list of words within that phrase:

    struct Input<'a>{
      words: Vec<&'a str>,
      phrase: String,
    }

I ended up asking in https://users.rust-lang.org/ where people are super fast, friendly and helpful. I understand that it isn't allowed, but I still don't quite understand why it isn't. It seems completely straightforward and unambiguous to me.

Something else I'm surprised the article didn't mention is the combination of `Box<dyn X + Sync + Send>` which is a mouthful - both in terms of having to declare a type with this, and in terms of everything is implies/encapsulates.

View on HN

The "why is this an issue" has already been mentioned ("what if `self.phrase.clear()` is called? that creates dangling references in self.words"), but how to solve it is relatively straight forward:

    struct Input {
        words: Vec<(usize, usize)>,
        phrase: String,
    }

    impl Input { 
        fn word(&self, i: usize) -> &str {
            let (first, last) = self.words(i); 
            &self.phrase[first..last]
        }
    }

Now your Input type doesn't even need to be generic over the reference 'a anymore (not that it mattered before because that wouldn't work).

Now even if self.phrase is resized, you never get memory unsafety, because the `self.phrase[...]` does a bounds check and panics on out-of-bounds. So the worst you can get is a "logic error" in your program that shows up with a really nice backtrace that's super easy to debug, instead of a miscompiled program (due to UB) that's often very painful to debug.

So sure, you don't get "self referential struct with references" in Rust. But so what?

The fix for when you want this is always the same: use offsets like I did above. Offsets are much better than any kind of pointer, because they allow you to just "memcpy" the struct around (the address of a field changes on move and copy, but the offset to that field from the beginning of the struct always remains constant).

If you were to use a pointer to a field, you'd need to correct that pointer during copies and moves. C++ allows this, and this "feature" has many undesirable consequences for Rust:

- moves in Rust are O(size_object), moves in C++ can be of any algorithmic complexity, so any algorithm that moves has to take that into account this

- rust collections have one code path, C++ collections have two code-paths: one for objects that are "simple" to move, and one for objects that are "hard" to move

- move constructors that throw in C++ interact with exception safety in complicated ways

- probably many many more interactions that I'm missing here

There might be ways to improve the ergonomics of self-referential structs in Rust, but given that offsets are simple and always work correctly without many downsides, whatever improvement one considers for Rust would probably not be worth it if it has any of C++ downsides.

> If you were to use a pointer to a field, you'd need to correct that pointer during copies and moves.

The "transfer" crate allows for this, though not in a way that's directly compatible with C++ move. (The latter has been discussed somewhat in https://mcyoung.xyz/2021/04/26/move-ctors/ but that's only a first look at the problem area, not aiming for 100% accuracy just yet.)

Another bonus of the offset approach: it can be smaller. The size of a &str will always be two words (pointer and length), but if you're sure that your strings will never be longer than 4 gigabytes, and you're willing to put up with the necessary integer casts, you can store a (u32, u32), which is half the size of a (usize, usize) or a &str when your target uses 64-bit words. And if you're certain that your strings are never bigger than 65535 bytes, you can go even further and use a (u16, u16), and so on.

If you’re applying arbitrary invariants that aren’t codified in the type system, you may as well just use raw pointers.

Offsets cause bugs when you do things like assume your strings are of a particular length; that’s why they’re a bad choice.

Use a crate that provides a safe abstraction, don’t roll it by hand yourself, and the safe abstraction can be implemented in whatever way you want; but it’s not safe if it doesn’t enforce the invariants.

> If you’re applying arbitrary invariants that aren’t codified in the type system, you may as well just use raw pointers.

This isn’t really true, as the character of bug produced is quite different between the two cases. If you make a mistake with offsets, you’ll get a runtime panic that tells you exactly where the problem occurred. If you make the same mistake with raw pointers, however, you’ll get corrupted memory somewhere which won’t necessarily show up as a problem until long after the erroneous code is executed.

Who honestly cares? I sure don’t.

Yes, there is a distinction, yes one is worse than the other, but if you got a panic, your application terminated.

You’re screwed either way, web server or desktop app; fortunately there’s a simple solution: use whatever means you like to write a safe abstraction that enforces invariants.

I’m not advocating particularly for unsafe code; but avoiding unsafe and writing code that blatantly panics at the drop of a hat is ridiculous.

Invariants should be enforced by the type system.

I don’t care how; if you’re not doing that and just assuming your strings are shorter than 256 characters, you’re writing code that is unsound.

> You’re screwed either way,

Not really.

If the application terminates, it is infinitely easier to catch this during testing.

If the application silently continues, that might be an exploitable security vulnerability, might result in corruption of pretty much anything (data-bases, loss of user data, ...).

Pretty much everybody with an exposed web-server would prefer it to terminate with a nice debug message over the alternatives that using raw-pointers would cause.

> you’re writing code that is unsound.

No you are not, code using offsets is perfectly sound because it does not exhibit undefined behavior (which is how "soundness" is defined in Rust).

Your suggestion of using unsafe instead of offsets here would indeed be unsound. Don't do that.

> but avoiding unsafe and writing code that blatantly panics at the drop of a hat is ridiculous.

This is the most retarded thing I've read all day. `assert` is a tool used on _many_ standard library API to ensure soundness.

> I don’t care how; if you’re not doing that and just assuming your strings are shorter than 256 characters, you’re writing code that is unsound.

I really have no idea why you sound so angry, but the claim that someone anywhere in this thread is "assuming your strings are shorter than 256 characters" is completely made up. Nobody has suggested that.

The offset optimization that the OP proposed is trivial to implement correctly even without bound checks (just put an assert on the methods that take &mut self to ensure that the string doesn't grow beyond that).

> If you’re applying arbitrary invariants that aren’t codified in the type system, you may as well just use raw pointers.

You have to correct raw pointers on move and copy.

Offsets remain correct on move and copy.

I’d just like to say, don’t feel bad.

The responses you’ve received aren’t particularly kind and as a rust user, I’m sorry for that.

It’s entirely fair to say you don’t understand why it’s not possible to have self referential structs, because it is possible;

If you have raw pointers you can have self referential types... so why not refs?

The issue here is that, like references passed to closures, the type checker isn’t sophisticated enough to track the references and ensure things are correct.

You could very plausibly have a struct in the form described above where moving ‘phrase’ is forbidden until the vec is cleared.

You can literally do that inside a function.

So... the problem here is the rust compiler, and that it’s difficult to implement.

(Not sure if this is the official explanation, but makes sense to me)

The issue is that you have references to the String (it's borrowed). That means it can't change, which isn't/can't be expressed in this struct. Imagine clearing the String: Suddenly all references in words are invalid which means you're referencing something invalid, very much not memory safe and a big rust no-go

Let's assume for a moment self-referential references were allowed. Let's further say we have such a struct containing self-referential references and we move it.

Moving actually moves things in memory, including the thing those references reference. This means the references now point at the old and now incorrect location. Accessing them would trigger undefined behavior.

Self-referencing is only acceptable, if the struct is guaranteed not to move. Rust does provide a mechanism to provide such a guarantee with std::pin but outside of async code, you probably don't want to be using it.

std::pin: https://doc.rust-lang.org/std/pin/index.html

If you're curious about pinning and have a few hours to spare, https://www.youtube.com/watch?v=DkMwYxfSYNQ is a great video that explains it in detail.

Self-reference is allowed, but it prevents moving.

  fn main() {
      // ok
      let mut inp = Input { words: vec![], phrase: String::from("foo bar") };
      inp.words.push(&inp.phrase[0..3]);
      inp.words.push(&inp.phrase[4..7]);
      println!("{:?}", inp);

      // this would give "cannot move out of `inp` because it is borrowed"
      //let x = inp;
  }

It also prevents borrowing `inp` or `inp.phrase` as mutable which makes it useless for most cases.

> I'm pretty sure the compiler inferred a lot less back then (maybe 5-6 years ago now?)

If it was post Rust 1.0 then it did not. The more likely case is the combination of NLL and match ergonomics, which has made a lot of code way more comfortable, respectively by relaxing local borrows and avoiding a lot of minutia in pattern matches.

I understood the “inference” comment as “the Rust compiler is more helpful now”. Which it is. @ekuber and gang have put a lot of effort into better error messages over the years and continue to do so.

Match ergonomics was one of the greatest improvements to the language, I’d personally put it almost on par with NLL.

For anyone who’s interested:

In Rust, you tend to do a lot of pattern matching and destructuring. But you also often only have a reference to your data, because copies can be expensive.

Combining pattern matching and references was always possible, but confusing: you had to place lots of “ref” annotations in various places, which I never got right on the first try. But now, you can magically destructure e.g. a reference to a tuple into a tuple of references. You don’t have to think about it, it simply works. Combined with Rust’s algebraic types, this allows you to write very elegant, almost fp-like code that’s also zero copy.

Those match "ergonomics" really tripped me up when first trying to learn Rust.

I was trying to figure out references and deref, and I kept thinking I "got it" and trying to make sure by writing examples that should and shouldn't work.

However, because of all the deref "ergonomics" magically spread around thinks like match, it was really hard to confirm whether I had the right mental model or not. Lots of things that surely shouldn't work somehow worked, and I couldn't explain why. I kept having to go to forums and people just told me "oh, this is a special magic case we added". Very frustrating.

Yes, while "match ergonomics" made many "just work", the magic it implies also obscures the language, and code, a lot.

I remain of two minds about it.

If you want to make sure you have the right model you can always set edition to 2015 tho. You will also lose NLL and a bunch of other conveniences, but match ergonomics will be disabled.

NLL is in Rust 2015 these days. (As of Rust 1.36.)

> If you want to make sure you have the right model you can always set edition to 2015 tho.

That's a really interesting idea. I wonder if that would work well for teaching languages in general. Like teaching Java 1 instead of 16 where the language was much simpler.

I can totally see how this kind of convenience might hinder learning. For me, the fact that “&a” does so much magic type conversion stuff had a similar effect. One thing I sometimes find helpful is to run an IDE (like VS Code with the Rust Analyzer plugin) and hover over every single variable to see its type.

Out of curiosity, since I expect to give a short Rust introduction in the near future, what was your language background when learning Rust? How familiar were you with C-style pointers and/or C++-style references?

A similar trick I learned from Jon Gjenset is to assert something has the unit type and read the error.

    let foo: () = complex_thingy();

> Out of curiosity, since I expect to give a short Rust introduction in the near future, what was your language background when learning Rust?

I have a background in lots of high level languages, like Python and Haskell, but never thought too much about how the types were represented.

But prior to learning Rust I spent some time writing C (I've never done C++), so I guess I came in with a simple idea of how pointers work.

I've ran into that exact same problem before too. My solution would be just to have a Vec<Range<usize>> instead. Then you can generate the substrings on the fly when you need to use them

Another aspect is that in Rust, all structs have "move semantics" in that you can copy the raw bytes of a struct to another location in memory without anything breaking. This is very nice in general but does rule out creating structs in safe code which contain pointers to themselves.

Works for me:

  #[derive(Debug)]
  struct Input<'a>{
      words: Vec<&'a str>,
      phrase: &'a str,
  }

  fn main() {
    let s="foo bar baz";
    let input = Input {
        words: s.split_whitespace().collect(),
        phrase: s,
    };
    dbg!(input);
  }

   Compiling playground v0.0.1 (/playground)
    Finished dev [unoptimized + debuginfo] target(s) in 1.15s
     Running `target/debug/playground`
  [src/main.rs:13] input = Input {
    words: [
        "foo",
        "bar",
        "baz",
    ],
    phrase: "foo bar baz",
  }

Wouldn't a Box<&'a str> put the reference itself, not the contents, in the heap?

Input isn’t self-referential in your example, because both of its fields are referencing the same data owned by the main function.

For it to be self-referential, Input needs to take ownership of the string in phrase, and words needs to then be referencing phrase.

You can do this with Pin, but it’s not simple.

Yep, it can be done with Pin and unsafe only, AFAIK. In theory, Rust can be extended to have relative references, but it's much simpler to use arrays and indexes and ranges instead.

With Pin:

  use std::pin::Pin;
  use std::marker::PhantomPinned;

  #[derive(Debug)]
  struct Input<'a>{
      words: Vec<&'a str>,
      phrase: Box<str>,
      _marker: PhantomPinned,
  }

  fn test<'a>(s: String) -> Pin<Box<Input <'a>>> {
    let bs = s.into_boxed_str();

    let mut input = Box::pin( Input {
        phrase: bs,
        words: Vec::new(),
        _marker: PhantomPinned,
    });

    unsafe { 
        let phrase: *const Box<str> = &input.as_ref().phrase; 
        let words = phrase.as_ref().unwrap().split_whitespace().collect();
        input.as_mut().get_unchecked_mut().words = words;
    }

    input
  }

  fn main() {
    let s="foo bar baz".to_string();
    dbg!(test(s));
    
  }

Yeah, I tried picking rust up a few years ago and ran into a few weird issues that would block me from making good progress. I've now used it to create a parser that reads data and converts it to other formats (for my bespoke needs) and it's so fast it's crazy. I had much less issues this time, CLion integration with Rust is really helpful as well. The tooling around rust is now great (rustup, cargo, etc).

I just tried that in nightly and it seems to work? Not sure what the issue with that is.

It works, but it isn't what they want. The compiler won't allow borrowing the str slices from the owned String field.

Yes it will. https://news.ycombinator.com/item?id=27164504

This specific code compiles, you just can't create an instance of this struct and store references to `phrase` in `words`.

Reply on news.ycombinator.com