Borrow checking without lifetimes
smallcultfollowing.com
smallcultfollowing.com
I vehemently disagree with the article that lifetimes are at all nebulous or hard to grasp, IMO they're a pretty straightforward concept, and they map really nicely onto single ownership, move, RAII based memory management, and underlying C style memory management concepts, whereas OTOH it feels like loans are perhaps less nebulous, but also map more poorly onto the best comprehensible, and specific ways to think about low level memory management (and also less well to concepts in other low level languages lile C++). Instead of seeing your program as a mostly 1D collection of mostly contiguous scopes that the program counter jumps around in, now you have to view it as a gigantic thicket of constraints.
So in essence, by making Rust's static analysis more powerful and less annoying at first, we're actually making it harder to fully grasp in the long run. It's sort of like the Haskell monad problem — the more powerful you make your compiler/language abstractions to allow proving more code, the less comprehensible everything gets. And I think in both cases, trading some power, past a certain point, for long term comprehensability with more straightforward concepts is better.
I'm not sure what's neutered about Rust's current plans for generators, and they aren't separate from async, they're the foundation that async desugars to.
I'm also not sure what your objection is to Polonius, which, so far, is still just a strictly more permissive version of the borrow checker, with nothing new to learn on the user end. AFAICT the notation in this blog post is not actually proposing any new Rust syntax, and is instead proposing syntax in the formal language that is being used internally to model Rust's type system.
They're neutered because they can't suspend and transfer control to a function other than the one that called them ("Note also that "coroutines" here are really "semicoroutines" since they can only yield back to their caller." https://lang-team.rust-lang.org/design_notes/general_corouti...) and you can't pass values into resume and get them out from the yield statement in the coroutine (https://github.com/rust-lang/rust/issues/43122#issuecomment-...).
> and they aren't separate from async, they're the foundation that async desugars to.
Yeah I just looked it up again and I don't know why I had it in my head that they were separate, you're correct, they are the same thing under the hood, so honestly that eliminates my biggest problem with them.
> 'm also not sure what your objection is to Polonius, which, so far, is still just a strictly more permissive version of the borrow checker, with nothing new to learn on the user end.
The entire model is different under the hood, though, since it switches from lifetimes+borrows to loans, and so in order to fully understand its behavior the user really would have to change their mental model, and as I said above I'm a huge fan of the lifetimes model and less so of the loan model. I just feel like it's much more natural to treat the ownership of a memory object and therefore amount of time in your code that object lives as the fixed point, and borrows as wrong for outliving what they refer to, then to treat borrows as the fixed point, and objects as wrong for going out of scope and being dropped before the borrow ends, because the fundamental memory management model of Rust is single ownership of objects, moves, and scope based RAII via Drop, so the lifetime of an object kind of is the more basic building block of the memory model, with borrows sort of conceptually orbiting around that and naturally being adjusted to fit that, with the checker being a way to force you to adhere to that. The loan based way of thinking would make more sense for an ARC-based language where references actually are more basic because objects really do only live for as long as there are references to them.
I think that the linked comment is out of date, and that this is supported now (hard to tell because it hasn't been close enough to stabilization to be properly documented): https://github.com/rust-lang/rust/pull/68524
As for Polonius changing the underlying mental model, I think this is a natural progression. Rust 1.0 tried to present a simple lexical model of borrowing, and then enough people complained that it has long since replaced the simple model with non-lexical lifetimes in order to trade simplicity for "do what I mean". And since it's not allowed to break any old code, if you want to continue treating borrowing like it has the previous model then that shouldn't present any difficulties.
Huh? At first glance wanting to do this sounds absolutely insane to me. As in, it sounds a lot like "imagine a function could return not just to its caller function, but to other functions as well! So if you call it, it might be that you won't even end up returning to the same function with which you started, but somewhere completely different".
What am I missing? This sounds absolutely insane at first glance, like a straight-up go-to. What's the most simple use case for these sort of weird yields to other functions?
This isn't something that really causes headaches, it's desirable. So long as when you come in conflict with the borrow checker there's an ability to unwind it, I don't see the harm.
My assumption is the lifetime explanation while incorrect will also still work (I couldn't imagine it wouldn't as that'd break too much). So assuming you do run into these compiler problems, you can still revert to the old simpler mental model and move forward.
Examples of languages with some form of automatic memory management and RAII.
Swift, D, Nim, C#, Active Oberon, Modula-3, Delphi.
Those types don't have any sort of explicit lifetime that are different from a regular object type. If you put a `int` on a `Foo` object that `int` lives as long as the `Foo` object does, for example.
Being a value type simply means that the memory representation of the value is used instead of a pointer/object reference.
That means that even when you do have user defined value types the same rules apply. How long these things "live" depends entirely on the context of what they are associated with. If you have a `Bar` value type on a `Foo` object then `Bar` will live as long as `Foo` does, which means till the next garbage collection.
Instead you decided to point out the philosophical meaning of value types.
> The ones I listed have classical C and C++ like value types available to them
I don't really know what that means or how it's related to the discussion of lifetimes.
And in fact, confusing the two can lead to problems. C#, for example, does not necessarily store value types on the stack. [1] Those can be allocated on the heap. C# doesn't even give a guarantee of how long a value type will live. It's only guarantee is the one I outlined earlier, that this thing is represented as a block of memory rather than a pointer and that when you pass this around it's done as a copy.
If your assumption is that "this thing is freed up immediately when it's unused" that's a faulty assumption. Being a value type imparts no guarantee on how long something will live.
This is more than just a philosophical definition.
If you want deterministic resource management in C#, you use the `using` statement. If you are in java, it's `try-with-resources`. If you are in C++, you rely on RAII. If you are in C... good luck. None of those things have anything to do with value types. Because lifetime and type aren't related in those languages
> Once you abandon entirely the crazy idea that the type of a value has anything whatsoever to do with the storage, it becomes much easier to reason about it.
[1] https://ericlippert.com/2010/09/30/the-truth-about-value-typ...
In terms for article contents - structs absolutely do go on the stack when you declare them as locals except select scenarios: async and iterator methods, both of which can capture variables that live across yields or awaits into a state machine struct or class (debug/release difference, and also ValueTask which does not alloc when it completes synchronously).
If you care about memory lifetimes, the compiler will helpfully error out when you are returning a 'ref' to a local scope, violating the lifetime (because it has rudimentary lifetime analysis underneath hence scoped and unscoped semantics).
It was posted yesterday, 4 March 2024.
It talks about that spec does not say where structs go. And yet, all existing implementations today have more or less identical behavior (and by all I mean .NET (CoreCLR), Mono (the dotnet/runtime flavour), Mono (the Unity flavour) and .NET Framework.
With that said, only respect to the article's content and, of course, Eric Lippert. For the context of the discussion, however, it may be misleading. C# has gained quite a few low level primitives since it has been written too.
It does if you use the right set of keywords, learn them.
Also we should take into consideration that while these concepts are somehow hard to use in Rust versus other languages, in Cyclone they are even more complex, and Rust is actually a more ergonomic version already, despite its complexity.
Usually feels like many Rust users never seen other ML linage languages.
I use manual lifetimes very infrequently. I honestly used to use them more when trying to represent referenced types in structs, but usually find myself reaching for Arc<Mutex> now.
The classic "they were so interested in whether they could, they never considered whether they should"
shared(x.a.b.c)
mut(y.d.e.f)
Superficially it looks very much like what I did in Regent [1] though I am sure the details differ widely.
Currently structs are allowed to borrow from any data outside of them, but not from any of their own fields. Rust doesn't want to have move constructors, and borrowing from inline fields can't be safe, but borrowing from heap-allocated data owned by the same struct should be possible, and currently there's no non-hacky way to express that.
There's widespread desire to add some form of language support for pinning someday, if only to make the ergonomics a bit nicer (e.g. not needing a method call to reborrow mutable pinned references). This would probably also be needed in order for the compiler to support self-referential structs. From I've seen there are two quite different proposals for how to add language support for pinning. One is to add sugar to `Pin` while keeping the design mostly the same. For example, `Pin<&mut T>` could turn into `&pin mut T` or something. The other is to essentially throw away `Pin` and replace it with a `Move` auto trait: instead of `Pin<&mut T>` you would just have `&mut T` where `T: !Move`.
A `Move` trait would be quite disruptive, but would also have many benefits and ultimately simplify the language. For example, with `Pin`, every type has two possible kinds of mutable references to it: `Pin<&mut T>` and `&mut T`. In the case of self-referential structs, how would a non-pinned reference to one work? And how would that interact with `Drop`, which always takes `&mut self`? You could answer these questions one way or another, just like they're answered for futures (although futures have it easier because they always start in a non-self-referential state). But it all becomes simpler if you take the `Move` approach, where `&mut T` is pinned if `T` is `!Move`, and there is no second reference type to worry about.
I also don’t really know of any other methods / cases besides those using `mem::swap` where `Pin` and `Unpin` are relevant. It works for `swap` because if you have a self-referential data structure, there’s never a good reason you want to tangle the references.
Lastly, normally a mutable reference guarantees there aren’t other references, but a mutable reference to a self-referential data-structure doesn’t have this guarantee because it references itself. Is this why `Pin<&mut T>` is necessary, because a real mutable reference to a set-referential data-structure violates Rust’s not-fully-defined “borrowing rules”? And do we want to keep that specific rule (since we’re mutable borrowing the entire region including the self-referential borrow, and self-referential structures are already unsafe, I doubt it would cause any issues)? Maybe `&pin mut T` should be added, but as its own kind of reference instead of syntax sugar…
Pin and Unpin came years after Rust had reached 1.0; wouldn't adding a new bound to an existing function be a breaking change?
For example [T]::sort (obviously) needs to be able to move the elements in the passed array, but there is no Move bound on T or something similar.
> It basically does if you use pin.
And we have Pin today, but I assumed the OP meant 'self-referential struct' as in, without using Pin.
Only in a new edition the default can then be swapped.
I did, but with Pin would be fine as well
https://play.rust-lang.org/?version=stable&mode=debug&editio...
The compiler will correctly prevent you from moving the value. (Try adding `drop(foo);` at the end.)
The other way to have a struct that requires mutation to move in and out of self-referential-ness (as `async {}` needs, for example) can be achieved with `unsafe` and `Pin::new_unchecked`.
They 'work fine' in that the Rust compiler will let you create one and then never let you move it or access the whole object mutably again while it is self-referential. They don't 'work fine' as in being first-class objects in the language that work as any other object.
For what it's worth, I think Rust made the right choice in not allowing custom move constructors that would enable first-class self-referential types.
> In follow-up posts I’ll dig into how we can use this to support interior references and other advanced borrowing patterns.
[..]
> But also [current] Rust is not able to express some important patterns, most notably interior references, where one field of a struct refers to data owned by another field.
I want to be clear here, it's been a few years since my job was coding in rust, but I found lifetimes to probably be among the easiest parts of rust to understand. Is the issue actually not understanding what lifetimes are, or just that people need to annotate them explicitly (which obviously other languages don't do[1])? The former seems surprising to me, the latter I could understand more readily due to the combination of the newness/novelty of explicitly annotating the lifetime, and also the ' prefix to lifetime names which I can imagine triggering a "that's weird looking" response when people first need them - I think I may have had that response at first, but it was fairly brief as it's just an initially new/novel syntax, and even now having the sigil prefix seems like a reasonable way to distinguish lifetimes from anything else, it's similar to why there's a * in generators in JS despite it not being strictly necessary for parsing or type checking.
One thing that did just occur to me though is how many adopters of rust were coming from languages with entirely automatic lifetime management (GCs, etc) so having to think about object lifetimes at all is new? I realize I've mostly assumed people adopting rust have been C, C++, etc devs where thinking about such things is mandatory and continuous, but if there are lots of adopters from managed languages then maybe that's new. In that case though I don't think the issue is rust's lifetimes specifically in as much as it's just object lifetimes at all?[3]
[1] Because they don't need to because of implicit lifetime management (GC, automatic refcounting[2], etc) or because their programming model is "what memory safety?"
[2] Yes I know refcounting is technically GC, but this is comparison to scanning GCs
[3] [this is an addition] I want to be clear, I am not critiquing devs who have come from managed languages for not being aware of lifetimes, this is literally just saying "if you're from a language where it's not a thing you have to think about, the entire concept may be new"
Maybe not "new" but something that's tedious once you've drank the GC koolaid.
With C++ explicit lifetime annotation is never required. You are expected to just do it right and know how long something will live (RAII helps a lot here).
With Rust, when you get into a situation where lifetimes are required, it can often feel really tedious to have to insert them everywhere. Especially when dealing with structs and fields on structs.
Heck, some things in rust simply aren't possible (in safe rust) that are possible with C++. For example, a class with a pointer to itself. In C++'s case you'd say "Well of course this lives as long as itself does so we don't need to free it" In rust, you literally have to use an `unsafe` block to make such a construct.
And, of course, in managed languages you wouldn't take a second thought to why something like this.
This particularly rears it's ugly head when you come from a managed language and you want to do multithreading in rust. Rust's promise of "We can do safe multithreading" sounds REALLY nice, until you learn exactly what that entails. It may be safer, but it's not easier. C++ is easier to multithread than rust (perhaps not correctly) and it's a LOT easier to do correctly in managed languages.
A comment [0] in this very thread lends credence to this.
[1]: https://github.com/apple/swift-evolution/blob/3055becc53a3c3...