My reference was dropped, why is the compiler complaining about multiple borrow | Hacker News Reader

My reference was dropped, why is the compiler complaining about multiple borrow

ntietz.com

48 pointstodsacerdoti52 comments

Right. A local reference is escaping the loop scope at the "return". This is safe but weird, and the current borrow checker isn't smart enough to confirm that it's safe. The fancier "Polonius" borrow checker in development is supposed to be less restrictive.

The code example is Rust written as if it were classic C. In Rust, that example would usually be written like this:

    /// Find leading zeros in an array of bytes.
    fn find_leading_0s(bytes: &mut [u8]) -> Option<&mut [u8]> {
        if let Some(pos) = bytes.iter().position(|v| *v != 0 ) { // search
            Some(&mut bytes[..pos]) // find
        } else {
            None // no find
        }
    }

Rust isn't a fully functional language, but the functional features play well with the lifetime system and are optimized well.

(Rust playground example: https://play.rust-lang.org/?version=stable&mode=debug&editio...)

A simple "--make-borrow-violations-into-warnings" flag (that for the sake of the panicky would work only on debug builds) would let you get on with your work, and come back before pushing to a shared repository to settle up.

Probably need to release it as a patch so as not to need approval of the purity gatekeepers. People using it get an immediate boost in productivity as it enables exploration with no ultimate reduction in safety.

Sounds like a great way to paint yourself into a corner, and explore solutions that are impossible to be made viable in the general case, even if they happen to work for certain inputs during debug.

I would like that mode exclusively for the test suite to use. That way you start writing your tests, and if there's a borrowck error You delay that until the test runs. If it runs, and then it fails when accessing data, you can give more information to the programmer about the failing case. If it runs successfully, you just fail that one test with the borrowck error. I would love it if we could have something akin to prop test for borrowck so that even the test isn't needed.

You would not ever be obliged to use the flag, so its presence would not affect you.

It doesn't matter whether it "works for certain inputs", or for any inputs. The point is to be able to direct attention to more immediately important algorithmic matters now, and then address borrow checker pedentism once actually difficult questions are resolved. The point is not to be able to ship code with borrow violations. It is hard for me to understand what is so difficult about this distinction.

If that's the workflow you want during development, you can always replace references with a RwLock. That of course is not made very fun by neither the language (you need to touch too many places in your code to switch back and forth) nor the tooling (we need more advanced refactoring tools, if the language won't change, I'd love it if we had an "add a lifetime to this type everywhere" action in my editor).

The distinction is quite clear, the reason to ever use such a flag - less so. Toolchain features are very much a popularity contest, if there is no real reason to add a feature (and, in fact substantial reason not to), it shouldn't be implemented.

The idea to use it during rapid prototyping is especially problematic, since that's the point where pernicious data access patterns will infect the entire design. Once the application "works, save from some double free bugs in weird testcases we don't expect in production, and which require a one month refactoring", the temptation to ship the flag into production will become overwhelming.

You should just not use Rust. A language should hold strong to its philosophical principles. These kind of changes are fundamentally design by committee.

I agree in principle - what's the point in deciding to use Rust but turning off the borrow checker?

But on the other hand

- Rust already has unsafe mode, which is exactly for allowing you to avoid some safety checks when you've decided you are smarter than the compiler

- this really is an example where the compiler isn't smart enough, as evidenced by the fact there are plans to make it work

So another "unsafe" type of marker that watered down the borrow checker would not be totally inconsistent with the language philosophy. It should work at the block or the file level, though - surely you wouldn't want to turn that off for all your code.

The whole point of the current `unsafe` mechanism is that you have to explicitly opt one specific chunk of the code out of stricter checking, and it's obvious which chunk of code it is that needs that behavior (the bit inside the block) and which isn't (everything else) -- that bit can be subjected to extra scrutiny upon review, and there's a community-wide shared understanding of what invariants that chunk needs to uphold for the program as a whole to be sound. I'd argue that the alternative mechanism you propose (a compiler flag), which just opts the whole program out of strict checking, _is_ fundamentally philosophically different. Having the opt-in/opt-out mechanism be block-level is a language-design/philosophical decision.

Right, I didn't propose a compiler-level flag (though the original suggester did), I specifically said it needs to work at the block or file level.

Do you think there should also be a “--make-type-violations-into-warnings’ flag?

I think there would be value of such an option, as long as it would be completely forbidden to publish to crates.io with code that only works with those flags.

It could work by replacing any function with type errors (including those caused by type definitions) to panic on use, but otherwise still generate a binary. For type errors it is less useful than for lifetime errors because usually the problems of the former are very local while for the later it they are not.

A program with borrow checking violations is incorrect the vast majority of the time. Not something to be "settled up" but something that is wrong.

Most of the code you write is similarly incorrect at first. Then you fix it, and then it's right. Unless you ship the buggy code, the bugs harm nobody. Likewise, borrow failures. It is hard to understand what is so difficult about this concept.

Idiomatic: https://godbolt.org/z/6ndYYxhx3

Original: https://godbolt.org/z/rPfEWPGc1

It’s cool to see the same code be generated. You can use “vec!” To stop it specializing the function in the length

It's surprisingly difficult to write something where the compiler doesn't detect that it's working on a constant. I had to code up a loop that pushed values onto the vector before the compiler generated the general case. "black_box" is not, apparently, opaque to the optimizer.

Ideomatic form, inner loop: [1]

    LBB4_3:
        cmp     byte ptr [rax + rcx], 0
        jne     .LBB4_4
        inc     rcx
        cmp     rdx, rcx
        jne     .LBB4_3

C-like form, inner loop: [2]

    .LBB4_3:
        cmp     byte ptr [rax + rcx], 0
        jne     .LBB4_4
        inc     rcx
        cmp     rdx, rcx
        jne     .LBB4_3

Both ways, the compiler found the optimal solution. In the C-like loop, it was able to hoist and then elide the subscript check. In the ideomatic form, it's implicit in the iterator that you won't go off the end of the array. The lambda function was expanded inline and optimized. There's zero call overhead from using a lambda function there.

Very nice work by the Rust compiler people and LLVM.

[1] https://godbolt.org/z/5hn7Y9Mf9

[2] https://godbolt.org/z/rW4MG4v3c

Currently fighting the borrow checker.

My project has not moved much this last week due to an issue I just can't understand. Have rewritten the offending code 4 times but it always fail to compile somehow.

At this point, I am only staying with Rust because of the ergonomics of enum and iterators. The memory safety stuff is 90% writing safe code then fighting the compiler to agree with you.

> The memory safety stuff is 90% writing safe code then fighting the compiler to agree with you.

This isn’t my experience, from a decade of occasionally watching and helping others learn Rust, personally and professionally. Rather, it’s normally writing code that you believe is safe, wrestling with the compiler and finally realising that it was right all along, or giving up before reaching this stage but it was still. There are certainly some cases where this isn’t the case, and this article talks about the most common and most notorious one, but seriously, the compiler is normally right.

Somewhere along the way, things click and your intuitions subsequently almost always match the compiler’s, and you have a much happier time of it, seldom needing to wrestle. And your coding in other languages tends to improve, or at least focus on ownership more in ways that tend to make later maintenance easier.

Me, writing a thing in Python: La, la-la, la-la.

Me, writing it in C: Grumble, grumble, la-la-la.

Me, writing it in Rust: I HATE THIS THING WHY DO YOU oh hey I'd been thinking about this all wrong.

I go through that cycle all the freaking time. At first, I'm irked that Rust is being such a pain in the neck. Then I give in and do things the way it wants me to, and realize that it's genuinely much better that way, even if I then take that approach back to Python. At the least, now I'm more aware of when I'm counting on the garbage collector to atone for my sins.

Your “experience” (bias?) is easily falsified. Obviously memory issues happen all the time in unsafe languages, but even then, not nearly at the frequency that a novice Rust user will conflict with the borrow checker. The article we’re commenting on is literally about safe code that the borrow checker fails on.

If you could put your project up on GitHub or similar, I'd be happy to take a look at the errors. Otherwise without knowing anything about your specific situation, any chance this helps? https://jacko.io/object_soup.html

Use OCaml if you can live with a GC.

Why not Go?

Because it doesn't have ADTs and pattern matching, which the gp explicitly called out as their reason to use Rust. There are garbage collected languages that have that, but Go isn't one of them.

Depending on the issue you could get away with either using unsafe code or just using a RefCell (if you can’t convince the compiler a mutable reference isn’t being shared, wrap it into a RefCell and then you have a shared reference that you can do mutable operations on.)

> I am only staying with Rust because of the ergonomics of enum and iterators

Don't other languages have things that are close enough?

Have you tried asking GPT4 for help? It understands Rust fairly well, and can also explain what's going wrong quite clearly.

I haven't used GPT4 to help with any borrow checker issues, but just yesterday I figured out how to make my FFI use case work in about 3 minutes after asking. I'd been struggling with it for hours.

Is your problem domain relevant to low-level programming? Because if not, I would drop Rust immediately and just go with an easier language.

there are many excellent languages that have adts (and much more) without having to deal with the borrow checker.

Maybe give ocaml or haskell a try :)

FWIW, you can also write Rust without having to "deal" with the borrow checker either. A common problem I see from experienced programmers picking up Rust is trying to write code with minimal allocations, before learning enough of the language to properly represent them (or whether the pattern can be represented) in safe Rust. I recall an argument I got into with a colleague who was adamant that boxing closures was the wrong call and spent an inordinate amount of time trying to represent a pattern that wouldn't work without doing so. Or the time I myself spent a lot of time trying to return borrowed values when in hindsight I should have called .to_owned() once and spared the time I spent looking for an allocation-free alternative. (Cow would have worked in that case, but I didn't have the experience to know that for sure, and today I wouldn't have chosen it because it would have made using the API more cumbersome than needed.)

A lot of the borrow checker problems go away if you're ok with allocations, storing trait objects on the heap, making your ADTs not have references, etc. But getting familiar with the language enough to know "this requires unsafe or for<'a>" takes a while.

This is not fighting the compiler, this is writing code in a way which can be proved to be safe.

Some code can be safe but is hard to reason about. The compiler tells you that. The solution is, unsurprisingly but perhaps disappointingly, to stop writing code like that.

> Some code can be safe but is hard to reason about.

This is a point I made back when I was doing verification. Yes, you can write code for which termination cannot be proven. If termination for your code is theoretically undecidable, it probably won't work right anyway. So don't go there.

Microsoft took a hard line on this with their Static Driver Verifier, a proof of memory correctness system for Windows kernel drivers. If your driver code is so hard to analyze that automatic path analysis can't verify memory correctness, it doesn't get signed to run in kernel space. Driver-caused Windows kernel crashes, formerly a huge problem, mostly went away.

What you basically wrote is that my problems would go away if only I wrote code that the compiler approves..

But that is not easy, and much of that is due to the complexity of Rust and the way it handles memory safety. Which is where fighting the compiler comes in.

Yeah I get it, I was there.

It is super hard if you actually know C/C++ or mastered a GC language. Rust won’t like some patterns at all and will fight you.

The point is, there are replacements for those patterns that are safer, about as performant and simultaneously easier to reason about by both you and the compiler. To stop fighting, you must switch to them. Old ways will not work. This is by design and a key point of Rust.

A single, concrete example: do not use references in object graphs. Use an array and indexes into it. If you have a dynamic structure with many additions and deletions, use generational indexing.

If it's not a project under NDA, like another user, I'd be happy to put eyeballs on it and help you work out the errors

Add more braces. Not joking.

I find that a lot of Rust borrow checker problems can be solved by inserting more blocks. At the very least, it helps you isolate precisely where the issue is.

I don't see this recommendation often for how much it seems to help beginners. To be fair, it's a lot less of an issue than it used to be as the borrow checker has gotten quite a bit better.

Read https://quinedot.github.io/rust-learning/lifetime-intuition...., the solution to your issue is probably in there

I read that and honestly I can't see how that article could improve anything.

The author goes into a bunch of other issues or uses his own terminology, which makes things even more confusing.

If you have gone through the entire guide and are still stumped, I would suggest asking for help on https://users.rust-lang.org/

(Not sure what you are referring to wrt the terminology)

It’s ok to box or use Arc.

I rewrote my rust project in zig. Final nail to coffin was finding out static building is basically unsupported and cross-compiling is even more annoying. Ended up with way smaller codebase, the code is more readable and im more confident about it because im aware wherever memory is allocated and where it isnt. Async in rust is also complete mess and mixing async / non async is not fun. I dont even want to start on the meta-programming and traits which needs a degree on rust to get started.

Minor point: `let padding = &mut bytes[..index]; padding[index]` is an out-of-bounds index and will panic even if the borrow checker accepted it. You probably wanted `let padding = &mut bytes[..=index];`

Thank you.

For some reason, I’ve always had immense trouble with index bound arithmetic (questions like: is this inclusive? Exclusive? Is ..0 a single element or none? What about ..=0?). I have to actively think about these as I do when telling East from West. North versus South is simply intuitive, no conscious thought needed. Weird effect.

Glad to see I wasn’t wrong in stumbling over this particular one.

First example is completely broken. rust:nietz 1:0

laughs in "never needed any of these weird mechanisms to manage my own memory"

Classic blub programmer looking up.

https://wiki.c2.com/?BlubParadox

I mean, when coding memory management by hand in C or C++ (although C++ of course has things like RAII and smart pointers which simplify life a lot), you do need to have similar mechanisms within your own head. Otherwise you'll get all sorts of fun stuff like pointer aliasing, use-after-free, or what-have-you.

Of course not, these mechanisms are so that other people can manage your memory :)

What, you never deallocated the bit of memory holding where you put your keys before collecting the keys?

Don't forget to zero the memory first. If you don't, the keys may still be there depending on the allocator.

Also, be careful about how you zero it, so the compiler doesn't optimise it out.

Reply on news.ycombinator.com