Hacker News Reader

Comment by LAC-Tech | Hacker News Reader

Needs a companion blog, "perfectly safe code the rust compiler will nag you about". And the contortions rust programmers go through to avoid that.

Rust is really impressive in a lot of ways. Type classes and pattern are a great fit for systems programming.

But they're fixated on the idea that everything possible should be a static analysis error, language ergonomics or usability be damned. I'd much rather these be warnings, because no static analysis on earth is going to stop you from actually needing tests to see if your code works.

View on HN

> ... no static analysis on earth is going to stop you from actually needing tests to see if your code works.

Correct. However, the intent of Rust is that the code will not fail [1] due to some silly machine thing below your current level of abstraction.

As an example, I'd point to one thing that Rust does that solves a lot of problems: the Vector. This gives you:

- a buffer to load as needed - bounded memory usage ( no more than 2x what's actually needed, and the ability to tailor it) - automatic resizing as needed

IOW, eliminates 'C programmers disease' (eg: #define BUFFERSIZE 1024 // big enough for anything :-) )

I am sick and tired of writing either the tedious resizing stuff by hand, or using a linked list [2] (which in today's world isn't performant: a self-resizing array will deal with locality-of-reference issues much better).

Disclaimer: I've only peripherally used Rust professionally (ie., corporate innovation sessions, self-contained utilities). I have done enough 'fitness exercise' stuff that I'm confident to comment.

[1] "Fail" as in, do something random like start a cryptominer at root-level-privilege. "Fail to compile" is fine, "panic at runtime" is not fine but not worst case.

[2] Opinion: the reason why linked-lists are an example of something hard to do in Rust is because Vectors make them irrelevant, so why bother?

Vectors really don't make linked lists irrelevant - if you want to be able to quickly pop from the middle, no Vec magic will currently help you

Vectors by themselves don't make linked lists irrelevant, but they remove 90% of the use cases, and most of the remaining ones would be better served by a hashtable, or some sort of fancy tree. Linked lists (especially as C programmers typically implement them) are absolutely awful for performance.

Who forces you to keep rewriting a vector in C?

The lack of generics/templating in C.

Something I find interesting about Rust is that we can do those safe patterns, as long as we're willing to lose some performance.

The way I think of it: Rust forces us to choose between flexibility and zero-cost memory safety.

If we choose zero-cost memory safety (in other words, we don't use Rc or unsafe or large Cells) we can't do things like dependency injection, basic observers, backreferences, many kinds of custom RAII, etc. But we do get speed.

On the other hand, if we allow e.g. Rc into our codebases, we can do these patterns just fine, though there is a performance hit.

The final challenge in learning Rust (IMO) is to figure out when Rc is better, and when we can afford the complexity cost of zero-cost memory safety. I've seen a lot of Rust projects move mountains to avoid Rc, and ironically end up adding more run-time overhead and complexity.

> On the other hand, if we allow e.g. Rc into our codebases, we can do these patterns just fine, though there is a performance hit.

Interesting... I was using one of the Project Euler problems as an exercise for learning Rust; I found that Rc actually improved performance (guess: by eliminating a lot of copies/moves as things went in and out of scope).

Maybe you should have been using more immutable references?

But I'm actually curious why the compiler would be making many copies after optimization passes. Release builds or debug builds?

If you try to code Rust like JavaScript by using Rc everywhere, you will quickly run into panics when you try to borrow from a RefCell that a caller has also borrowed from. The limitations of &mut cannot be trivially worked around.

That's true, one should not overuse Rc.

And the other extreme, avoiding Rc entirely, railroads one into a certain architecture which is good for only some use cases and not others.

I like to think the community will someday learn when to use Rc. It had better, lest low-overhead languages with more flexibility like Cone [0] overtake Rust.

[0] https://cone.jondgoodwin.com/

The "trivial" equivalent to JavaScript is not RefCell, but plain old Cell, applied at the level of individual fields you want to write to. No panics and no (relative) overhead there.

RefCell exists more as a bridge back to the world of &mut, since so much of the ecosystem uses that approach. But if you actually have a good reason to write JavaScript-like data structures (e.g. actual interop with a JavaScript-like language) then it just kind of gets in the way.

Of course, Cell has its own overhead, which is why using large Cells is often worse than just using Rc. It's also very situational; we often have types which don't work with Cell, and it often doesn't make sense to copy objects in many use cases.

It's one of those things that works well in theory, but in practice can fall short.

That's totally beside the point here: you can do a mechanical (thus "trivial") translation from JavaScript-like code to Rc+Cell, without ever hitting that overhead.

That is, JavaScript-like code simply doesn't do large assignments in the first place. It doesn't directly hold move-only objects that are trickier to use with Cell.

If you find yourself hitting those cases, you have left the realm of JavaScript-like code, and entered the realm of manual memory management.

> The limitations of &mut cannot be trivially worked around.

> The "trivial" equivalent to JavaScript is not RefCell, but plain old Cell, applied at the level of individual fields you want to write to. No panics and no (relative) overhead there.

I thought you were implying that we could trivially work around Rust's problems by just using Cell everywhere, my comment was in response to that. It seems now that you were talking about something else, so nevermind =)

I don't understand what you have in mind. Here's some basic JS, iterating over elements of a Node:

    for (let child of getElementById('parent').children)
       func(child)

The equivalent in Rust is "risky" because func might also call getElementById('parent') and then that would panic.

I think you are saying that Cell solves this by temporarily moving all children into a local, iterating over that, and then add them back as children? That seems pretty sketchy to me, you have these weird transient states where Nodes are temporarily removed from the DOM.

No, I'm talking about matching JavaScript's memory layout and memory management, indirection and sharing and all.

The children field is itself a reference to a collection. Reading the field creates a new reference to the same collection, and that is what the for loop holds. If func grabs parent that's fine, it's just one more reference (in the Rc sense).

That collection also just holds references, and the for loop copies them out too. The trickiest bit here is the iteration state itself- you wouldn't be able to use e.g. Rust's typical slice iterator and still retain the ability to mutate the collection, but JavaScript doesn't do that either. Its iterators just have to be implemented taking the possibility of external mutation into account.

The fundamental tradeoff here is that JavaScript simply puts everything behind a reference, all the time, while Rust allows you to work with objects directly. This is what makes sharing+mutation feel so simple in JavaScript- anything you can do to change the "shape" of a data structure is really only changing a reference, and leaving the old version around in case anyone else was still using it.

Thank you, that clarifies the idea for me and it's an interesting approach. It seems like an "all in" design that you might apply to e.g. an interpreter.

Of course in practice we don't abandon Rust's slice iterators to achieve basic observers or backreferences, and I've definitely tripped over "someone else is already borrowing this thing" runtime errors.

Do you have any examples of the “perfectly safe code the Rust compiler will nag you about”? Not trying to start language wars, just genuinely curious as someone who writes Rust on occasion.

I dont have an example in front of me. I had write some weird code to prevent a variable from being dropped unexpectedly.

my go-to example is the non-lexical lifetimes stuff, like when you're somewhere in a match and you don't get to return a reference to a thing because there's another reference that's clearly not relevant anymore at the top of the match

Though it's worth mentioning that Rust gained support for many non-lexical lifetime patterns a few years ago, and has designs on supporting more in the future.

That's why I'm surprised I keep running into this :D

For many, not all, and for those it is the usual RC<RefCell<>> dance.

> Do you have any examples of the “perfectly safe code the Rust compiler will nag you about”?

Not an example, but if the borrow checker approved every proper program in a reasonable time, then it could solve the halting problem.

I don't think the compiler can tell if you are mutably indexing to two different items in a collection like Vec.

It's not very good at knowing when something doesn't alias, for example. E.g. if you do this, the compiler doesn't realize it's safe to do because the array locations are different so I'm not borrowing the specific location mutably more than once. Instead it nags me that I have to split the array into non-overlapping slices.

    if (i != j)
       swap_items(&mut arr[i], &mut arr[j]);

A contrived example and obviously the same can be achieved in many other ways, most of which the compiler would be happier about - but that's often the case with Rust: a seemingly safe thing isn't quite safe enough for the compiler so you have to do it differently. And that's the main problem of ergonomics in the borrow checker imo.

This is helped enormously by helpful error messages, and there is great progress on fixing little paper cuts and improving the borrow checker to make more valid programs accepted by the borrow checker. But it doesn't contain a massive AI or theorem prover so there will always be situations where you'll need unsafe despite not actually being unsafe, or when you'll do something a bit more contrived than you might have expected.

This is a good example. Another common situation is a struct that has both `foo` and `bar` members and then exposes both `.get_foo_mut()` and `.get_bar_mut()`. (Again this is a trivial example, and maybe in the real world they do more work before returning those references.) The problem is that it's illegal to call either of those while the return value from the other is still alive. Even though we know they don't alias each other, and we could totally accomplish the same thing if the members were public, there's no way for the method signatures to express what parts of the struct they don't touch.

I think there is work going on in this area (borrowing parts of structs)? Might have dreamt.

I ran into a borrow check error on code like this[0]:

    enum Inner {
        A(i32),
        B(i32)
    }
    
    enum Outer {
        Foo{
            field: Inner
        }
    }
    
    fn do_foo(val: &mut Outer) {
        match val {
            Outer::Foo{field: f @ Inner::A(id)} if *id == 3 => {
                *f = Inner::B(25);
            },
            _ => {}
        }
    }

The compiler is seeing the `id` and `f` references as overlapping for the entire arm, even though the use of `id` and `f` are not interleaved. Bearing in mind that I don't actually know how the compiler works here, but I don't think this is a borrow checker limitation in and of itself, rather what I think is happening is that in the match expression the compiler is creating both `id` and `f` directly from `val`, creating the overlapping borrow.

The reason I believe that is that this equivalent code results in the same error[1]:

    fn do_foo(val: &mut Outer) {
        let f = val.get_inner();
        let id = val.get_inner().get_a();
        if *id == 3 {
            *f = Inner::B(25);
        }
    }

Whereas if you create the `id` reference from `f` instead of from `val` the compiler accepts it because `f` is not used between `id`s creation and death[2]:

    fn do_foo(val: &mut Outer) {
        let f = val.get_inner();
        let id = f.get_a();
        if *id == 3 {
            *f = Inner::B(25);
        }
    }

[0] Playground link: https://play.rust-lang.org/?version=stable&mode=debug&editio... [1] https://play.rust-lang.org/?version=stable&mode=debug&editio... [2] https://play.rust-lang.org/?version=stable&mode=debug&editio...

> no static analysis on earth is going to stop you from actually needing tests to see if your code works.

I might have been convinced if mathematical proofs were not expressed in code. If a proof can exhaustively cover the problem space, then there's no need for further testing.

https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...

Fair point, but would you deploy un-tested code to production based on a static proof? AFAIK the issue with this stuff is that the kinds of problems proofs are able to practically cover are very small.

Static analysis is an incredibly useful tool, not denying it. I am saying we should avoid worshipping at its altar. Any useful program is incredibly complicated and needs some kind of run time testing.

I don't feel like static analysis and unit testing are very different things. Given that my unit tests run roughly at compile time (ideally they would run during compile), it really isn't different than static analysis. Especially unit tests where all the inputs are mocked and expected outputs are provided.

When I wrote Ruby/Javascript, I remember needing to write unit tests that verified types of input/output variables. These kinda tests were undifferentiated boilerplate that needed to exist but didn't need to be written by me. Especially since there were many tests I'd forget to add or would intentionally not add because the test file was getting obnoxiously large.

Factoring out those boilerplate tests into the compiler (when using Java, Rust, or TypeScript) was very valuable to me, but didn't change the fact that they are basically automatically generated unit tests. The borrow checker in Rust is a similar factoring out of automatically generated unit tests, which I wouldn't have typically written.

Continuing to push more undifferentiated unit testing into the compiler/libs/proofs helps make sure I only need to write domain specific unit tests.

Why do you feel static analysis and compile time unit testing are different? or do you mean, domain agnostic testing is ok, but really we need domain specific testing?

When I wrote Ruby/Javascript, I remember needing to write unit tests that verified types of input/output variables.

When I write in dynamically typed language I never write tests like that. I'm more familiar with static than dynamic, but are experts really doing that kind of thing?

That'd be a case for design by contract.

Sure, sometimes I even deploy code without a proof nor tests ;)

Jokes aside, if my problem is small enough to be proved (for whatever value of small), I do not see any more value to be extracted from tests. Who bothers checking that 3+2 = 2+3 after proving that x+y = y+x?

I took computer language theory in college literally 20 years ago, had a sandwich with chips during every class. I clicked on that link and could taste the sandwich.

I wish static analysis were always sufficient. Curry-howard doesn't obviate the need for tests. Anything affected by the physical implementation of the computer still needs to be tested, for example. That includes corruption safety for databases, timing side channels in cryptography, all high-integrity code, and so on. Proofs can also be buggy themselves, as can the verifiers.

But yes, formal methods make testing a lot easier and taken far enough, can suffice on their own.

> Anything affected by the physical implementation of the computer still needs to be tested, for example. That includes corruption safety for databases

I think that comes under "taken far enough". If you can model the corruption in your proof, you're good. I'm less confident about timings. But you're right that testing is still useful for bugs on other levels. After all, going high enough, humans can have buggy requirements, and no proof will catch that. Tests might.

It is very ergonomic language though, when you consider how much of C++ complexity it manages to simplify.

> I'd much rather these be warnings

Delusional attitude. If it can be caught at compile time it should be; there is no reason to leave it as a warning for programmer to maybe handle.

For some schizophrenic reason, when I look at GAT proposals or macros, I rather be dealing with C++ code.

C++ can be as simple as you want.

There's unsafe for that.

There's a very good chance that in whatever language you're using you're not testing remotely close to what you'd need to in order to reach parity with that of the rust compiler. It's very Dunning-Kruger of a developer to believe that they are achieving such a level of edge case testing.

Reply on news.ycombinator.com