Hacker News Reader

Comment by simias | Hacker News Reader

This article demonstrates well something I've seen a lot with people coming from GC languages getting into Rust: they just write the code the way they're used to and work around the borrow checker by slapping `Arc<Mutex<_>>` all over the place.

Then that leads to frustration and wondering why you even got rid of the GC in the first place if you end up with a crappier non-transparent reference-counted garbage collection with all these Arc/Rc. Some devs even seem to think that non-GC devs are silly poseurs who refuse to have a garbage collector only to reimplement it manually with reference counting. We are silly poseurs alright, but we're more efficient than that!

I share the author's conclusions: don't do that. If you find yourself slapping Mutexes and Arc/Rc all over the place it probably means that there's something messed up with the way you modeled data ownership within your program.

View on HN

In fairness, "just use Rc<>/Arc<>/clone()/etc" is common advice from the Rust community in response to criticism that the borrow checker puts an undue burden on code paths which aren't performance sensitive.

> If you find yourself slapping Mutexes and Arc/Rc all over the place it probably means that there's something messed up with the way you modeled data ownership within your program.

It only means that the data model doesn't agree with Rust's rules for modeling data (which exist to ensure memory safety in the absence of a GC). This doesn't mean that the programs the user wants to express are invalid. And this really matters because very often it doesn't make economic sense to appease the borrow checker--there are a lot of code paths for which the overhead of a GC is just fine but lots of time spent battling the borrow checker is not fine, and I think Rust could use a better story for "gracefully degrading" here. I say this as a Rust enthusiast.

EDIT: I can also appreciate that Rust is waiting some years to figure out far it can get on its ownership system alone as opposed to building some poorly-conceived escape hatch that people slap on everything.

> In fairness, "just use Rc<>/Arc<>/clone()/etc" is common advice from the Rust community in response to criticism that the borrow checker puts an undue burden on code paths which aren't performance sensitive.

Yes, and I think it's good to push back on that. I personally feel it's pretty misguided. I've been meaning to write about this, but haven't... so I'll just leave a comment, heh. Someday...

A related comment I wrote a while back about this: https://news.ycombinator.com/item?id=24992747

If all you do is publish that link, it would not be a disservice to your readers IMHO.

"Just use clone" seems like reasonable advice to give people on day 1 or week 1. I guess the hazard there is that they might end up writing `fn foo(s: String)` everywhere instead of `fn foo(s: &str)`, but they can gradually learn the better thing through case-by-case feedback, and correcting this is usually a small change at each callsite rather than a total program rewrite.

On the other hand "just use Arc" is definitely smelly advice. I think going down that route, a new Rust programmer is likely to try to do something that really won't ever compile, and wind up really frustrated. Maybe we can distinguish this often-really-bad advice from the other mostly-ok-at-first advice?

Agree. Starting with .clone() is very useful for beginners, in my opinion. It gives people a chance to learn what ownership is gradually.

"Just use clone" is absolutely fine. I even put it in the book!

Yes, I guess to me they're distinct, but I can totally see how it may seem similar. I'll try to make sure to make that explicit when I talk about this, thanks!

Thanks you for that example. You really should put that somewhere public.

However, that example commits the Rust "sin" I always see with "spawn" explanations in that it packs everything into a closure.

Please, please, please ... for the sake of newbies everywhere ... please define a function and call that function from inside the spawn closure. This is one of the Rust things I spend the most time explaining and unpacking for junior engineers.

The separate function makes explicit the variables that are moving/borrowing/cloning around, which names are external/passed/closure-specific, and how it all ties together. It breaks apart the machinery that gets all conflated with the closure. They can always make it succinct again later once they understand the machinery more completely.

This gets particularly bad with "event loops" (I have a loooooong rant I need to write up about event loops and why they are evil--and it's not Rust specific) and deep "builder chains" (which I consider a Rust misfeature that desperately needs to go away).

That is a very interesting idea that I find viscerally unsettling but will consider, thanks.

It is probably because of all my years with Ruby that this seems so off to me, but also, I hear you. Hmm.

I get why its unsetlling. I hear you, too.

The issue in Rust is that "spawn" is generally the first time that a programmer is forced to confront closures--before that they can completely ignore them.

It is a quirk of Rust that someone who "just wants to spawn a thread" suddenly gets all this language overhead dumped on them in a block.

What about iterators? My instinct is that people would run into wanting to use `map` or `filter` or something before they feel the need to spawn threads when using a new language, although that might be my bias coming from a more functional background before learning Rust. The types of closures used as predicates tend to be a lot simpler though, so I guess this may not be what you mean by "confronting" them.

Map/filter closures almost always never actually enclose variables, instead just acting on the elements from the iterator. They’re basically a one-off nameless function. On the other hand, closures used for spawning threads almost always enclose variables from the outer context. That’s an important distinction that makes them seem quite different even if they’re the same underneath.

When I started with Rust, I actually thought that the common

spawn(|| …)

syntax was something special, not just a closure with no variables.

Iterators? Nope. You can happily live in "for x in foo" land without ever touching map/filter/collect.

> might be my bias coming from a more functional background before learning Rust

This is exactly the problem. The people I'm dealing with are coming from "imperative land" and haven't had a functional background. Someone stepping up to Rust as "C with memory safety" does not have any of that functional background.

Please do remember that a "closure" is built on a lot of prior abstractions. What is a "scope"? What is "variable capture"? What is "heap" and "stack"? Why does that matter here and not normally?

No programming language is only used by experts, and Rust is no exception.

The issue is that "spawn" smacks you with a bunch of that baggage all at once.

I guess someone coming from only C might not have encountered closures, but most languages have them these days (modern javascript code uses them everywhere), and I don't think they should be considered especially advanced. "How do closures work?" Was a question I was expected to know how to answer when going for my first ever junior programming job with zero professional experience (and no CS degree either).

Someone coming from C also would certainly know about the difference between the stack and the heap too, so it's a little strange to me that GP called that out

Why?

A lot of embedded code NEVER calls malloc (thus the need for no_std in Rust). Consequently, you can go a long way in embedded without really knowing the difference between stack and heap.

"Embedded programmer" does not imply "Linux Kernel Hacker".

There’s a pretty wide chasm between kernel hackers and people who have used malloc before. :) And an embedded programmer ought to have the ability to intuit (a naive) malloc on their own (you can have malloc without a kernel).

Yes! That time I wanted to learn Rust every time I saw a complicated closure my mental parser/linter crashed (from a lifetime C developer PoV).

I don't understand how a function that is defined in-place is more clear than a nice function on its own. Perhaps because it gives a sense of continuity (like in async code), but man, it's really painful to read (at least to me). In fact, if I have to write some Rust right now, all my closures would be defined elsewhere, whenever possible.

The recommended solution uses "scoped_threadpool".

But "scoped_threadpool" uses "unsafe".[1] They could not actually do this in Rust. They had to cheat. The language has difficulty expressing that items in an array can be accessed in parallel. You can't write that loop in Rust itself without getting the error "error[E0499]: cannot borrow `v` as mutable more than once at a time".

And, sure enough, the "unsafe" code once had a hole in it.[2] It's supposedly fixed.

If you look at the example with "excessive boilerplate" closely, you can see that it doesn't achieve concurrency at all. It locks the entire array to work on one element of the array, so all but one of the threads are blocked at any time. To illustrate that, I put in a "sleep" and some print statements.[3]

You could put a lock on each array element. That would be a legitimate solution that did not require unsafe code.

To do this right you need automatic aliasing analysis, which is hard but not impossible. (Been there, done that.) Or a special case for "for foo in bar", which needs to be sure that all elements of "bar" are disjoint".

[1] https://github.com/reem/rust-scoped-pool/blob/master/src/lib...

[2] https://www.reddit.com/r/rust/comments/3hyv3i/safe_stable_an...

[3] https://play.rust-lang.org/?version=stable&mode=debug&editio...

> They could not actually do this in Rust. They had to cheat.

That isn't true at all. The implementation of scoped_threadpool being in rust is all the evidence you need that they did not cheat. unsafe is a perfectly valid construct in rust that does not subvert its benefits. It just is a reversal of the defaults in other languages (unsafe by default - opt-in to safer constructs like smart_ptr).

People freak out too much over this stuff. If you are learning - feel free to use escape hatches from "idiomatic rust" but recognize that you aren't being idiomatic (and therefore may be handicapping your learning!). Its okay to use unsafe despite idiomatic rust trying to avoid it unless necessary (and even then providing safe wrappers over it). Its also okay to use Arc<Mutex> and the like if you need to just recognize there may be a better way.

unsafe is not cheating. It is part of the language for a reason, there to be used when appropriate. It is literally impossible to make a useful program without any unsafe code anywhere, because there aren’t Rust CPUs. By this definition, “Rust” does not exist.

thread::spawn uses unsafe code. sleep uses unsafe code. Is your own example not in Rust? Are you cheating?

(And yes, the spawn version isn’t optimally concurrent. The comment was long enough without getting into that. People struggle to get examples to compile before they worry about things like this. That would be next steps.)

If you need "unsafe" in pure Rust code, not to talk to something external, the language has failed you at some point.

There are a relatively small number of trouble spots. They include, at least:

- No way to talk about partially initialized arrays in Rust, which leads to Vec needing "unsafe".

- Back references, which makes some kinds of trees, and of course doubly-linked lists, difficult.

Those I've mentioned before. From the discussion above, add:

- Problems around interior mutability within arrays, where you need some way to say, and prove, "a is disjoint from b", before working separately on a and b.

These are classic proof of correctness areas. It's fairly straightforward to state the conditions that have to be proven, and usually not that hard to prove them. The RustBelt crowd seems to be working on this.

Maybe they'll come up with a way to prove out conditions like that, short of grinding away by hand in Coq. Most of those problems are within reach of a fully automatic prover, like a SAT solver.

> These are classic proof of correctness areas. It's fairly straightforward to state the conditions that have to be proven, and usually not that hard to prove them.

You're right, but current Rust cannot express logical preconditions in code, much less compiler-checkable proofs. Many groups are working on this via a variety of approaches, but it will take some time before there's a common standard for expressing these things as part of the Rust language itself.

From a language like Modula-3 or Ada point of view, unsafe should be used only for low level system programing stuff, like passing data into DMA buffers, OS APIs, Assembly and similar.

I agree with Animats here, to work around typesystem issues means that the type system still isn't expressive enough to define certain proprieties in a safe way.

> But "scoped_threadpool" uses "unsafe".[1] They could not actually do this in Rust. They had to cheat.

Using unsafe isn't "cheating". The whole point of using Rust is that you can encapsulate very small amounts of "unsafe" code within a safe abstraction, and know that there is no way to abuse the safe APIs to produce undefined behavior. The formal verification project "RustBelt" [0] has proven exactly this -- safe Rust code composes arbitrarily to produce safe behavior.

There is still a burden of proof on anyone writing a safe function that uses unsafe functionality internally. Keep in mind that "unsafe" Rust still poses more restrictions on the programmer compared to raw C/C++.

> The language has difficulty expressing that items in an array can be accessed in parallel.

The Rust borrow semantics is based on `exclusive XOR shared` references. At the first order of approximation it's not possible to mutate a value through a shared reference. Interior mutability complicates this image somewhat, but it's enough to explain why what you wrote is wrong.

Spawning `n` threads and giving each thread a mutable reference to the vector `v` goes against the above borrow semantics because that would result in `n` exclusive references. Only through exclusive references is mutation allowed. Since this is not allowed it becomes a compile-time error.

In other words, it's not that the language has "difficulty expressing" that scenario. It was explicitly designed to not allow it.

  [0]: http://plv.mpi-sws.org/rustbelt/

"Within a safe abstraction" is the question. The question is whether a piece of encapsulated code is always memory-safe for all uses. There's no guarantee of that from the language. The RustBelt people are working on tools for that, but it will probably require proof work in Coq for each bit of code containing "unsafe" to get there.

Look at the scoped_threadpools example again [2]

    use scoped_threadpool::Pool;

    fn main() {
        let mut pool = Pool::new(3);
        let mut v = vec![1, 2, 3];
        
        pool.scoped(|scope| {
            for i in &mut v {
                scope.execute(move ||{
                    *i += 1;       
                });
            }
        });
        
        println!("v: {:?}", v);
    }

There's an implicit assumption here that all the values returned from the iterator at

    for i in &mut v

are disjoint. But iterators, in general, do not guarantee disjoint outputs. An iterator which returned each reference value twice, for example, is a valid iterator. But used in the context above, two threads would receive mutable access to the same element of a vector. That seems to violate a core Rust safety assumption.

This would be a nice test to run through the version of the Miri interpreter that implements the dynamic "stacked borrow" checks from the RustBelt group.[2] That tool should catch this.

[1] https://news.ycombinator.com/item?id=24992747

[2] https://plv.mpi-sws.org/rustbelt/stacked-borrows/paper.pdf

> There's an implicit assumption here that all the values returned from the iterator at

> for i in &mut v

> are disjoint. But iterators, in general, do not guarantee disjoint outputs. An iterator which returned each reference value twice, for example, is a valid iterator. But used in the context above, two threads would receive mutable access to the same element of a vector. That seems to violate a core Rust safety assumption.

Your assumptions here are wrong. "An iterator which returned each reference value twice" could only be implemented using unsafe Rust, and code emitting multiple co-occuring mutable references to the same value are instant-UB. In other words, a "safe abstraction" being able to do this is actually not a safe abstraction at all. This would be a bug in the implementation of the custom iterator.

In fact, if you tried to prove such an implementation using the formal tools (Iris) from RustBelt you wouldn't be able to do it. I know, because I've done it as part of my course work under Lars Birkedal.

I'm not sure if you misunderstanding how `&mut v` turns into an iterator, which values it spits out etc, or what is happening. Let's look at how that code is translated into a program and typechecked. First, let's look at the for-loop itself. Anything on the form of

  for x in y

must have an implementation of the `IntoIterator` trait for the type of the expression `y`. Whatever that type might be. In this case `y` is actually `&mut v`, where `v`'s type is `Vec<i32>`. Luckily enough, an implementation of `IntoIterator` exists for any `&mut Vec<T>`. [0]

Looking at the expanded signature we can see that the associated type `Item` is `&mut T`. This tells us that the types the resulting iterator will produce are `&mut i32`. Keep in mind that this reference is into the backing memory store of the `Vec<i32>` of `v`. This is where Rust's semantic help us.

You say the following:

> But iterators, in general, do not guarantee disjoint outputs.

The issue with that statement is that iterators don't have to. Rust guarantees that condition. Safe Rust can't return multiple mutable references to the same value. Further, any unsafe implementation, even when exposed through a safe interface, giving this behavior is a bug. It would be great if the compiler could statically verify unsafe code too, but if it could there wouldn't be a need to call it unsafe.

> The question is whether a piece of encapsulated code is always memory-safe for all uses. There's no guarantee of that from the language.

That is ultimately impossible to prove completely and statically. Turing and Gödel made sure of that. However, some things that the compiler can't prove we can still prove as outside observers. Often these things will boil down to controlling and checking the states of several variables before performing an unsafe operation, but knowing that in a given context the operation is safe.

The first part of the above quote is partially true. However, if we assume a given unsafe implementation is indeed safe then the rules of Rust's semantics make the composition of any number of safe APIs a completely safe composition. This is a strong, useful statement because it means the very few places where unsafe is required can be easily checked, and every other piece of code doesn't have to spend mental energy for the programmer to consider the safety of their implementation.

  [0] https://doc.rust-lang.org/std/iter/trait.IntoIterator.html#impl-IntoIterator-19

The issue with that statement is that iterators don't have to. Rust guarantees that condition. Safe Rust can't return multiple mutable references to the same value.

So how does Vec return mutable references to elements? Unsafe code, apparently. You can't write your own safe collection class with an iterator and return mutable references. You'd need a proof of disjointness system to do that.

The question is whether a piece of encapsulated code is always memory-safe for all uses. There's no guarantee of that from the language. That is ultimately impossible to prove completely and statically.

One can construct code for which memory safety is not decidable. That's a good reason to reject it. The Microsoft Static Driver Verifier has a simple solution - if symbolic execution can't verify safety within some time limit, it rejects the driver.

> So how does Vec return mutable references to elements? Unsafe code, apparently. You can't write your own safe collection class with an iterator and return mutable references. You'd need a proof of disjointness system to do that.

If you have a mutable binding to a vector `v` you can get a single mutable reference to one of the elements of that vector at a time. Because getting a single mutable reference to an element requires taking a mutable reference to the vector the disjointedness invariant is upheld. This is the core of the Rust borrow checker. If you're unsure about the semantics you should be reading about it, not trying to sus them out from a fairly surface level discussion on HackerNews.

This code:

  for x in &mut v {
     *x = // ...
  }

is perfectly valid. For every iteration of the loop only a single mutable reference to an element exists, and the reference is dropped before the next mutable reference is taken. This following piece of code is a full example that showcases how taking two mutable references is not allowed.

  struct ComplexNonCopyClone {
      a: u32,
  }

  fn main() {
      let mut v = vec![
          ComplexNonCopyClone { a: 1 },
          ComplexNonCopyClone { a: 2 },
          ComplexNonCopyClone { a: 3 },
      ];

      let mut_one = v.get_mut(0).unwrap();
      let mut_two = v.get_mut(0).unwrap();

      mut_two.a = 4;

      println!("{}", mut_one.a);
  }

Do note that using `mut_one` after `mut_two` is important, as otherwise the compiler will infer that `mut_one` can be dropped before `mut_two` is create, removing the clash.

> So how does Vec return mutable references to elements? Unsafe code, apparently.

The implementation can be found quite easily[0]. It's essentially invariants upheld by runtime checks and knowledge about the state of the given references/pointers. Notice the comments starting with "SAFETY". They explain the assumptions/conditions/reasons that make the code inside the `unsafe` block safe. If these assumptions can never be violated with malicious input or calling patterns then the code is actually safe and it can be a safe wrapper.

  [0]: https://doc.rust-lang.org/src/core/slice/index.rs.html#149-190

We're talking about Vec's iterator, not simple slice access. That "SAFETY" stuff in slice.rs is just a subscript check. Vec's iterator is in [1]. The iterator which returns a mutable reference is unsafe code, because you can't express that concept in safe code.

To recap: the interesting issue is the thread pool example which spins off a new thread with a mutable reference to each element of an array. That's a rather unusual thing to do. It is only safe if each iteration returns a reference disjoint from all other references. Because Rust at the language level does not understand disjointness within an array, it needs a hack using "unsafe" to make this work. The borrow checker would not allow this in safe code.

chunks_mut uses the same trick, with more unsafe code.

My point in all this is that because the language doesn't have syntax for talking about disjointness of parts of an array, each time this comes up, another hack in unsafe code is required. But enough here; I may say more on the Rust forums.

[1] https://doc.rust-lang.org/src/core/slice/iter.rs.html

Clone is "okish", but I don't remember a case where I had to use Rc or Arc (sure, there are use cases for that, but not for basic stuff)

Some people just try to force their way into a new language and don't realize that if you keep doing something that looks stupid or weird it probably is (and no, yours is not a special case)

> Clone is "okish", but I don't remember a case where I had to use Rc or Arc (sure, there are use cases for that, but not for basic stuff)

I think the use case is "I haven't yet completely grokked the borrow checker and/or I don't have time to pacify it, but I would prefer not to copy potentially large data structures all over with Clone".

> Some people just try to force their way into a new language and don't realize that if you keep doing something that looks stupid or weird it probably is (and no, yours is not a special case)

You're responding to my comment which is about the Rust community prescribing this as a solution to newcomers. We're not talking about newcomers obstinately refusing to learn new idioms in the language they allegedly want to learn (although no doubt this happens, especially if the language in question is Go :p ).

> I think the use case is "I haven't yet completely grokked the borrow checker and/or I don't have time to pacify it, but I would prefer not to copy potentially large data structures all over with Clone".

For basic stuff I agree, though I wouldn't use it.

> about the Rust community prescribing this as a solution to newcomers.

There's probably a sweet spot for using those constructs in not so obvious places while going through the simpler stuff in a more idiomatic way

I hypothesize that this mostly comes from a laziness in response as it's an easy response to give to people who are used to having a garbage collector. I come from the C world (still learning Rust) and every time I see one of these pieces of advice given I'm forced to facepalm.

> It only means that the data model doesn't agree with Rust's rules for modeling data (which exist to ensure memory safety in the absence of a GC).

This doesn’t seem correct at all?

GC merely solves freeing memory when it is no longer needed. But it does not solve parallel access.

One of the beauties of Rust is that you can write highly parallel code and if it compiles it works. Meanwhile Python languishes behind the GIL. GC does not even attempt to solve multithreaded memory access.

I’m happy to be corrected if I’m wrong or missing something here.

That's a fair point.

I'd add: some GC'd languages solve this with sending around deeply immutable objects (functional languages mostly), in a way that can be more flexible than the way Rust handles immutable objects.

You're right on the local point that Rust's ownership rules guarantee correctness in the case of parallel access while GCs do not; however, the broader point is that Rust's ownership rules also reject many programs which would be correct with GC.

For example:

    fn foo<'a>() -> &'a [u8] {
        let v = vec![0, 1, 2];

        // returns a value referencing data owned by the current function
        v.as_slice()
    }

vs Go's:

    func foo() []uint8 {
        // subject to escape analysis, but code like this would likely return
        // a fat pointer into the heap--no complaints because there's a GC.
        return []uint8{0, 1, 2}
    }

> One of the beauties of Rust is that you can write highly parallel code and if it compiles it works. Meanwhile Python languishes behind the GIL. GC does not even attempt to solve multithreaded memory access.

Python's GIL is unrelated to GC, but yes, Rust's borrow checker guarantees correct parallel access of memory. But I think this benefit is overblown for a couple reasons:

1. contrary to popular opinion, if you've learned how to write parallel programs, it's not tremendously difficult to write them correctly without a borrow checker. In my experience, whatever time I've saved debugging pernicious data races is lost by the upfront cost of pacifying the borrow checker. Maybe this wouldn't hold for people who aren't experienced with writing parallel code (but I imagine such people would have a harder time grokking the borrow checker as well).

2. most data races in my experience aren't single threads on a host accessing a piece of shared memory, but rather many threads on many hosts accessing some network resource (e.g., an S3 object). The borrow checker doesn't help here at all, but you still have to "pay the borrow checker tax".

Again, this isn't a tirade against the borrow checker, but an insistence that tradeoffs exist and it's not just a "you're just doing it wrong" sort of thing.

The equivalent to the go is returning the Vec.

My example is probably not very good. Let’s say you have a function which implements a callback. The function takes a &str and returns some subslice. Let’s say you want to implement another function which implements that same interface, but returns the string lowercases. In Rust, you have to update the interface, you have to update all implementations, and you have to jump through hoops to avoid unnecessary cloning in the subslice case. In Go or other GC languages, there aren’t “owned” vs “unowned” types, so you don’t have to make any updates to the interface or other implementations. In a sense, GC allows us to abstract over ownership.

>Rust's ownership rules guarantee correctness in the case of parallel access while GCs do not

That’s a dangerous misconception. The Rust ownership model only guarantees that the program is free of data races. That’s a necessary but not sufficient condition of program correctness.

Ehhh I’d have to hard disagree with #1. But we’ll likely just have to agree to disagree.

Maybe I’m just a bad enough programmer I write parallel bugs. But C++ certainly doesn’t make it easy to write correct code in any way.

I personally think it’s pretty darn difficult to ensure correctness in a large program. Especially when multiple programmers are involved. And especially when you are adding features to an existing system.

However I will also admit that I haven’t written a large Rust program so I can’t claim to have run into all of its warts.

There’s no silver bullets in life. I work primarily in video games. GC is the bane of existence and is something that provides seriously negative value.

We definitely agree it’s all a trade off. GC provides some value and some costs. Borrow checkers provide some value and some cost.

> It only means that the data model doesn't agree with Rust's rules for modeling data (which exist to ensure memory safety in the absence of a GC).

I think my initial response was that Rust’s model exist for more than just that single reason. Whether those reasons or trade offs are useful depend on the program in question.

In my work I never want a GC, but damn would I love a borrow checker.

> Maybe I’m just a bad enough programmer I write parallel bugs. But C++ certainly doesn’t make it easy to write correct code in any way. I personally think it’s pretty darn difficult to ensure correctness in a large program.

IMO the key to writing parallel code is to keep the parallelism confined to a small kernel rather than sprawling throughout your codebase. If you try to bolt on parallelism then you’re going to have a bad time. It needs to be part of your architecture. It’s not easy, but it’s easier than writing parallel code that will pass the borrow checker IMHO. But yes, we may have to agree to disagree.

> We definitely agree it’s all a trade off. GC provides some value and some costs. Borrow checkers provide some value and some cost.

Agreed!

> I work primarily in video games. GC is the bane of existence and is something that provides seriously negative value.

I’m very curious about videogames development. In particular, I get the impression that aversion to GC in videogames comes down largely to experiences with Java back when pauses could be 300ms. I’m very curious if Go’s sub-millisecond GC (and its ability to minimize allocations, etc) would be amenable to videogame development. Thoughts?

> I think my initial response was that Rust’s model exist for more than just that single reason. Whether those reasons or trade offs are useful depend on the program in question.

Heartily agree.

> In my work I never want a GC, but damn would I love a borrow checker.

In my line of work, I like the idea of using Rust but realistically the economic sweet spot is something like “Go with sum types” or “Rust-lite”.

> I’m very curious about videogames development. In particular, I get the impression that aversion to GC in videogames comes down largely to experiences with Java back when pauses could be 300ms. I’m very curious if Go’s sub-millisecond GC (and its ability to minimize allocations, etc) would be amenable to videogame development. Thoughts?

Well, Minecraft is written in Java, and it runs fine from what I’ve heard. In .NET land, there was a short lived toolkit for C# called XNA - Terraria is (was?) written in it. Both Java and C# are garbage collected.

I haven’t looked at Unity too deeply, but isn’t Unity (and the games made in it) built in C#?

It really just comes down to average/worst case time.

60Hz - 16ms 90Hz - 11ms 120Hz - 8ms 144Hz - 7ms 240Hz - 4ms

Modern games are expected to run anywhere from 60 to 240 frames per second. 60 is the new baseline. VR runs anywhere from 72 to 120. Gaming monitors regularly hit 144Hz. And esports goes as high as 240 and even 360.

In Unity C# GC can take tens of milliseconds. This is, uh, obviously very bad. High tier unity games spend a LOT of time avoiding all allocs. This is not fun in a GC language. Most indie games just hitch and its pretty obvious. I’m not sure if Unity’s incremental GC has graduated from experimental mode.

If a GC had a worst case time of less than a millisecond that’d be fine. That’s actually a pretty big chunk of a 7ms frame, but hey probably worth it. If it’s usually 250us but once a minute spikes to 3ms that’ll cause a frame to miss. If once every 5 minutes it’s a 50ms GC that’s a huge hitch. For a single player game it’s sloppy. For a competitive multiplayer game it’s catastrophic.

Unreal Engine actually has a custom garbage collector. But it’s only for certain days types and not all memory. That’s a nice compromise. Games in particular are good at knowing if the lifetime of an allocation is short, per frame, long-term, etc.

Game programmers mostly want tight control of object layout and lifecycle. GC doesn't matter much when use ECS all over your codebase. As long as they can run the GC when they want and can layout the objects flat without needing pointer indirection it would be very suitable.

I think C# is popular because it allows the above. When Java has proper value types it might be suitable for writing games.

> As long as they can run the GC when they want and can layout the objects flat without needing pointer indirection it would be very suitable.

To be clear, the problem isn’t pointer indirection, but rather lots of objects on the heap, right? Pointers should be fine as long as there aren’t many allocations (e.g., pointers into an arena)?

> I think C# is popular because it allows the above. When Java has proper value types it might be suitable for writing games.

Go also has value types, FWIW, and they are a lot more idiomatic than in C# from what I’ve observed.

> In my line of work, I like the idea of using Rust but realistically the economic sweet spot is something like “Go with sum types” or “Rust-lite”.

AKA Ocaml.

Or F#, with the bonus of .NET ecosystem and IDEs.

Yeah, in theory, but I’ve never had much luck with OCaml and every time I dig into my problems here it ends with the OCaml fanboys berating me so I’ve pretty much given up on it.

C++ provides the tools to build robust parallelism that is also optimal, with the giant caveat that implementing it is left to the programmer (and good libraries are not abundant). Rust offers built-in correctness but not optimality, and C++ offers optimality but no built-in correctness. Many massively parallel codes and virtually all massively concurrent codes necessarily lean on latency hiding mechanics for concurrency and safety. Ownership-based safety can’t be determined at compile-time in these cases, but you can prove in many cases that safety can be dynamically resolved by the scheduler at runtime regardless of how many mutable references exist. This has a lot of similarities to agreement-free consistency in HPC, where no part of the system has the correct state of the system but you can prove that the “system” will always converge on a consistent computational result (another nice set of mechanics from the HPC world).

The problem with ownership-based safety for massive parallelism is that the mechanics of agreeing on and determining ownership don’t scale and often can’t be determined at compile-time. Some other safety mechanics don’t have these limitations. C++ doesn’t have them built-in but you can implement them.

>Rust's ownership rules guarantee correctness in the case of parallel access while GCs do not

Only for threads accessing in-memory data structures, it does nothing for other kinds of data access scenarios either to external resources or via OS IPC mechanisms.

Agreed, see my #2 above.

> very often it doesn't make economic sense to appease the borrow checker--there are a lot of code paths for which the overhead of a GC is just fine but lots of time spent battling the borrow checker is not fine

To me it's not about performance. A little bit of time spent now appeasing the borrow checker will pay off ten fold later when you don't have to deal with exploding memory usage and GC stalls in production.

GC is great for quick hack jobs, scripts, or niches like machine learning, but I believe at this point it's a failed experiment for anything else.

> To me it's not about performance. A little bit of time spent now appeasing the borrow checker will pay off ten fold later when you don't have to deal with exploding memory usage and GC stalls in production.

I'm confused by the "it's not about performance. [reasons why it is, in fact, about performance]" phrasing, but in general a lot of applications aren't bottlenecked by memory and a GC works just fine. Even when that's not entirely the case, they often only have one or two critical paths that are bottlenecked on memory, and those paths can be optimized to reduce allocations.

> GC is great for quick hack jobs, scripts, or niches like machine learning, but I believe at this point it's a failed experiment for anything else.

That sounds kind of crazy considering how much of the world runs on GC (certainly much more than the other way around). I feel the need to reiterate that I'm not a GC purist by any means--I've done a fair amount of C and C++ including some embedded real time. But the idea that GC is a failed experiment is utterly unsupported.

> reasons why it is, in fact, about performance

That's the story that GC sold us. History has proven it wrong. Citation: the fire hose of articles on HN about how GC bit people in the ass, and they now have to go back into their code and write a bunch of duct tape code to work around Garbage Collector Quirk #4018 de jure that results in hitching, insane memory usage, and random OOMs.

> That sounds kind of crazy considering how much of the world runs on GC

And much of HN runs on comments complaining about the _absurd_ amounts of memory all those non-bottlenecked applications use to do otherwise simple tasks. Or the monthly front page articles about developers and companies working to fix their otherwise straightforward, non performance critical production services that are choking themselves because the GC is going wild.

I say GC is a failed experiment because it promised that programmers would be able to write code without worrying about memory. But ever since its popularization 26 years ago with the dawn of Java, coders writing in garbage collected languages have been doing nothing but worrying about memory. The experiment failed. It's time to move on.

The borrow checker is an infantile incarnation of a bigger idea that is finally panning out. Rather than garbage collecting during run-time; garbage collect during compilation using static analysis. Being in its infancy it's not as easy and free to use as we'd like. But it's the path forward. And just like garbage collection before it, in the vast majority of cases, programmers don't care whether it's more or less performant. Garbage collection was vastly less performant than manual management. But it required _so_ much less developer time to build the same applications. My argument is that Rust's borrow checker, as painful as it is, results in more developer time up front, but less developer time overall when you consider the long tail of code upkeep that garbage collected applications demand.

Hence my comment: "A little bit of time spent now appeasing the borrow checker will pay off ten fold later when you don't have to deal with exploding memory usage and GC stalls in production."

It's not about performance; it's about saving yourself the time of having to come back to your code a month later because your TODO app is using a gig of RAM and randomly hitching.

> It's not about performance; it's about saving yourself the time of having to come back to your code a month later because your TODO app is using a gig of RAM and randomly hitching.

If your GC program is using excessive RAM, that’s because of a memory leak, not the garbage collector. This can happen in C/C++ as well; just malloc/new and forget to free/delete. Last I checked, C and C++ aren’t garbage collected languages.

It can also happen in Rust.

It's rarer, and you can rule it out entirely by just not using types that let you leak memory. Afaik, circular `Rc` references, `Box::leak` (and friends), `MaybeUninit` and overzealous thread spawning are the only ways of leaking memory in safe Rust.

Even if you avoid those things, how does safe Rust make leaks more rare than in a GC language? Presumably leaks in a GC language or safe Rust are almost always going to be stuff like “pushing things into a vector repeatedly even though you only care about the last item in the vector”, and clearly safe Rust doesn’t stop you from doing this any more than a GC.

Note also that GC languages don’t even have the circular references case to worry about since they don’t have any need for reference counting in general.

Look. If I want to use two gigabytes of my users’ memory and half a core just to display a glorified IRC app, I should be allowed to do so.

That’s my memory and battery capacity, not theirs.

I’ve been working in tech for a decade, I’ve scaled several large products and worked on distributed complex systems with a lot of users and some seriously workloads. Memory use has been a major issue perhaps two or three times total.

> That's the story that GC sold us. History has proven it wrong. Citation: the fire hose of articles on HN about how GC bit people in the ass, and they now have to go back into their code and write a bunch of duct tape code to work around Garbage Collector Quirk #4018 de jure that results in hitching, insane memory usage, and random OOMs.

I follow HN daily and very rarely do I see articles lamenting GC (I'm only familiar with a small handful of incidents including some pathological cases with Go's GC on enormous heaps (many terrabytes) and some complaints about Java's GC having too many knobs), certainly not in the general case. Indeed, for the most part people seem quite happy with GC languages, especially Go's sub-ms GC. In particular, memory usage (and thus OOMs) have nothing to do with GC--it's every bit as easy to use a lot of memory in a language that lacks GC altogether. This is incorrect, full stop.

> I say GC is a failed experiment because it promised that programmers would be able to write code without worrying about memory.

GC promises that programmers don't have to worry about freeing memory correctly, and it delivers on that promise. I'm not a GC purist--there's lots of criticism to be had for GC, but we don't need point at patently false criticisms.

> The borrow checker is an infantile incarnation of a bigger idea that is finally panning out. Rather than garbage collecting during run-time; garbage collect during compilation using static analysis. Being in its infancy it's not as easy and free to use as we'd like. But it's the path forward.

Maybe. I like the idea, but I'm skeptical that putting constraints on the programmer is going to be an economical solution, at least for so long as the economics favor rapid development over performance. Conceivably rather than rejecting code that aggrieves the borrow checker, we could picture a language that converts those references into garbage collected pointers transparently, but we kind of have this already today via escape analysis--and indeed, I think this is the economic sweet spot for memory management because it lets users have a GC by default but also minimize their allocations for hot paths.

> Hence my comment: "A little bit of time spent now appeasing the borrow checker will pay off ten fold later when you don't have to deal with exploding memory usage and GC stalls in production."

But the borrow checker is strictly less effective in preventing memory leaks than a (tracing) GC (borrow checker will happily allow circular refcounts). More importantly, having to pacify the borrow checker on every single codepath when only 1-2% of code paths are ever going to be problematic is not a good use of your time, especially when you can do some light refactoring to optimize. With respect to GC stalls, these are particularly rare if you have a GC that is tuned to low-latency (Go's GC can free all memory in less than a millisecond in most cases).

> It's not about performance; it's about saving yourself the time of having to come back to your code a month later because your TODO app is using a gig of RAM and randomly hitching.

That sounds like the textbook definition of a performance concern, but again memory usage is orthogonal to GC and random hitching isn't a problem for latency-tuned GCs. Even while Rust is faster than many of its GC counterparts, this difference typically comes down to the ability of the compiler to output optimized code--not the memory management system. That said, for realtime applications, nondeterministic GCs aren't appropriate.

The situation is a bit more nuanced than this. Yes, a GC intrinsically incurs an integer factor performance cost for many codes relative to e.g. optimized C++ (I’ve optimized both). However, while ownership-based safety models are a material improvement in some cases, in other cases the borrow-checker encourages materially suboptimal software architecture e.g. in cases where ownership can only be optimally evaluated at run-time. There are other deterministic safety models often used in these cases.

tl;dr: there are significant GC performance problems that a borrow-checker doesn’t solve. I can imagine cases where the performance improvement is significantly less than you might hope.

> Yes, a GC intrinsically incurs an integer factor performance cost for many codes relative to e.g. optimized C++ (I’ve optimized both)

I'm guessing this performance difference isn't caused by GC but rather correlated with GC. I.e., GC langs tend to output relatively poorly-optimized code relative to the absolute beastly C/C++/Rust compilers or else idiomatic code results in objects scattered all over the heap (killing cache coherency) while idiomatic C, C++, Rust, etc tend to allocate coherent objects (objects which tend to be accessed in succession) next to each other in memory.

TL;DR: No doubt a GC can be slower than manually managed memory in some cases, but it's insufficient to conclude that the entire performance gap between C/C++/Rust and Java/Go/etc is attributable to GC.

No, there are major classes of macro-optimizations that are impossible to implement with a GC even in theory. Some of these optimizations are idiomatic for state-of-the-art systems code. The impact on performance is integer factor.

You're on the peak of mount stupid

An exception: `static Mutex<RefCell<Option>>>` is a great way to store non-copy data structures, for sharing across interrupt contexts in embedded.

I'm ambivalent about it.

In my opinion, Rust is optimized to be a systems programming language. As such, we'd expect that it shouldn't really be your first choice for writing an "application". (I'm not referring to the OP at all here)

HOWEVER, Rust is such an excellent language that we all want to use it to write applications ANYWAY. That's kind of amazing in and of itself- that people want to deal with a non-GC'd language at all to write applications. Because, really, garbage collection is awesome and there's almost zero reason to avoid it unless you need extremely predictable performance, or very low runtime overhead, etc.

As far as wrapping everything in Arcs and Mutexes is concerned: Yes, that's ugly and it's a lot of extra typing. On the other hand, your performance is still likely to be orders of magnitude faster than Python for some general application-type tasks, and it will likely avoid headaches with the borrow checker, etc.

So, honestly, I don't know if I recommend doing that or not. Want I want is to be able to tell people to not use Rust for what they're doing. I'd like to have a different recommendation for a "garbage collected Rust" but there really isn't anything that I think is good enough for the title. Maybe Scala 3 (I haven't played with it yet) or OCaml when its multi-core stuff is done. Maybe F# or C# are good enough, too.

The problem with OCaml is not multicore. If it had an easier to use build system and a larger community (so more libraries), it could be great for server-side software. But dune is way harder to use than Cargo and you can often miss libraries. Here's an example of Dark moving away from OCaml to F# https://blog.darklang.com/leaving-ocaml/, which doesn't have as much this problem because you can use C# stuff. Scala also doesn't suffer as much from it, part of it because it was really popular at some point, and part of it because you can use Java anyways.

I hope at some point we'll have this "garbage collected Rust", but for now OCaml, Scala and F# all have a worse developer experience than Rust. I could add Haskell to that too.

I've been thinking for some time that something like Go but based on a ML would fill this "garbage collected Rust" niche quite well. Maybe something built over Rust itself to leverage all the ecosystem? You could write all of the glue code in this language, have access to a large ecosystem of libraries and have an option for high performance code. This would also complement Rust nicely: I know that OCaml has a really fast compiler, which would be a breath of fresh air for the community.

I think you're talking about Swift. Alas it doesn't seem to have caught on outside of the Apple ecosystem, so library support isn't great there either.

I do agree that language built on top of the Rust ecosystem (kind of like how PHP exposes a lot of C libraries) could make a lot of sense.

I don't think I am talking about Swift. The compilation is slow because of LLVM and you still have to handle cyclic references because of ref counting. As you said too, library support isn't great. I don't know how exactly is the dev experience too, at least outside of XCode. Last time I played a bit with it, adding the language server to VSCode was a bit of a pain. Still, I think Swift is a step in the right direction.

I've heard good things about Nim too for this niche but never tried it. I think it'll suffer from the same problem of not being able to leverage an existing ecosystem though.

Nim compiles to C (through C?), so at least leveraging the C/C++ ecosystem is straightforward/first class.

It is certainly difficult to justify breaking away from the huge established multiple decade ecosystems and standard libraries seen with the JVM/.NET/Python platforms for instance.

https://nim-lang.org/docs/backends.html#interfacing

OCaml is also pretty good at leveraging the C ecosystem, but that's not what people are looking for in a "garbage collected Rust". If they want something close to C they'll just use Rust. What they're looking for is something easier to write that still looks like Rust and having all the developer experience that Rust has. For this, the Nim ecosystem is way too small and can't leverage an existing platform. The same do apply to OCaml and Swift so it's not a Nim specific problem, it's just that Nim doesn't bring much compared to OCaml.

For what it's worth, OCaml is a great under-appreciated language forever dented by ugly (in my opinion) syntax and oddities like different operators for floats and ints. There's a good reason the folks behind the Reason ML fork changed things up.

Swift is another great language - forever dented by being tied hand and foot to the Apple holy garden and Objective-C legacy and runtime with memory-related keywords such as 'weak' and 'unowned'.

Nim (and perhaps D) are the closest in my mind to the "garbage collected Rust" in that they are not only garbage collected, but they are procedural-oriented, can write highly generic code and macros, and come with near-C performance. Given the "Pythonic Rust" title, another edge Nim has is its semantic whitespace, very Python-esque syntax.

There are/were many less staunch languages that interoperated with Rust in a clean way.

    * https://gluon-lang.org/
    * https://github.com/PistonDevelopers/dyon

Rust is waiting for that interior language so that everyone doesn't need to subscribe to the same complexity budget.

I've looked a bit at Gluon before and it was a bit too much like Haskell for what I imagined. I don't think using monads for IO is a good idea if you want to keep the Rust approach of pragmatism first.

I don't know anything about dyon but a quick look tells me it's more meant for game development.

> I hope at some point we'll have this "garbage collected Rust", but for now OCaml, Scala and F# all have a worse developer experience than Rust. I could add Haskell to that too.

F# on Visual Studio + .NET ecosystem (which includes C++/CLI, C# and first class support for COM/UWP) is worse than Rust?!?

How many of those things work well on Linux/MacOS? Rust, Cargo, rust-analyzer, VSCode work on every major plateform. You also have access to the .NET ecosystem but it's not as easy as the Rust ecosystem since most of the packages are C#, not F#, and idiomatic C# and idiomatic F# are quite different. In terms of resources too, F# is quite poor compared to Rust.

Edit: another thing, compiling to native is not really a first class citizen on .NET. That makes it a bit worse to distribute tools compared to Rust, Go or OCaml. Scala also suffer from this.

F# can just use C#, that is the magic of CLR.

VScode F# plugin and Rider also have much more resources into them.

Then 20 years of production code to chose from.

You can use C# from F#, of course, but then you can run into issues with nulls and things like that. There's a reason every programming language has their own implementation of JSON/XML parsing, HTTP and things like that: offering an idiomatic API is important.

Last time I use the VSCode F# plugin (around a year ago), the experience was strictly worse than rust-analyzer. The compiler errors are also not as good. I've never used Rider so I can't comment on that.

You're right about the 20 years of production code for C#. NuGet lists 241418 packages compared to crates.io 61579 crates, and there's a good chance most of them are more mature than the Rust alternatives. However that's C#, not F#, and I doubt F# represents more than 10% of the packages. That's still better than doing it yourself if there's no package or writing bindings to C though.

I still think F# and Scala are solid choices if you want a functional language today, but it's not the "garbage collected Rust" ideal I wish we had.

> HOWEVER, Rust is such an excellent language that we all want to use it to write applications ANYWAY.

Is this really the reason, or is it because people think they will get free performance wins by choosing the correct language? The plethora of articles detailing a developer's story trying to write X-style language in Rust shows that most are not approaching Rust with the correct mindset.

I have limited experience is Rust, but seems like most people are attracted by shiny new toys in the language. Many people have issues with OO languages, and the separation of data and function is attractive. But it lacks the run time most application developers have come to know, specifically GC and easy references.

I'm sure there's a bit of both. Many people talk about how they love the tooling, the type system, enums, traits... stuff unrelated to performance.

I can only speak for myself, obviously. But, I think it's obviously about more than performance. Otherwise, people have always had C, C++, LISP, and even Java and Go that will be generally faster than Python, PHP, and JavaScript.

I'd say there are three things that most people rave about with Rust:

* traits as type classes (even if they don't know the term "type class") * enums and pattern matching (not even as flexible as some other languages like MLs, but way more than is possible in most popular languages with only Swift coming close) * the Iterator trait. I might write a sonnet about how much I love Iterator in Rust. It's lazy iteration, but it's also optimized at compile time to basically just be turned into a for loop.

> I'd like to have a different recommendation for a "garbage collected Rust" but there really isn't anything that I think is good enough for the title.

D and Nim are both good candidates IMO.

Could you expand on that? They both seem like nice language, but why them over other proposed languages such as OCaml, Scala and F#?

Not the person you replied to, but OCaml, Scala, and F# all suffer from the potential drawback of being functional-first, and thus difficult for people to adapt to. If you are someone who is an OO programmer and are considering Rust for its potential performance and different model, it would be hard for me to recommend a language whose model is as different as OCaml's. Learning Rust entails learning the borrow checker and unlearning some of your OO habits; learning OCaml requires a sea change in your whole mentality & approach to programming if you've never experienced a functional lang before.

Meanwhile, both D and Nim have the benefits of being procedurally grounded, and hence more familiar to someone coming from (say) C# or Java.

Having said that, I think both of those languages may suffer from a lack of tooling/library breadth & depth. Nim in particular seems really cool, but it seems to be struggling to catch enough attention & interest to build a larger community -- probably in large part because it targets many of the same people who are currently high on Rust.

I don't really agree here, even if they're not used much by the community OCaml can be written as a regular procedural or OO language. There are a few thing that can look weird for people new to this (immutable by default, the type system) but they're precisely what people looking for a GC'd Rust want I think. Rust also takes a lot from functional languages, and idiomatic Rust code usually looks like functional code. I think the biggest shock coming from Rust to OCaml would be the compiler errors and the quantity of online resources. There's a world of difference between the two, and they matter a lot when learning. I think the same is true for Scala. I don't know if F# is the same but since it takes from OCaml it may still have the procedural capabilities, and it seems it still has the OO features of OCaml.

I do agree that D and Nim may be easier to approach from C# or Java, or even Python for Nim or C and C++ for D. These already represent a huge chunk of programmers, but not really what the typical person looking for a "garbage collected Rust" is, in my opinion. Especially since Java and C# programmers already have Scala and F#. While they're both different from what they're used to, they leverage the same plateform and can use the same libraries which is a huge plus.

I agree with your conclusion too, and I think OCaml suffers from this too. Scala and F# have at least the libraries part covered.

> I'd like to have a different recommendation for a "garbage collected Rust"

I have exactly the same problem. Rust is nearly perfect to me, due to its non-nullable type, traits, package manager. It's just that losing garbage collection is not justifiable in my use cases. I've researched several languages but nothing "clicks" unlike what Rust did to me several years ago. C# might be my best bet but it lacks statically compiled binary. Go is my next bet but it lacks a sophisticated typesystem. Kotlin is my third bet but the Java toolchains are slow and too "enterprise-y". Swift is too Apple-centric and has weird corner cases like you can't change the order of the keyword parameters, due to its Objective-C origin. Life is full of trade-offs, I guess!

Seems like c# is gaining lots of traction on AOT stuff very soon.

This is also a problem with Rust conversations and tutorials online: There's a lot of talk about what not to do or what you can't do in Rust, but it's rarely followed up with an explanation of what users should do.

Mutexes aren't an entirely foreign concept to many programmers. Obviously if someone can architect their system in a way that doesn't require Arc<Mutex<_>> then go for it, but we need to be careful about giving blanket advice without alternatives.

Indeed, this blanket advice does not apply in situations where shared resources that require mutable access need to be synchronized. If my async HTTP router is serving two requests that are using the same database client to persist changes, that needs to be synchronized. How does one do this without a mutex?

Use a database pool, although some pool implementations may use mutexes internally.

By acquiring and releasing resources. I’m not intimately familiar with Rust, so I’ll use Go as an example.

Create a one-buffered channel: `make(chan T, 1)`. To acquire, receive from the channel. If the object is in use, the receive will block (goroutine put to sleep) until it’s available. To release, send the acquired object to the channel.

On a language level, no mutexes.

There are STM (Software Transactional Memory) implementations in Rust.

The lifetime/borrow-checking is the mutex. If you've got a database client that must only be used by one caller at a time, make a function that takes the client as &mut.

I think the broader problem is that: one _cannot_ just arbitrarily “go” from a higher level language to a systems language.

Folks really ought to read the C/C++ literature to understand why Rust evolved in a unique direction. That gives better compare/contrast.

Also, a lot of Python idioms are actually from C++ (e.g. mixins, iterators, operator overloads, etc).

Anyway, to reiterate: folks interested in systems languages should read the C/C++ literature and actually become a systems engineer first.

>Folks really ought to read the C/C++ literature to understand why Rust evolved in a unique direction. That gives better compare/contrast.

Sounds like a good idea for the post-singularity age of infinite lifespans, but what folks really need to do is learn what they need to learn as best they can when they need it.

Maybe if one just wants to scrape by. I wouldn’t want systems code from those folks to be honest... Maybe that’s fine up the layers but the whole point of Rust is correctness and stability—to get the that point, maybe invest a month hitting the books?

Whenever I see the word "pythonic" I am cautious. Not all languages are Python, nor should they all become Python, nor are they worse for _not being Python_.

Python developers try to write idiomatic Python in whichever language they use.

Learning the idioms and conforming to a new language are hard, especially at the beginning, and people are lazy.

Python has a reputation as "executable pseudocode", a language that makes it easy to express the essence of what you're actually doing without clutter or ceremony. That's a worthy design goal for every language IMO.

> a worthy design goal for every language

I suspect that every language in the world can be clear and direct. I am sure that some people can write unintelligible pseudocode.

We don't expect speakers of other languages to write more Englishonic phrases, although of course that does happen. (English speakers often criticise people for not speaking good English.)

Every language is readable by and intelligible to people who know the language.

Pseudocode is a useful concept. There is no "correct" pseudocode because it is meant as a DSL for _thinking_. Any non-trivial implementation of thinking will quickly become more detailed and more subtle in notation than a shopping list (for example). Opinions about the subtlety of notation abound.

I am not aware of any high-level language that makes a shopping list hard to understand. Above that, we are deeply into opinion about what defines "clutter", "ceremony", and effective notation.

We pretend all human languages are equal for political reasons, not because it's reality. I would hope that programming language preference is still neutral enough that we can talk about what things different languages do better or worse. There are plenty of things wrong with the Python ecosystem, but the language syntax is widely known as a success.

> We pretend all human languages are equal for political reasons, not because it's reality.

Hmm, what is the "non-political" reality in the question of equality of languages?

> the [Python] language syntax is widely known as a success.

There are other successful languages, too.

> Hmm, what is the "non-political" reality in the question of equality of languages?

Human languages aren't equal; plenty of languages have problems either in specific areas or just generally, and any academic linguist can tell you (off the record) which.

> There are other successful languages, too.

Python is probably the most successful teaching language; I can't think of another language with a better reputation syntax-wise. (And if another language has such a reputation, it's probably because that language is also "Pythonic").

Just want to point out that Rust and Python live on opposite sides of the spectrum. A systems language requires good understanding of systems (whether rust or c/c++) and also some assembly language. So Pythons “ease of teaching” actually could yield bad systems engineers, because it shields users from the system...

Honestly, I would personally teach plain C for algorithms because it has the simple for loops and while loops, tail-call recursion, arrays and structures.

Python has a kitchen sink.

> Pythons “ease of teaching” actually could yield bad systems engineers, because it shields users from the system...

Not necessarily. Most of what it "shields" users from is unnecessary ceremony and clutter rather than relevant system behaviour.

> Honestly, I would personally teach plain C for algorithms because it has the simple for loops and while loops, tail-call recursion, arrays and structures.

C's loops are not simple, C implementations do not generally have tail calls, C makes the distinction between arrays and pointers far too subtle (which not only confuses learners but also causes bugs in real code), C's structure support is bad (primitive unchecked unions, but no support for proper disjoint sum types). It may have been useful as a portable assembler at one point (back when it was compiled in simple fashion), but it's not a good language for anything these days.

> any academic linguist can tell you (off the record) which.

Hmm, once again you allude to unspoken truth. What is your understanding of this unspoken truth about which languages are "better"? I assume that if you hold an opinion, you can present it clearly without vague references to unnamed sources.

> Python is probably the most successful teaching language; I can't think of another language with a better reputation syntax-wise. (And if another language has such a reputation, it's probably because that language is also "Pythonic").

In pseudocode, what you have written is: I assert P because I think P is best and prefer P; also, any other language that meets my requirements is P.

> I assume that if you hold an opinion, you can present it clearly without vague references to unnamed sources.

I don't want to get banned.

> In pseudocode, what you have written is: I assert P because I think P is best and prefer P; also, any other language that meets my requirements is P.

Your pseudocode has a type error.

I don't much care for Python myself. But I believe languages with reputations for good syntax are Python-like. If this isn't so, it should be easy to provide a counterexample: a language that has a reputation for a good syntax that isn't Pythonic.

> I believe languages with reputations for good syntax are Python-like.

"Only syntax like P has a good reputation". That is your opinion and I uphold your right to hold it. I would however like to understand it better.

Given this dependency on the "reputation" of "pythonic syntax", whom do you accept as "recommenders" for reputation? What constitutes a "good reputation" to your mind?

Do the importance of project, or perhaps durability of code have any value in this reputation?

Is it possible that more detailed notation has a purpose, or is it always "clutter" because it is not "pythonic" or "Englishonic"? Math notation itself is eloquent in the extreme, yet it is not "pythonic" and certainly not "Englishonic".

Is it enough that millions of other people use different tools? Literally billions of other people speak and write a language that does not have the "Englishonic" properties of English, for example. Some of these languages have a notation that is superior to English. [see GB Shaw on English notation]

You say quite clearly in another comment that "C" is not a good language, despite its importance as a language and its influence on several other important, long-serving programming languages. I will just cite the TIOBE index as something tangible; if imperfect, it is at least not a mysterious allusion.

Take one of these languages. Javascript, for example, is a very successful language. Apart from the "ecosystems" of languages, I assume you accept that the notation of Javascript is -- by any measure other than aspersions -- a successful notation system.

> Given this dependency on the "reputation" of "pythonic syntax", whom do you accept as "recommenders" for reputation? What constitutes a "good reputation" to your mind?

Do you actually disagree with me about the reputation of different language syntaxes? I don't have a specific list of influences, just a general impression from e.g. HN-like discussion sites, programming meetups, work colleagues...

> Do the importance of project, or perhaps durability of code have any value in this reputation?

No - we're talking solely about syntax.

> Is it enough that millions of other people use different tools? Literally billions of other people speak and write a language that does not have the "Englishonic" properties of English, for example. Some of these languages have a notation that is superior to English. [see GB Shaw on English notation]

I see your last two sentences as the demonstration that the answer to your question is "no". A language may become very popular despite having a very poor syntax, and even clear improvements to syntax are often not adopted.

> You say quite clearly in another comment that "C" is not a good language, despite its importance as a language and its influence on several other important, long-serving programming languages. I will just cite the TIOBE index as something tangible; if imperfect, it is at least not a mysterious allusion.

Sure, and I'm aware that this is a controversial view. I think few would defend C's syntax; rather they tend to claim that it has good performance or is close-to-the-machine (views that I disagree with, but would acknowledge the popularity of, and be prepared to argue my case against).

> Take one of these languages. Javascript, for example, is a very successful language. Apart from the "ecosystems" of languages, I assume you accept that the notation of Javascript is -- by any measure other than aspersions -- a successful notation system.

I don't accept that a language being popular means it has good syntax, if that's what you're saying.

Your notion of "pythonic" syntax being superior remains vague. You haven't clearly said what non-biased measure supports your claim.

"Pseudo-code" itself can be anything, as it is a term coined for a loose notation of ideas.

If your objection were "line noise" (i.e. non-Pythonic or non-Englishonic characters), the bias in this is striking. There are other languages that are as effective or even better. The fact that we may not understand these other languages is not an elevated argument for converting them all to a language that we do understand, except perhaps as a study exercise.

As far as English itself is concerned, it is a disastrous confusion of phonemical artefacts. People learn to speak and write English despite the major problems with the writing conventions.

You say "reputation" is your measure, but this is nothing more than "group opinion". You responded that professional popularity is not your measure, so C, Java, and Javascript do not receive your approval. Yet they are successful languages. They are not perfect, but neither is Python.

As far as human languages are concerned, your position is untenable. A fluent speaker speaks $language and understands $language in its subtleties. A person who does not speak $language understands little (or perhaps nothing) and is either trying to learn or has installed opinions instead of knowledge.

> You say "reputation" is your measure, but this is nothing more than "group opinion".

One could say the same about e.g. the scientific consensus on a given topic. Ultimately anything nontrivial in today's world relies on other people.

> You responded that professional popularity is not your measure, so C, Java, and Javascript do not receive your approval. Yet they are successful languages.

They are popular but not for their syntax. Their fans and advocates largely admit as much. Plenty of professionals will say things like "I use language X despite its cumbersome syntax, because ...".

> As far as human languages are concerned, your position is untenable. A fluent speaker speaks $language and understands $language in its subtleties. A person who does not speak $language understands little (or perhaps nothing) and is either trying to learn or has installed opinions instead of knowledge.

And yet it's possible to be fluent in multiple languages, and also to study languages in an objective way without being fluent in them. Linguistics is a legitimate field of study with a wide body of existing research (computer language design, on the other hand, has not yet reached that level of maturity).

> Folks really ought to read the C/C++ literature to understand why Rust evolved in a unique direction. That gives better compare/contrast.

Unfortunately the C/C++ literature is dense and not at all approachable. The Rust literature is much, much better for newcomers to systems programming (partly because it doesn't have to cover a load of weird failure cases that simply don't exist in Rust).

> one _cannot_ just arbitrarily “go” from a higher level language to a systems language.

I mean, it's like learning anything new. You have to do a bit of unlearning, and grasp the core concepts. I don't think there is anything especially difficult about systems languages. I had a background of JavaScript and PHP, and I was able to pick up Rust well enough to use it in my day job in a couple of weeks.

C/C++ has been around for a while and there is gobs of literature. For this I’m guessing you mean “the website literature.”

Many developers these days scarcely read books on subject matter. But if they did read books, they would find there’s actually more C/C++ literature and established best practices than there is for Rust (naturally).

The effective C++ series comes to mind as something not just approachable, but enjoyable and largely insightful.

The truth is... systems engineering is a discipline which requires an understanding of historical context. You get that with the C/C++ books! I highly recommend reading on it to better appreciate Rust.

Much of Python's influence is actually from Modula-3, not C++. The only reason C++ is similar is because C++ also borrowed liberally from Modula-3.

In particular, Python's import, exception handling, and object systems are based on Modula-3's (though IIRC it did also borrow some C++ innovations for the object system).

(Modula-3 is the Velvet Underground of programming languages: your average programmer/listener has never heard of it, but it influenced so many languages/musicians you have heard of)

>Anyway, to reiterate: folks interested in systems languages should read the C/C++ literature and actually become a systems engineer first.

This is an extremely weird take. So before I can code some rust I should first get a new career and master C/C++? Talk about barriers of entry and gatekeeping. I think I may have toyed around a bit with rust - because its fun, and as I am not a systems engineer (just an other type of engineer) I am deeply sorry.

Couldn't have summed it up better if I tried. Rust is an awesome language but at the end of the day it's a systems language. As such it's not particularly suitable for writing just any type of application. Seems it shares one of my many problems with javascript: The fact that everything is being written and re-written in javascript(even if there are different reasons for both). As I've said a million times, "just because you can, doesn't mean you should". The endless Arc<Mutex<_>> is a symptom of it. In larger application this turns into an endless hell of read/write locks and sooner rather than later it blows up straight into your face. Which is further amplified by the somewhat recently introduced async/await. And let's not overlook the many incompatible tokio versions and the fact that tons and tons of crates use different versions of tokio: Just as everything looks buttery smooth, BOOM: thread 'main' panicked at 'not currently running on the Tokio runtime.' Damn it... Awesome... Which one is it this time...

On the subject of python, I think Rust has a very powerful niche it could take: slightly modifying an old Google philosophy(which they can't stick to anymore for a million and one reasons of course): "Python where we can, Rust where we must". To my mind it could be a very pleasant recipe for a large number of startups.

I come from C# and Rust absolutely drives me bonkers. I try to reference all the things all the time, cause its ingrained in my brain as more efficient than copying stuff around or/and requesting things from hashmaps. The first thing I've tried to implement was a string interning module and I was stuck on it for three days (I still don't know how to do it without duplicating each string twice). Rust is completely orthogonal to the concept of a nice garbage collected graph of objects.

Maybe check out one of the existing string interning modules? There are some good ones. I believe it's common to internally use indexes instead of references to get around the borrow checker in these kind of situations.

That's my impression with Rust for now - you use indexes instead and borrow as needed. Long lived borrows are a no go, I was shocked to figure out you cannot return a new thing and a reference to it from a function.

I’ve never tried that so i don’t know the challenge but my first thought was that it doesn’t seem incompatible with rust’s ownership model - assuming an interned string has static lifetime and then hand out immutable references?

I wanted to drop strings which are not used anymore, you do need Rc for that (no way around it I think), but my problem was that I also needed that strings as a key to HashMap<String, Weak<String>>. So I use that HashMap to check if the string is interned and give a new counted reference as needed and also periodically check any weak refs that should be dropped. This works, but the key better be immutable (I think?) so you cant do HashMap<Weak<String>, Weak<String>> (HashSet?) and hence need second copy as a key.

That's an interesting case. This won't solve your whole problem, but you should be able to make the key the hash of the string and then have a HashMap<u64, Weak<String>>. I feel like there's a pretty solution where you stick the whole map in a Rc and stick that in every interned string. Then strings can remove/lookup themselves. To make that work you'll want some sort of concurrent hash map.

But how would you handle hash collisions if the value is just a Weak<String> rather than some kind of nested container type?

I started thinking along the lines of an arena allocator, that would allow a shorter lifetime than static so you could remove unused references by dropping the arena and moving over still live references to a new arena. I guess arenas the wrong term with that behaviour, maybe generational mark and sweep garbage collector is closer to describing this approach.

It a avoids the double reference though of the Rc<> model.

You'd need to pick a hash that won't have collisions within your expected use.

Edit: this might be interesting to you: https://docs.rs/crossbeam-epoch/0.9.4/crossbeam_epoch/

What about using a HashSet<Rc<str>>. And returning a type that implements Drop. The drop method can remove the string from the cache when the last reference is dropped. Since the cache must be accessed by the interning function and drop, I used interior mutability. By putting the cache behind a Rc<RefCell<>>.

https://play.rust-lang.org/?version=stable&mode=debug&editio...

With the same principle, we can use a static, arc and mutex to use a the cache as a global variable. And implement thread-safe string interning. Here, locking the cache at the wrong time during the drop can lead to a race condition.

https://play.rust-lang.org/?version=stable&mode=debug&editio...

To make the InternedStr behaves a bit more like &str or an immutable String, I implemented Clone, Display, Pointer, conversion from &Interned to &str. Ideally it should probably resemble how String de-references to &str for everything (Deref<Target = str>).

On this very specific point, for this specific problem, I think that `Rc<str>` is criminally underused. It's a great type if you need a heap allocated string, but want to hold on to copies of it in various places; and it derefs to `&str`, so you can use it transparently with most of `std`.

I have a similar experience. I initially naively thought that implementing data structures would be a good exercise in grasping syntax and ownership. It's usually one of the earlier exercises I do in learning a new language.

However data structures in Rust are not at all straightforward. Especially implementing traits, such as an Iterator for a Hashmap. And std implementations use loads of unsafe so beginner me became increasingly confused as to whether these things were not expressible in safe rust.

> I try to reference all the things all the time, cause its ingrained in my brain as more efficient than copying stuff around or/and requesting things from hashmaps.

Many people don't realize that actually referencing all the things all the time is LESS efficient than copying stuff around on modern architectures, unless you copy really big things. This is because the majority of improvement of memory access in the recent years is seen in sequential speeds but not latency.

> I share the author's conclusions: don't do that. If you find yourself slapping Mutexes and Arc/Rc all over the place it probably means that there's something messed up with the way you modeled data ownership within your program.

That's just not a very useful statement to a beginner. Rc/RefCell and Arc/Mutex have their use, for data with patterns of lifetime or mutability that cannot be derived at compile time based on program syntax - something that comes up all the time in practical programs. Of course one should seek to refactor these things out whenever possible (among other things, programs that use shared data heavily are also harder to survey wrt. correctness), but to say as a generalized statement that there's no case for this feature is just not correct. The feature is also intentionally heavy on syntax so there's no missing where you're using it.

Yup, definitely a frustration I have reading through rust tutorials as well. So many of them will get out of memory problems by "Oh, just add Arc around it".

I’m new to Rust, and definitely guilty of all this :) do you know of any good resources which discuss transforming your data model to align with the borrow checker? Or is this just something that comes naturally with time?

More of the latter right now, but things like “data oriented design” can help nudge your thinking in the right direction.

I would love to see more writing on this.

Reply on news.ycombinator.com