Why Rust closures are somewhat hard

I think the OP makes it clear that this nuance is actually quite tricky in a native systems language like Rust.

A good example in C++ would be:

- A function (non capturing/lambda) is created with

  auto f = [] { ... };

- A function (capturing/closure) is created with

  auto s = "...";
  auto f = [&] { s; ... };

In source code both of those look similar and you can even define them with the same type: std::function<void()>

However, the resulting assembly for both couldn't be more different. The Lambda case is as raw as any normal C function type, which the closure case creates a C++ class that closes over the outer method's state. If you're not a seasoned C++ engineer, this nuance will be lost on you.

It's fair to say that everyone should understand Lambda Calculus rules, but that makes the language less accessible to new-comers. In the case for non-systems languages, you can blur the lines with ease, but something Rust and C++ just cannot afford to do. That makes understanding the nuance important to be effective.

Not sure I agree with "relatively clean code" when one of the examples show "fn compose <T>(f1: impl Fn(T)->T, f2: impl Fn(T)->T) -> impl Fn(T)->T {" which is just a mix-match of keywords and other things, with tons of syntax embedded in just one line.

But as always, depends on where you come from. I mostly deal with lisp languages nowadays, so guessing it's just my view that the line quoted above seems complex enough to not be interested one bit in Rust.

> The fact that you can even do stuff like this in Rust is amazing.

Stuff like this? You mean closures? That's what you find amazing? What is considered ordinary in other programming languages is considered amazing in Rust. Amazing.

> It’s simultaneously: relatively clean code, very efficient and type-safe. I like it.

You like it? Do you work for mozilla? If there is a tech evangelist of the year award, I will vote for you. I've never seen someone take absolutely nothing and try to spin it into something positive.

Edit: Instadownvotes by the evangelists. I like it!

In general, it's not the safety alone, but the combination of safety and eschewing implicit runtime costs. It would be 100% safe for Rust to always implicitly box closures; it just wouldn't be as efficient in the cases where boxing them isn't needed.

It seems to me that it's not that Rust is hard, it's that all programming is hard but Rust exposes the complexity so we can make a more informed decision about the trade offs of safety and performance etc, while other languages tend to hide the complexity (null reference, race conditions, etc).

In my experience that’s true. But once you code in it consistently for some appreciable amount of time, you start to think in ways that the compiler will be happy with and that make sense from a safety standpoint. It’s getting to that point that requires a bit of work.

> You just limited the set of programs you can write to those that can be written in the block-lifetime-scoped manner, which is smaller than the set of good programs (see: pretty much any graph-heavy data structures).

Given that most of my Rust programs have involved graph-heavy data structures, I'd like to see you explain why it's impossible for me to have written what I have written.

"You just limited the set of programs you can write to those that can be written ..."

Yes! That's awesome! I want the set of programs I can write to be small, but still large enough to contain correct programs.

For example, I really like statically typed languages! But what is static typing? It's rules that reduce the set of valid programs. In a dynamically typed language, 'def double(x): x*2; double(os.Open("foo"))' or other such nonsense is a valid program that errors out at runtime reliably. In a statically typed language, that program is not valid. Woo! That's a whole class of programs that do bad stuff which aren't valid anymore!

If rust restricts the set of valid programs to those that structure the ownership of data more rigidly, I'll take it!

I do realize that rust has escape hatches that let you model those things when you need to, but I'll gladly take a language that pushes programmers away from difficult to reason about data-ownership-models even if it doesn't prevent all such cases.

If you want a garbage-collected language, there's already literally a thousand of them out there. Including dozens that are more popular than Rust.

Every GC is a trade off. You can't replace RAII with GC without losing some control. No matter how much you innovate in it, you have to pay with something. That's the whole point of GC after all.

Here is a good read about it: https://blog.plan99.net/modern-garbage-collection-911ef4f8bd...

Gargabe collection is a much bigger crutch. Once you use it your entire program is contaminated by the performance characteristics of a single improperly written function. You cannot mix real time code with non real time code. If you do your real time code will be delayed by the non real time code. The unpredictability of the GC has become a property of your entire application.

First of all, Andrei Alexandrescu is not the creator of D.

> With Rust, programmers have to spend most of their mental energy worrying about management of memory, which has largely been automated already.

This is a common misunderstanding. I think the word "automated" is misleading, and suggest that we should look at two phases of memory management: the plan and the execution.

In traditional low-level languages like C, you plan how to manage memory and you write code to execute that plan yourself. Your code may not match the plan, thus all sort of problems ensues.

In languages with GC, you are allowed to not think how to manage memory and GC will execute their own inferred plan... until it is not. At some point you are forced to think how to manage memory and also tune GC and/or code to fit to your belated plan. This point may not be reached (simple scripts) or can be delayed much further (good modern GC), of course, so it still remains very useful.

In (a typical use case of) Rust, you plan how to manage memory but give that plan to the compiler to check if the plan is executed correctly. The compiler also generate code for common executions that you don't have to. So the execution is mostly automated (or rather, delegated), while planning is not.

A common theme here is that you can't avoid at least the planning once you've got past a threshold. If GC would supplant even that planning, you won't need value types at all; isn't that what the generation hypothesis is all about? But many languages with GC but not value types now get them. While it can be argued that Rust's affine typing is not sufficient to capture some common memory management patterns [1], that fact doesn't diminish the value of explicit memory management planning, and thus Rust's.

[1] And I think it is actually true! Indeed, I believe affine typing plus GC can be actually better than affine typing or GC alone (and value types mimic a good subset of this). There are some Rust libraries for tracing GC (e.g. [2]) as a starting point.

[2] https://github.com/Manishearth/rust-gc

I think Gluon is a good example of how Rust can be 'bulked up'.

The idea is to have a DSL inside Rust that essentially has haskell semantics. This allows you to write performance sensitive code in Rust and then marshall it into Gluon where you can do high level manipulations with type safety before you unmarshall into a performance sensitive section again.

To me the failure of Rust is more that it didn't go full-on-haskell syntax wise and (to a lesser extent) that there isn't a specification (or some standards) beyond the test suite.

> You just limited the set of programs you can write ... which is smaller than the set of good programs

That's true of all static type systems.

Also, the whole point of Rust is to cover cases where performance and memory management are important. If you can use a GC'd langauge for you program, then pick one of the many fantastic existing ones.

There are indeed some nuances you are missing as well as at least one misconception you are holding:

> The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd)

You can't touch variables that have been garbage collected, indeed. A variable captured in a closure will have a reference to it and therefore not be garbage collected until the closure itself goes out of scope and is garbage collected.

Closures allow encoding some state into a function, which is especially useful for higher order functions that return specialized functions, for example. This is, for example, how decorators in Python work - a function is captured in a closure, which is then referenced and called by a new function wrapping the original function.

I don't have a problem with the intuition but I'm with you on programming style.

It's sort of a contrary opinion but I prefer to be explicit about state, and closures are sort of silently bundling up state for you behind the scenes. It's not very apparent from the syntax.

I think languages should have a lighter-weight syntax for classes and that would subsume many use cases for closures.

-----

While I've never programmed in PHP, it appears PHP actually has a pretty nice solution for this problem! You declare the variables captured using 'use':

https://stackoverflow.com/questions/1065188/in-php-what-is-a...

    $callback =
        function ($quantity, $product) use ($tax, &$total)

In other languages the capture of 'tax' and 'total' are implicit.

In the article they discuss that Rust closures are just sugar for structs that work exactly as you describe.

As with any other syntactic sugar, it's easy to reason about the behavior once you know what the syntax desugars into.

One classic example I see with beginning programmers all the time is the order of the parts of a for loop.

e.g.

    for(initialization statement; bounds check expression; increment statement) {
      ...
    }

is just syntactic sugar for:

    {
      initialization statement;
      while (bounds check expression) {
        ...
        increment statement;
      }
    }

But once you understand what a for-loop desugars into, it's trivial to use and reason about.

Closures aren't much different. In Rust, if anything, they're less magic, because you have to be explicit about the lifetime of the closure relative to the references it's borrowing.

In many ways, closures can be more difficult when there is not garbage collection because we precisely do not want to refer to values that have been freed or collected. For example, in C++, we can write code like:

  #include <functional>
  #include <iostream>
  auto foo(double a) -> std::function <double(double)> {
      return [&a](auto x) {
          return x + a;
      };
  }
  int main() {
      auto f = foo(2.0);
      std::cout << f(1.0) << std::endl;
      std::cout << f(1.0) << std::endl;
  }

This code is bad because the function returned by `foo` depends on a reference to `a`, which is freed at the end of `foo`. Running this gives:

  3
  1

which is bad. Rust has the advantage of finding such errors. For example, the following Rust code:

  fn foo(a: f64) -> impl Fn(f64) -> f64 {
      |x: f64| -> f64 { x + a }     
  }          
  fn main() {                
      let f = foo(2.0);      
      println!("{}", f(1.0));                                                     
      println!("{}", f(1.0));                                                     
  } 

fails to compile with the error:

  error[E0597]: `a` does not live long enough
   --> test01.rs:2:27
    |
  1 | fn foo(a: f64) -> impl Fn(f64) -> f64 {
    |                   ------------------- opaque type requires that `a` is borrowed for `'static`
  2 |     |x: f64| -> f64 { x + a }
    |     ---------------       ^ borrowed value does not live long enough
    |     |
    |     value captured here
  3 | }
    |  - `a` dropped here while still borrowed

That forces us to correct the code by adding the move annotation to the the closure, which moves ownership of a to the closure itself:

  fn foo(a: f64) -> impl Fn(f64) -> f64 {
      move |x: f64| -> f64 { x + a }
  }          
  fn main() {                
      let f = foo(2.0);      
      println!("{}", f(1.0));                                                     
      println!("{}", f(1.0));                                                     
  }

This gives the correct result:

  3
  3

Of course, all of this nuance can be avoided in a garbage collected language. For example, in OCaml, we can write:

  let foo a =
      fun x -> x +. a
  in
  let f = foo 2.0 in
  Printf.printf "%f\n" (f 1.0);
  Printf.printf "%f\n" (f 1.0);

which also returns the correct result:

  3.000000
  3.000000

More generally, closures are nice because it's a useful feature to return a function from a function. For example, we may want call a function parameterized on some parameters repeatedly and we don't want to have to pass all of the parameters each time we invoke the function. Functional programming and closures allow for a variety of other programming techniques as well.

Beyond that, my point is that a garbage collector can track the use of variables and memory extremely well, which makes the process of constructing closures moderately easy. We can have closures without garbage collection, but then we must be careful about who and what owns this memory. C++ does not enforce this, but Rust does, which is one of its benefits.

In JavaScript and similar languages, “yes.” It’s really a lot simpler than you’re making it. Closures are great in these languages and incredibly useful for rapid prototyping. Especially in Node.js and in browsers.

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function?

I believe that would be 'lambda-lifting', i.e. taking a lambda which has free variables, adding those free variables to its input variables, moving the lambda out into the top- level, and modifying the call site to pass in the no-longer-free-variables. That should give you completely reasonable lifetimes and only require the stack.

It gets trickier when you want functions to be able to return functions. For instance:

  add(a, b) {
    a + b
  }

  myFun() {
    var add5 = x -> add(5, x);
    var add6 = x -> add(6, x);
    add5(1) + add6(2)
  }

In this case I've made two variants of add. I can't just run add5 on the stack, get rid of it, and then run add6 afterwards, because they both need to be alive at the same time.

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function?

Pretty much. When implementing closures, the variables that are going to be used by the nested functions are allocated on the heap instead of on the stack. The nested function is then set up to receive a pointer to the outer variables as an additional parameter.

If you want to read more about it, the word to search for would be "closure conversion"

http://matt.might.net/articles/closure-conversion/

It's unfortunate the the Wikipedia article for closure conversion now redirects to the article about Lambda Lifting. They are related but aren't quite the same thing.

> The idea that I can touch variables that have gone out of scope (and that have ostensibly been GC'd)

I think this might be due to your mental model of GC not quite matching the way it's usually implemented (at least, in traditional GC, as opposed to reference counting); most GC gives no guarantees that memory will immediately be cleaned up once the last reference goes out of scope, just that it will happen some time in the future (usually the next time it checks). Languages like Rust (and C++, if you use destructors) will inline the memory cleanup at the end of the scope, but this is pretty explicitly not what will happen in a garbage-collected language.

Note also that this isn't really any different than the fact that a variable can go out of scope and not be cleaned up if another reference to it exists. Consider the following Java-esque code:

    void setNewChild(Parent parent) {
        var child = new Child();
        parent.child = child;
    } 

The `Child` created in the method body will not be garbage collected even though the variable goes out of scope because it's still referenced somewhere else (i.e. in the Parent object). The key insight to understanding closures is that using the local variable in another function makes another reference to it, so naturally it would be kept alive until that function is gone.

I've got just the paper for you! The Implementation of Closures in Lua:

https://pdfs.semanticscholar.org/73a2/e3c03f799956aa5a3188e4...

This is just one way to do it, but it's an efficient way, and easy to understand.

I know you've already got a storm of replies, but I'd like to illustrate part of what I love about closure (especially when combined with anonymous objects like in JavaScript): you can replace classes, private vs public, "this", and other features with a simpler, smaller number of features.

    const Counter = () => {
      let counts = new Map();

      return {
        reset: () => {
          counts = new Map();
        },

        count: (key) => {
          const count = counts.get(key) + 1;
          counts.set(key, count);
          return count;
        }
      }
    }

    const items = ['apple', 'apple', 'banana', 'canteloupe'];
    const counter = Counter()

    items.forEach(item => {
      console.log(item + ": count is", counter.count(item));
    })

    counter.reset();
    // do some more counting now

IMO, this is much simpler [and prettier ;)] than the alternative using classes: you only need to know closures and objects, and the rules apply the same as they do in all other contexts. Classes in most languages usually come with their own twists and surprises.

> Is it literally just the language invisibly adding the parent-scope variables and their values to the top of my function?

Yup. A "first class" closure is just reifying some arguments of a future function call and bundling them up nicely in a struct/record. This is especially clear in languages that support currying, since that means you'll literally be building that record step-by-step - only after a full set of arguments is provided does a true function call really occur.

Because Rust doesn't have higher-kinded types. Well it does but it doesn't allow you to define or implement traits for them.

Also, what happens when you compose one function with FnMut and another with just Fn? The answer is clearly FnMut. These three traits are related by a subtyping relationship. I don't know enough Rust to answer the question of whether such a hypothetical compose function can return the right one from this hierarchy.