Understanding a Python Closure Oddity | Hacker News Reader

Understanding a Python Closure Oddity

utcc.utoronto.ca

67 pointsingve45 comments

Typical workaround:

  def loop():
     for number in range(10):
        def closure(number=number):
            return number
        yield closure

Many languages have surprising behaviour when closing over iteration variables.

That's why Javascript introduced the "let" keyword years ago

This logs from 0 to 9:

for(let i = 0; i < 10; ++i;) setTimeout(() => console.log(i));

This logs from 10, 10 times: for(var i = 0; i < 10; ++i;) setTimeout(() => console.log(i));

Yep, with exactly the same semantics as perl5's 25+ years old 'my' -

  foreach my $x (@input) {
    push @callbacks, sub { $x };
  }

works how you'd expect.

I was spoiled by this and driven nuts by function level scoping like python's and javascript's 'var', and the adoption of 'let' made me very happy.

My current annoyance with JS is its lack of -

  my $foo = do {
    ...;
    ...;
    <expression>
  };

but there's a proposal for that, and the 'match' proposal (which would be glorious) depends on it on the basis it's a shoo-in, so I am very hopeful.

Note that I know:

  let foo; {
    ...;
    ...;
    foo = <expression>
  }

already works but it's Not The Same (although significantly less upsetting than (() => { ... })() is ;).

Do you think you could elaborate on why the results differ? I'm somewhat unfamiliar with how JavaScript works.

'var' is JavaScript's older variable declaration construct. Variables created this way are live from the beginning of the function that contains them (or globally if there isn't one). So a block with braces (such as you'd use for a for or while loop body) doesn't actually restrict the scope of var `v` below:

    console.log(v); // <-- v is a variable here, we can access its value even though it is only declared below
    // prints 'undefined'
    {
        var v = 1;
        console.log(v); // prints 1
    }
    console.log(v); // prints 1

You used to (and might still) see a workaround to recover more restrictive scoping, known as the "IIFE" (Immediately Evaluated Function Expression): (function () { var v; ... code using v here ... })() creates a function (and thus a more restrictive scope for v to live in) and evaluates it once; this is a sort of poor man's block scoping.

`let` and `const` were created to fill this gap. They have block scope and are special-cased in for loops (well, `let` is; you can't reassign to a `const` variable so nontrivial for loops won't work):

    console.log(l); // <-- throws ReferenceError: l is not defined
    {
        // pretend the console.log line above is commented out, so we can reach this line
        let l = 1;
        console.log(l); // prints 1, as expected
    }
    console.log(l); // throws ReferenceError: l is not defined
    // ^^ l was only in scope for the block above

The interaction with `for` is explained well on MDN, including the special casing: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

"More precisely, let declarations are special-cased by for loops..." (followed by a more detailed explanation of why)

Fun(?) fact: it's not technically true that const can't be used in for loops:

    for (const i = 0; i > 0; ) {
        console.log('this is stupid');
    }

    let ran = false;
    for (const i = 0; !ran; ran = true) {
        console.log('this is also stupid');
    }

Further proof that Python is not fit for functional programming and that Python's namespaces and default lookup by reference are confusing.

In safe languages like ML you would have to create an explicit reference to get that effect, and then it is obvious to the reader.

Python is not built for complex abstractions, but unfortunately it is (ab)used for that, particularly in machine learning.

No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created. And it gets called differently when a list comprehension calls it each step of the loop or when the list function loops completely through and then the list comprehension calls the instances. Nothing to see here.

> No. The code is completely consistent: the closures are different instances of a function that returns the value of 'number' at the moment it gets called. Not the value of 'number' when it gets created.

Isn't that exactly what GP said? And gave an example how to create that effect explicitly in functional languages, using refs?

Yes. But I found the explanations of the GP a little misleading, especially the eager/lazy part.

Naively I would expect that each iteration of a for loop creates a new loop variable (all with the same name, but effectively each in their own scope) and so each closure holds a reference to a variable named number whose value never changes, and there is a distinct one of these number variables per closure .

That is not how variables in Python work. Variable assignment is just name binding. This is clearly mentioned in every Python beginner tutorial I know of. Python code is a script. Follow the script and you know what Python does (with only a few exceptions, see e.g. below).

About the scope: Python knows in principal only 2 scopes: local and global. If you define a function (or a class) Python creates a new emtpy/filled with the function arguments local scope (system table). That gets filled successively by local variable assignments etc. There happens a little bit of magic: if you use a variable of the next higher scope, it gets copied (!) to the local scope. You can follow that easily by print(locals()). (You can also use the global scope, but that is another topic).

Edit: had to clarify some points

It's consistent, but unexpected. Especially if you don't know python well enough. And it leads to ugly effect sometimes, even for experienced pythonistas.

Yes, this bit me last week. I stumble over it about once a year. Recently encountered it in an open source project and filed an issue. I don't know where I fall on the "experienced" spectrum, but I've written a lot of python in the last 10 years.

No.

If you want to "close over" the value in the closure then there's a simple trick you can use:

Pass a keyword argument with the default value equal to the value you want to close over.

Voila!

Functions.partial can also supply captures in that way, and is slightly more obvious (and less likely to generate linter complaints if the kwarg is mutable).

er, functools.partial, too late to edit.

all one needs to understand is that there are names that refer to objects in Python. Nothing is passed by value.

The closure returns whatever object the name refers at the moment. That is all.

    def loop():
        for i in range(10):
             yield lambda j=i:  j

`j` refers to the same object (int) `i` refers to at the time of the lambda creation (the desired result). `j` is local to the lambda (each lambda has its own local namespace).

Just `lambda: i` code would result in an object that `i` refers to at the time of calling the lambda (`i` name is not local to the lambda).

https://docs.python.org/3/faq/programming.html#why-do-lambda...

This is less wat. when you write it like

   eager:
   for item in loop():
      item()

   lazy:
   items = []
   for item in loop():
      items.append(item)

   for item in items:
       item()

Each next() into loop increments a shared counter so when you print it out as you go it 1, 2, 3 but when you print it out once you've called it a bunch is 9, 9, 9.

You can put an extra 'level' in, to make both options return the same 0,1...9

    def loop():
        for number in range(10):

            def outer(n):
                def inner():
                    return n

                return inner

            yield outer(number)

Is there a neater way?

Yes. The usual way to do this is to bind them as defaults to an argument, for example:

    def loop():
        for number in range(10):

            def func_w_closure(_num=number):
                return _num

            yield func_w_closure

This works because default arguments in python are evaluated exactly once, at function definition time. So it's a way of effectively copying the ``number`` out of the closure[1] and into the function definition.

[1] side note, closures in python are always late-binding, which is what causes the behavior in OP

I’ve never thought that leaking this type of implementation detail into the return value (and return type!) was a nice solution. I like the double closure better, and one can shorten it a bit with a lambda.

For those who prefer a functional style, functools.partial can also solve this problem.

(I use Python, and I like a lot of things about Python, but I don’t like its scoping rules at all, nor do I like the way that Python’s closures work. I would use a double lambda.)

Does this work for user defined classes (objects) as well?

Yes, and by the same mechanism and for the same reason.

I would have just done something like

``` def loop(): for number in range(10): fixed_number = number def inner(): return fixed_number yield inner ```

Correctly formatted (two spaces preceding each line, one blank line before the first code line, no extra lines needed between code lines):

  def loop():
    for number in range(10):
      fixed_number = number
      def inner():
        return fixed_number
      yield inner

The output:

  eagerly = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  lazily  = [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

It doesn't work because of the way Python's variables are scoped. Your fixed_number variable is still shared across all instances of inner. Python doesn't have any sort of block scoping like you seem to think it has.

I'll give you one better:

    >>> strings = tuple((lambda: f'{i}') for i in range(10))
    >>> strings[1]()
    '9'

The f-strings are supposed to be evaluated at... what time exactly?

And some more stuff:

    >>> class Elephant:
    ...   def __init__(self, i):
    ...     self.i = i
    ...   def __call__(self):
    ...     return self.i
    ... 
    >>> elephants = tuple(Elephant(i) for i in range(10))
    >>> elephants[0]()
    0
    >>> elephants[5]()
    5

The answer is simple. f strings are evaluated at run time. The behaviour is correct. Up until you call the lambda, they are just a function definition.

What's so interesting about the Elephant case? You store the value of i at run time, so it can be accessed later. lambda doesn't have that capability, as it's inheriting i from the block.

> The answer is simple. f strings are evaluated at run time.

You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching.

> The behaviour is correct.

Python doesn't have a standard. Nobody knows what behavior is correct. That's the only behavior Python has, so, again, calling it correct is just not saying anything.

> What's so interesting about the Elephant case?

This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.

You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.

> Everything in Python is evaluated at runtime

That is not correct. Python generally has two "runtimes": one is on import, when all of the module level code is evaluated; and the runtime, when the bodies of various functions and methods are evaluated. The two can be combined, e.g. by importing a module from within a function, but generally they are separate.

> Bytecode is just a form of caching

Not quite. It is perfectly possible to deploy just the bytecode and disable recompiling on import.

> Python doesn't have a standard

Of course it does -- CPython is the reference implementation, so its behaviour is "correct".

> You didn't understand the question. Everything in Python is evaluated at runtime. Bytecode is just a form of caching.

You don't understand what runtime in python is. Compilation into bytecode is an implementation detail and not part of runtime. Even when you start a python scripts from a bytecode state, there is a runtime. f-strings are evaluated at runtime. When you declare them within a function, yes they will be parsed, but they will not be run.

> This class, in principle, shouldn't be different from lambda from the previous example, yet it acts differently.

In your case, you put them in lambdas, which is equivalent to putting them into a function context. The f-string will only be evaluated when lambda/function is called. In your Elephant case the runtime happens when i=n, whereas in lambda case, the runtime is when i=9 (because there is no local copy of the variable/context). Plain and simple. If you don't understand such a simple distinction, I recommend you to start reading the bytecode as it's much clearer at that level.

> You don't need to lecture me on why things behave the way they are. Since I found these examples, you can trust me, I know why they behave like this. The idea is to drive your attention to the inconsistencies that a "naive" reader would discover.

It's not an inconsistency, if you understand the principle.

but is it a closure oddity? Looks to me like it's a generator/comprehension oddity :)

It’s a closure oddity. The closure captures a reference to the variable, not the value of the variable at the time of its creation. Given that, the results are entirely predictable.

If we're really going to argue, I'd argue for a 'for' oddity. The for statement could've been defined with a fresh binding per iteration instead of a mutation.

How does that work in practice? Python stores local variables in the stack frame with a compiler-calculated index. Do you have a growable stack frame? Add stack frames for every loop iteration?

I don't think the Python compiler is to know if a loop variable is closed over and then adjust the code generation. So this change would likely introduce a lot of complexity, or be a pessimisation in the normal case.

I think the simple, although less helpful, semantics are clearly better here. FWIW I don't think this comes up in practice very much. Maybe because callbacks in Python APIs aren't common (and its not due to this).

See my other reply about how [x for x in y] now works. It'd be a variation on that.

I agree the current semantics are better by the measure of performance of the simplest possible implementation. Though that's not Python's main design goal.

it comes up often enough for me to be annoying. The defaulted argument works, but it is an ugly hack. It is not obvious to me that the surprising semantics are clearly better, but python has a long history to preferring implementation simplicity to user ergonomics.

It's unrelated to the for loop, the for only changes the variable, there are other ways you can do that and you'll run into the same "oddity"

    def test():
        n = 0
        def a():
            return n
        n = 1
        def b():
            return n
        return a, b

    a, b = test()
    a() == 1
    b() == 1

> for changes the variable

That's a design decision, not a law of nature. JS 'for let' lacks this problem; probably the committee would have named it just 'for' if not for legacy.

I get why it behaves like this, but I find it odd that Python claims to have "lexical scoping". This is not lexical (as you read it). This is based on somewhat hidden reference lookups.

That would make it work differently than the entire rest of the language. Python is function-scoped, not block-scoped.

That ship already sailed when Python 3 made comprehension expressions (like list comprehensions) introduce a nested scope. They're even a kind of 'for' too. (Previously, [x for x in y] would clobber any x in the same function. Fixing this meant implicit nested scopes.)

It's true that this would've been a further change (every loop entering the implicit nest, rather than entering one nest which then mutates per loop). But this design was a decision, not a necessity. JS's "for let" shows this by example.

Not really -- in practice that would imply that ``for`` needs to freeze the value of ``locals`` during every iteration, which doesn't make sense to me.

This is definitely a python gotcha. In fact, it's literally one of the canonical ones [1]. But it's purely the interaction between the way that names/variables work in python, and late-binding closures.

More explicitly: it's best to think of "variables" in python as "names", because in code, you refer to names, but the underlying variable might change. This is also why python is best described neither as call-by-reference nor call-by-value, but rather "call-by-assignment" or "call-by-object": you're passing a reference to a variable bound to a particular name. Unless you're writing C extensions, you never actually have a reference to the variable itself -- instead, to the name.

Meanwhile, closures in python are always late-binding: they are themselves "live", mutable objects, in the sense that they're just a reference to the namespace they're closing, and *not* a copy. So if you mutate that namespace between establishing the closure and making use of it, you'll see the mutated version.

[1] https://docs.python-guide.org/writing/gotchas/#late-binding-...

Yes, the best way to understand python closures is that scopes are first class objects in python (accessible via locals()) and closures close over the containing scope itself, not the single variables.

Also worth keeping in mind that, AFAIK, there is one scope per function, which differs from other languages that can have nested scopes inside a function. I think the list comprehension is the only exception in python3, which was changed so that names no longer leak out of it, which was frankly insane in py2.

Reply on news.ycombinator.com