Hacker News Reader

Comment by ot | Hacker News Reader

Very interesting, I've been bitten by the same problem with Python once (capturing the iteration variable in a closure). To make things even worse, in Python the behaviour is inconsistent between list comprehensions and generator comprehensions:

  In [10]: lambdas = [(lambda: i) for i in xrange(10)]
  In [11]: [f() for f in lambdas]
  Out[11]: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
  
  In [12]: lambdas_gen = ((lambda: i) for i in xrange(10))
  In [13]: [f() for f in lambdas_gen]
  Out[13]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The problem was not fixed even in Python 3. I don't know if there has been any discussion about it.

View on HN

This is because the following is simply not a closure in Python:

    lambdas = [(lambda: i) for i in xrange(10)]

This works, because it is a proper closure:

    def build_closure(i):
        return (lambda: i)

    lambdas = [build_closure(i) for i in range(10)]
    print("%s" % [f() for f in lambdas])

See http://stackoverflow.com/questions/233673/lexical-closures-i...

You can write something close only with lambda this way:

    lambdas = [(lambda j=i: j) for i in range(10)]

The behavior is actually consistent between generators and list comprehensions, but it's giving the same result as a closure because generators are lazily evaluated (so f() is evaluated right on time, but i is still not enclosed), and work only once: running it twice will have the generator exhausted:

  In [12]: lambdas_gen = ((lambda: i) for i in xrange(10))
  In [13]: [f() for f in lambdas_gen]
  Out[13]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
  In [14]: [f() for f in lambdas_gen]
  Out[14]: []

That's why forcing the generation with list() causes evaluation of the generator, and only then do you evaluate the f()s, hence the same result as the list comprehension case.

Again, changing it to the following properly encloses i:

    lambdas_genlist = list((lambda j=i: j) for i in range(10))

> This is because the following is simply not a closure in Python

Well, they should be closures, or they shouldn't be there.

This isn't a question of not understanding Python's semantic rules, it's a question of those rules being screwed. I understand why it's not consistent with generators (as you say - i isn't generated yet). I don't understand why it's not consistent with what you'd expect, namely lambdas not being closures.

It's an even weirder gotcha than:

    def f(x = []):
        x.append(1)
        return x

and we know how many people get hit with that one ;)

Actually, you know, I lied (for the sake of simplicity). They are closures, else how would the lambda evaluate 'i'? The difference is in the binding.

How closures work depend wildly on the language. With lexical closures it all comes down to how scopes are handled [0] and how and when variable binding is done [1] (notably §8). The fact that 'i' can be either bound late (giving the 'outer scope' effect) or bound early (giving the 'inner scope closure' you expect) is actually a quite useful feature (and I assure you both cases are equally useful), although admittedly a bit surprising when coming from other languages.

Default argument value evaluation is a nice gotcha, but it's a trade-off I'm more than willing to accept [2].

Anyway I would definitely not qualify this as 'screwed'.

[0] http://stackoverflow.com/a/292502/368409

[1] http://docs.python.org/reference/executionmodel.html

[2] http://stackoverflow.com/a/1651284/368409

The inconsistency is because, in the generator-expression case, the calls to f() are being interleaved with the iterations of the generator (so the closed-over variable has the 'correct' value when f() is called). If you change this by running the generator to completion first, the behavior is the same as the list case:

  In [1]: lambdas_listgen = list(((lambda: i) for i in range(10)))
  In [2]: [f() for f in lambdas_listgen]
  Out[2]: [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

It's still really complicated semantics.

it's only complicated if you expect something which isn't there. it's a gotchya, true.

It's simple semantics with confusing behavior.

It's important to note that scoping on Python for loop variables does not behave the same way it does for a C# foreach.

While it's possible to write this in Python:

    for i in [1, 2, 3, 4]:
        pass
    final_value = i

The same in C# is not possible, i.e.:

    int[] values = {1, 2, 3, 4};
    foreach(int i in values) {
    }
    var final_value = i;       // there is no i here!

It makes no sense to "fix" this in Python because the loop variable is created in the scope outside of the for loop. It seems to make sense in the case of the C# foreach (but not the C# for!) because that variable is inaccessible outside of the foreach loop scope anyway. I would still argue introducing inconsistent behaviour between for and foreach as they are doing in C# 5 is just going to further obscure this problem and not really eliminate it.

Anyway, as far as Python is concerned, closures close over variables not values. Creating special cases where this is not the case is bound to generate even greater confusion.

IMO, the problem here is not the closure but how i is updated instead of being a new variable at each iteration. I guess it's like that for the sake of performance, but if i was immutable, that wouldn't happen. I.e. i would be a whole new variable at each iteration.

Here's another example:

  in: lambdas = [(lambda i: lambda: i)(i) for i in xrange(10)]
  in: [f() for f in lambdas]
  out: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Why is it a different i? It's misleading because sometime we have a reference and sometime we've got a whole new variable.

Thanks for pointing that out. I would have guessed that the original form would desugar to

  map(lambda i: (lambda: i), xrange(10))

but apparently it doesn't.

Exactly, it desugars to something like a for loop, in fact the iteration variable remains visible in the outer scope (other possible source of confusion).

I believe it doesn't desugar to a map+lambda for performance reasons.

if it desugared to map, you could redefine map() and unleash all kinds of hell on yourself inadvertently.

Reply on news.ycombinator.com