Hacker News Reader

Comment by roenxi | Hacker News Reader

Python's lambda also has an interesting design flaw that anyone using it will come up against sooner or later [1]. The way it interacts with Python's scoping rules is quite broken. This hinders using it as freely as is possible in, eg, Clojure.

  >>> fs = [(lambda n: i + n) for i in range(10)]
  >>> [f(4) for f in fs]
  [13, 13, 13, 13, 13, 13, 13, 13, 13, 13]

[1] http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/

View on HN

A lambda is no different in scoping rules than a regular function def. If you put a def in a loop, you'll find the same behavior.

Closures are only created for functions defined inside other functions. Thus the term non-local. They're variables that are neither local nor global.

Don't call it broken when it works as intended.

“Works as intended” but also “unintuitive to most users” is still broken. Disclaimer: I have hated python ever since falling out of love with it 8 years ago for reasons just like this one so I’m definitely biased.

Since programming is a rapidly growing field and Python is the first language for many, I suspect that Python's scoping rules are in fact intuitive to most of its users. The ones who stumble are those with intuition built by some other language.

I'd argue that's the opposite. Python was the first dynamic language I learned. The scoping rules in Python didn't bother me because I didn't know any better, not because they're intuitive or well designed.

Fair anecdote. I'll offer that in my experience teaching hundreds of students, most don't have trouble with the cascade from locals to globals to builtins, and quickly grasp that globals are looked up for each call. That closures work at all confuses the ones who are thinking carefully about local scope. I may have a biased sample of students, but I think it's decent evidence.

As another comment said, this works the same way in def and lambda. This same thing happens in Javascript:

http://markheath.net/post/javascript-closure-loop-gotcha

Python stores the captured variables in __globals__ which is shared with all the different lambdas for some odd reason:

>>> fs[0] == fs[1]

False

>>> fs[0].__globals__ == fs[1].__globals__

True

>>> fs[0].__globals__['i'], fs[1].__globals__['i']

(9, 9)

Kind of weird?

--edit--

  >>> def f(i):
  ...     return lambda n: i + n

  >>> fs = [f(i) for i in range(10)]

  >>> [f(4) for f in fs]
  [4, 5, 6, 7, 8, 9, 10, 11, 12, 13]

It works if it has somewhere to store the captured variables so just don't pollute the global environment and all is fine...

I think you're looking at Python 2, because in py3 the list comprehensions create a nested scope for the loop variable to be local in. -- i.e. 'i' would be in a local environment, not in globals. This doesn't change the behavior of these particular closures, since i is still mutated in the loop.

Python 3.6.5 (default, Apr 4 2018, 15:01:18)

...even weirder:

>>> fs[0].__closure__ == fs[1].__closure__

True

Since I don't have py3 installed at the moment, I can't check anything, but I'd guess this is two closures with distinct ids but the same environments, and == is doing a structural comparison rather than an identity comparison. Wish I could look into it right now.

  >>> fs[0].__closure__, fs[1].__closure__
  ((<cell at 0x7f4e9111b048: int object at 0x7f4e98e99500>,), 
   (<cell at 0x7f4e9111b048: int object at 0x7f4e98e99500>,))

So lambda isn't broken it's the list comprehension going all wonky with its local variables methinks.

All right, I looked into this using repl.it. The __closure__ field is not actually a closure in the usual sense of the word -- it's the environment of the closure, and fs[0] and fs[1] are expected to share the same environment. If you look at the identities of fs[0] and fs[1] themselves, they are different. It's as I said originally, just with a confusing name for the environment field.

Reply on news.ycombinator.com