Hacker News Reader

Comment by acqq | Hacker News Reader

So can somebody explain to me, step by step with the "elements behind the curtain", what is actually going on in these two cases:

   >>> [ x() for x in [lambda: i for i in range(5)] ]
   [4, 4, 4, 4, 4]

Why?

   >>> [ x() for x in [lambda i=i: i for i in range(5)] ]
   [0, 1, 2, 3, 4]

And why?

Starting from the fact that the following give:

    >>> [lambda: i for i in range(5)]
    [<function <listcomp>.<lambda> at 0xC22>, 
     <function <listcomp>.<lambda> at 0xCAE>, 
     <function <listcomp>.<lambda> at 0xC9A>, 
     <function <listcomp>.<lambda> at 0xCD6>, 
     <function <listcomp>.<lambda> at 0xCC2>]

    >>> [lambda i=i: i for i in range(5)]
    [<function <listcomp>.<lambda> at 0xCF4>, 
     <function <listcomp>.<lambda> at 0xCFE>, 
     <function <listcomp>.<lambda> at 0xD08>, 
     <function <listcomp>.<lambda> at 0xD12>, 
     <function <listcomp>.<lambda> at 0xCCC>]

View on HN

This is your first snippet (expanded to two lines and max index reduced from 5 to 3):

   lambda_list = [lambda: i for i in range(3)]
   results = [x() for x in lambda_list]

It expands to this:

   i = 0
   f0 = lambda: i
   i = 1
   f1 = lambda: i
   i = 2
   f2 = lambda: i
   results = [f0(), f1(), f2()]

The body of those functions f0, f1, f2, each looks in the function's own dictionary of local variables for "i". But it doesn't exist there, so it looks in the enclosing scope (and keeps doing this recursively up to global scope if necessary). At the next scope it does find a variable "i". This happens at the point you call the function (the same as anything else inside the definition of a function), in this case on that last line when you assign to results. By that point, the "i" variable in the enclosing scope has value 2. So results = [2, 2, 2].

[edit: this feature of functions in Python (variables are looked up by name when the function is run) is what lets you define functions foo() and bar() in a module / script that call each other. When the first one is defined, the other one doesn't exist yet, but you're not actually running the code inside the function so it doesn't matter. By the time you come to call it later, the other function does exist in the enclosing (module level) scope so the lookup succeeds.]

This is your second snippet (I changed one of the variable names from i to j for clarity, but you can rename it back and it would work the same way):

   lambda_list = [lambda j=i: j for i in range(3)]
   results = [x() for x in lambda_list]

it expands like this:

   i = 0
   f0 = lambda j=i: j
   i = 1
   f1 = lambda j=i: j
   i = 2
   f2 = lambda j=i: j
   results = [f0(), f1(), f2()]

But defaults of function arguments are evaluated at the point the function is declared, not when it's called (as noted elsewhere in this thread - hence the def foo(x=[]) gotcha!). You can imagine the second line of code above is like this, just to really drive the point home [edit: I originally used a made up syntax, but I updated to code that actually works]:

   f0 = lambda j: j
   f0.__defaults__ = i,
   f0.__kwdefaults__ = {"j": i}

So, f0 has the specific value 0 baked into it, and similarly f1 and f2 have 1 and 2. Therefore results = [0, 1, 2].

I've tried to see __defaults__ and __kwdefaults__ here:

    i = 7

    lambda_list = [lambda: i for i in range(3)]
    results = [x() for x in lambda_list]

    print( i )
    for k in range( 3 ):
        print( lambda_list[k] )
        print( lambda_list[k].__defaults__ )
        print( lambda_list[k].__kwdefaults__ )

    lambda_list = [lambda j=i: j for i in range(3)]
    results = [x() for x in lambda_list]

    print( i )
    for k in range( 3 ):
        print( lambda_list[k] )
        print( lambda_list[k].__defaults__ )
        print( lambda_list[k].__kwdefaults__ )
    print( i )

The output is:

    7
    <function <listcomp>.<lambda> at 0xB2>
    None
    None
    <function <listcomp>.<lambda> at 0xBC>
    None
    None
    <function <listcomp>.<lambda> at 0xC6>
    None
    None
    7
    <function <listcomp>.<lambda> at 0xD0>
    (0,)
    None
    <function <listcomp>.<lambda> at 0xDA>
    (1,)
    None
    <function <listcomp>.<lambda> at 0xE4>
    (2,)
    None
    7

If I understand it, __kwdefaults__ are always None and the __defaults__ used only in the second case. I've also tried to see the "i" as not being inside of the closure, but it remains 7, not like (as noted by 'remram here):

    i = 9
    y = 1
    for y in range( 1,-1,-1 ):
        try:
            x = 4 / y
        except Exception as i:
            pass
        print( i )

Which gives, for me, fascinating:

    9
    Traceback (most recent call last):
      File "p.py", line 10, in <module>
        print( i )
               ^
    NameError: name 'i' is not defined. Did you mean: 'id'?

I'm learning a lot here.

Oops, `__kwdefaults__` is only for keyword-only arguments

   fn = lambda x, *args, y=3: 0
   print(fn.__defaults__, fn.__kwdefaults__)
   # prints None {'y': 3}

Source: https://stackoverflow.com/questions/17533929/what-is-the-use...

[Edit: quietbritishjim was faster than me, and makes more or less the same points. You can safely ignore this post.]

First let's have a look at this:

   >>> [ x() for x in [lambda: i for i in range(5)] ]
   [4, 4, 4, 4, 4]

Variable i in the function body is a nonlocal or global, taken from the nearest enclosing scope where i is bound. When a function (lambdas are functions just like any other) is executed, a nonlocal or global variable gets the value it has in its scope at the time of the function call. Step by step:

    >>> import inspect

    >>> i = 0
    >>> a = lambda: i
    >>> inspect.getclosurevars(a)
    ClosureVars(nonlocals={}, globals={'i': 0}, builtins={}, unbound=set())

We see that variable i in a refers to the global i, which currently has value 0. Obviously a() returns 0:

    >>> a()
    0

Next we create a new lambda, with the same definition as before, but we first increment i.

    >>> i = 1
    >>> b = lambda: i
    >>> inspect.getclosurevars(b)
    ClosureVars(nonlocals={}, globals={'i': 1}, builtins={}, unbound=set())

So i in b refers to the global i (as before), which now has value 1. Unsurprisingly b() returns 1:

    >>> b()
    1

But variable i in a also refers to the global i which is now equal to 1:

    >>> inspect.getclosurevars(a)
    ClosureVars(nonlocals={}, globals={'i': 1}, builtins={}, unbound=set())

And sure enough, a() now returns 1:

   >>> a()
   1

We can set i to any value, and both a() and b() will then return that value.

    >>> i = 'gotcha'
    >>> a()
    'gotcha'
    >>> b()
    'gotcha'

Going back to your example, the i used in the lambdas refers to a nonlocal instead of a global (the i used in the list comprehension), but that doesn't make a difference for the topic at hand. What does matter is that the current value of i is used, which is 4 after the for loop in the list comprehension.

Now to the second example.

    >>> i = 0
    >>> a = lambda i=i: i

i is now a local, since it's a parameter to the function. It's a bit unclear because i is used in two different meanings here; let's rename one of the uses to j. This is exactly equivalent:

   >>> a = lambda j=i: j
   >>> inspect.getclosurevars(a)
   ClosureVars(nonlocals={}, globals={}, builtins={}, unbound=set())

This time there are no closure variables. j in the function body does not anymore refer to anything outside of the function scope. Instead a is now a function with one parameter, which returns the value of that parameter:

    >>> a(10)
    10
    >>> a('hello')
    'hello'

What's of course more important here is that we have a default value for that parameter. Default parameter values in Python are evaluated when the function is defined, not when it is executed. i was 0 when we defined a, so a will always have 0 as default parameter. We can see that default value:

    >>> a.__defaults__
    (0,)

Add a second function:

    >>> i = 1
    >>> b = lambda i=i: i
    >>> b.__defaults__
    (1,)

Now b is also a function taking one parameter, but this time that parameter has default value 1. Meanwhile the default value of function a has not changed:

    >>> a.__defaults__
    (0,)

So now we can predict that a() will return its default parameter value (which is 0), and b() will return its default parameter (which is 1). Let's check.

    >>> a()
    0
    >>> b()
    1

So the big difference is that your first example uses nonlocal/global variables which are evaluated when the function is called, while your second example uses default parameter values which are evaluated when the function is defined.

Note: you can change default parameter values though, if you really really want:

    >>> a = lambda i=5: i
    >>> a()
    5
    >>> a.__defaults__ = ("don't actually do this",)
    >>> a()
    "don't actually do this"

I like your explanation, it has more missing information and also demonstrates the use of "import inspect", it's definitely not to be ignored!

Did I understand correctly, the fact that the variable is not belonging to the closure is the non-intuitive part, not knowing "how it was decided to do in Python" I'd expect the variable to belong to the closure, but instead just the reference to the variable outside of the closure belongs to the closure? Which AFAIK doesn't save any memory (the reference still has to be there), it was simply a design decision?

So in Python the closure just doesn't "enclose" the variables at all? So any use of any variable in it can destroy something used "outside"? How do those using these construct cope with that in practice? Just ignore? Invent some naming schemes? Something else?

Reminds me on an old Fortran which allowed changing the global constants passed to the functions via an assignment to the function parameter inside of the function. :)

I guess it's a design decision, I guess it could have been designed differently, but I'm not really sure. Doing it differently could very well break other things that people do rely on.

Referring to the original variables instead of making copies does save memory though: in your example there were 5 functions that each would have their own copy, instead of only 1 instance shared between them.

Reply on news.ycombinator.com