Python has closures | Hacker News Reader

Python has closures

blade.nagaokaut.ac.jp

69 pointssamratjp66 comments

Unfortunately in the example there is not a single closure shown. A closure is a function and its bindings. But there are no function objects created.

What he shows is some attempt to describe some kind of lexical binding. But even that looks very suspicious.

    def foo():
         global y
         x = 0
         def bar():
             print x, y
         bar()
         x = y = 1
         bar()
         x = y = 2
         bar()
         x = y = 3
         bar()
         x = y = 4
         bar()
         x = y = 5

    foo()

    0 0
    1 1
    2 2
    3 3
    4 4

Above is really trivial. The problem is that the assignments change a variable and the function BAR sees the changes. So there is only one variable binding for X. In a variant where each assignments were really bindings, BAR would only see the binding in its scope and the number would not change.

Above would be in Lisp:

    (defvar *y* 0)

    (defun foo ()
      (let ((x 0))
        (flet ((bar ()
                 (print (list x *y*))))
          (bar)
          (setf *y* 1 x *y*)
          (bar)
          (setf *y* 2 x *y*)
          (bar)
          (setf *y* 3 x *y*)
          (bar)
          (setf *y* 4 x *y*)
          (bar)
          (setf *y* 5 x *y*))))

Again, no closure. Just a function with a local function that gets called several times.

He then describes that in a 'subfunction' a variable from an outside scope can't be set, but that is almost a side issue. It also may be that this is not often found in his code, but it is found in a lot of Lisp code that actually uses closures.

The most trivial example is a counter closure:

    (let ((fn (let ((x 0))
                (lambda () (setf x (+ x 1))))))
       (print (funcall fn))
       (print (funcall fn))
       (print (funcall fn)))

1 2 3

In above code FN gets bound to a closure. This closure is called three times. Each time it updates the variable X in its lexical scope.

No, they absolutely are full closures. He just didn't demonstrate that fact to your satisfaction. Here is a code example that may be more convincing to you.

  def make_counter(name):
      position = [0]
      def counter():
          position[0] += 1
          return (name, "incr", position[0])
      def show_counter():
          return (name, "show", position[0])
      return counter, show_counter

  foo, see_foo = make_counter("foo")
  bar, see_bar = make_counter("bar")

  for i in range(5):
      print foo()
      print see_foo()

  for i in range(5):
      print bar()
      print see_bar()

  for i in range(5):
      print foo()
      print foo()
      print see_foo()
      print bar()
      print see_bar()

If you run that you'll see that the returned closures retain lexical bindings to the scope they were defined in. Closures generated at different times do not interfere with each other. And closures generated in the same scope are pointing at the same variables.

What more do you want?

Nothing. I never said that Python doesn't provide closures. I just pointed out that Mr. Prescod did not give an example for those, even though he writes about closures. He really was talking about lexical scope. He did not demonstrate closures for my satisfaction, he didn't demonstrate them at all.

This makes me guess that Mr. Prescod never understood what a closure is, since he failed to give any example. That one can set closed over variables in Lisp is a useful feature. He demonstrates that Python 2 can't set a variable of an enclosing scope and claims that he does not need that. Maybe if he were really using closures, he would have found it useful.

But he did give an example. In his example, bar is a function object that is a closure. He just didn't make it do anything that would demonstrate it had to be a closure.

As for Pascal, you need to look again. From the beginning Pascal had the ability to pass functions and procedures as parameters to other functions and procedures. If those functions and procedures are defined in an inner scope, they will carry the original lexical scope, and are in fact full closures. OK, they are clunky to use, but they work. Skip to the end of the section starting at http://en.wikipedia.org/wiki/Comparison_of_Pascal_and_C#Func... to see the necessary syntax demonstrated.

Don't quote me on this, but I believe that Pascal was actually the very first language with full closures. All of the necessary pieces are in ALGOL-68, but I think that Pascal was implemented first and so wins.

Incidentally Sussman claims that he got the idea of lexical scope for Scheme from studying ALGOL. Of course Scheme realized the possibilities of the construct as no language before it had, and a lot of credit is due for that. But you shouldn't lightly dismiss the contributions to Lisp from languages out of the Lisp family.

No, bar is not a closure. It is basically just a local procedure.

The original Pascal did not have full closures. Functions did only exist as long as their surrounding function was called. They could not be returned from functions. Pascal lacked also the necessary Garbage Collection. Pascal had lexical scope, but not full closures.

All you could do was pass functions to other functions. But that's trivial, since each time one passed function X, it gets its active environment, which always is on the stack. There was no way to have closures, where functions have their own set of variable bindings, even when the surrounding function is no longer active. Being able to pass a function downwards is not sufficient to require full closures.

Basically the same applies for Algol. Functions can't be returned and are not first class objects.

What Algol contributed to the evolution of Lisp was lexical scope, but not closures. Also note that it was not Scheme that introduced closures in Lisp.

That's what I fear, many people think they know what a closure is, but in reality there is a lot of confusion.

> No, bar is not a closure. It is basically just a local procedure.

bar is a variable whose value is a closure. In python, defining a named function just creates a variable whose value is a closure.

Python the language or Python the implementation?

I do mean the main implementation, CPython. I guess an implementation is free to avoid unnecessary closures and it would be nice to know if any others do.

You are right in your description of the limitations of Pascal.

As for your claim that it was not Scheme that introduced closures to Lisp, I'd like to hear more about that. I've run across the claim that Scheme was from a number of sources, including http://en.wikipedia.org/wiki/Closure_%28computer_science%29, and I haven't heard of any other prior art.

It lists the paper of Joel Moses. Look for 'funarg problem'.

The original post is from 2003. Much has changed since then. For example, in Python 3.x it's possible to write the "counter" example like this:

  def make_counter():
      count = 0
      def f():
          nonlocal count
          count = count + 1
          return count
      return f

  c = make_counter()
  for i in range(3):
      print(c())

(In Python 2.x and earlier, you would probably create a Counter class to do the same thing. That approach was, and likely still is, considered more "Pythonic", by the way.)

I understand that. Still he said 'Python has closures' in the title and failed to give examples of closures.

No, he gave examples of things that really are closures, but failed to test them in ways that would have demonstrated that they in fact are closures.

No, closures are created when functions are passed or returned and there are free variables used in such a function.

He did neither pass a function nor return one.

The function itself is not a closure. The closure is an object that bundles a function and an actual set of bindings.

In some languages, when you neither pass nor return a function a closure is not created. You can regard this as an optimization; in Python the closure is always created as far as I understand.

That looks more like a choice of some implementation and hopefully not of the language. I would hope for 'Python' that a Python compiler doesn't do that. There is no need to create a closure, when it is never needed. Requiring that for each call of a function with some local functions, closures are created for those, has little to do with 'we have closures', but more with a silly implementation.

I don't know if any python implementations avoid creating unnecessary closures, but you are right that I meant a specific implementation: CPython.

I am not too familiar with python in the present and maybe the following has changed: there isn't really a definition of the language, but the documentation (which was written for the main implementation, but is the closest thing to a definition of the language) does mention that declaring a function creates a function object (closure) and assigns it to a local variable. Of course, an implementation should probably feel free to contradict the documentation when the difference is not observable.

Python 3 already has the nonlocal keyword which does exactly what it sounds like. It's explained in mind-numbing detail by PEP 3104 at http://www.python.org/dev/peps/pep-3104/ but it's really what you'd expect.

  def make_counter():
      n = 0
      def incr():
          nonlocal n
          temp, n = n, n+1
          return temp
      return incr

I think `nonlocal` is a prime example of a leaky abstraction. Can anyone explain the advantages of using references for normally primitive variables?

There are no primitive variables in python. Even ints, strings, floats, etc point to a reference of an object. All nonlocal does is point the variables reference to a different object.

As to the advantages...

Well for one you can use the variables in closures...

I'm not entirely sure what you mean by "using references for normally primitive variables." Nonlocal doesn't change whether variables are references or not, only the semantics of assignment. By default, all assignments take place in the local scope; if you've indicated that a variable is nonlocal, then instead of creating a new variable in that scope you assign instead to the variable of that name in the nearest enclosing scope. All variables in Python are references to objects, so there is no such thing as a "primitive" type, but that was true even before the nonlocal keyword was introduced, so it's not relevant here.

Lisp has had proper support for lexical scoping and closures for ages, and nobody has ever accused it of being a leaky abstraction there, either. The Scheme example of the same program would be

  (define (make-counter)
    (let ((n 0))
      (lambda ()
        (let ((temp n))
          (set! n (+ n 1))
          temp))))

All that's happening here is that let is creating a new variable, and set! is looking for the nearest enclosing scope--which turns out to be the let--and assigning to that one. (Okay, there are minor semantic differences, because let creates a new scope instead of creating a new variable in the local scope, but that's not really relevant to the discussion at hand.)

nonlocal 'leaks' that assigning an new identifier declares a new local variable. This is part of the abstraction not a leak.

I don't think I understand your other question.

What is your definition of primitive?

I like the part where he notes that this isn't a real problem except that it creates centithreads of argument from people who believe it to be a real problem. As I grow older and less idealistic, I'm more sympathetic to the notion that problems in theory but not in practice aren't really problems.

I write "real code" in Common Lisp, and I used something like what he dismissed as "useless in practice" twice today to make my code more readable and maintainable. I do not consider myself a Lisp advocate when I bemoan the lack of sane scoping semantics most similar languages just get right; I just want to write better code.

Yes, but Lisp code and Python code aren't the same when program in the native idioms.

In Python if your function references are buried within each other deeply enough that this is even an issue you're already doing it wrong. This is not because Python is lacking power, but because you're not using the power Python actually has. With the generators, comprehensions, objects that implement __call_, bound method references, and all of the other tools available to you, you just shouldn't be defining functions in functions in functions. In Haskell where it's all functions everywhere, a function that is actually ten layer nested functions calling each other makes sense. (When using the "do" syntax sugar, every line that has <- is another nested function, so if you see ten <-s, you've got a ten-layer deep nested function. (Or eleven, if I've committed a fencepost error.))

If you're still fighting with Python by trying to jam functions in three layers deep, take a moment and try to learn to be more idiomatic. If you're in Haskell or Lisp and you never have functions three layers deep, take a moment and try to be more idiomatic. In practice, you do not need this in Python.

_But_ the addition would be useless in practice because this problem almost never arises in real code. Its only purpose would be to answer Lisp and Ruby advocates who want to say that Python doesn't really have closures.

A great example of the Blub paradox in action ;)

Yeah, and Haskell programmers love having it pointed out that their language makes them stupid by not allowing them to do this ;)

Oh wait. Nobody says that about Haskell programmers? They only say it about Python? Huh. When they both have this same shortcoming with their so-called "closures"? Wonder why that is?

Anyone who's reached that point with Haskell is comfortable with the idea of immutable state.

Anyone who's reached that point with Python is very comfortable with the idea of mutable state... everywhere else.

Which is funny, because in both cases they're symptoms of trade-offs made in the language design.

But a Haskell programmer who understands this and decides to accept it is just considered normal. A Python programmer who understands this and decides to accept it (as in the linked post) is declared to be a Blub user, which is essentially a polite-enough-for-HN way to say "LOL STOOPID NOOOB".

Not allowing what? Rebinding of variables? That's what IORefs, MVars etc. are for.

I believe it is a rare example of a symmetrical Blub paradox. We know objects are a poor mans closures, just as closures are poor mans objects.

A great example of the Blub paradox in action ;)

Actually not. There is no code you can write using closures modifying external state that cannot be trivially rewritten in Python using a one-element array in the outer scope which you modify in the inner (without assigning). Therefore Python has that feature in effect. It just doesn't have a piece of dedicated syntax for it.

As I understand CPython, problem with this, is that python's "closure" does not contain reference to parent environment frame, but it's copy, so this is not only syntactic issue. Also putting reference to parent environment would create reference cycle, which was clearly undesirable in times when python did only reference counting.

No, I don't think so. Closures in Python are very similar to Scheme (or modern Lisps). Python doesn't have an equivalent to set!. Instead assignment both defines a local variable (if it doesn't already exist) and sets it. You can use "global" or the new "nonlocal" keywords to prevent the definition.

Personally I don't mind the Python approach. It's more convenient than let/set! and I very rarely miss being able to set a variable in an outer scope.

One of the most powerful features of Smalltalk Blocks is the ability to reference everything in the parent context. This gives one the ability to Don't Repeat Yourself in even the stickiest of situations. A lot of tricks for not repeating yourself involve passing a Block from the current context into a function, where it is executed, possibly inside an exception handling block. 99% of everything you'd be hard pressed to avoid repeating yourself with can be taken care of using tricks based on this idea.

EDIT: Come to think of it, this is just a good-enough substitute for a fraction of the stuff you can do in Lisp with Macros.

Smalltalk's blocks came from Lisp. It's called lambda in Lisp and is arguably the most important idea in Lisp.

Right, and Smalltalk had for a long time no lexical binding, thus returned blocks were never closures. Just as in early Lisp dialects.

Instead of adding an "intermediate" keyword, I think JavaScript's approach is preferable. That is,

> var x = 0

Creates a new variable.

> x = 0

Assigns a variable if it has already been declared in a parent scope, or creates a new global variable otherwise.

Of course, this work break all existing Python code, so it would never happen.

This semantic is pretty much anathema to Python, which tries to be explicit and to default conservatively. I'm pretty experienced with each of these languages and I routinely forget to qualify variables with "var" and end up stomping on globals.

I think you're throwing the baby out with the bathwater. Using a keyword like "var" is probably the cleanest way to define where the variable is defined. And we don't need to have the default case be implied global. If you reference a variable which has not been declared with "var", have it crash and die. I think this is the best of both worlds.

The only way your argument makes sense is if variables are default-global and tightly scoped only when the keyword is applied. If you adopt that scheme, Python doesn't suffer from the syntactical inconvenience this article talks about.

Otherwise, you're making the same point that the article is making, which is that you could introduce a scoping keyword to make it easier to deal with closed-over variables.

Either way, I don't care; I'm a Ruby developer now.

It's not the same point: you would never have to explicitly tell the interpreter where the variable is (global, local or nonlocal). If you required all variable declarations to use var, the interpreter could automatically determine the scope of the variable when it is referenced or updated. This is cleaner IMO.

How is that the only way that my argument makes sense?

I'm defining a slightly different Python syntax, in which you always have to use "var" to declare new variables, otherwise the code breaks. That includes globals. With this structure, normal closure syntax is very clean:

    var x = 0
    
    def foo():
        var x = 1
        def blah():
            x = 2
        blah()
        return x
    
    print(foo()) # prints 2

My roommate (http://thejoshwolfe.blogspot.com/) pointed out the flaw:

The failure of the "var" idea is this code:

    def getDataFile():
      "returns the string 'data/data.dat'"
      var name = "data.dat"
      dir = "data"
      return os.path.join(dir, name)

now you've clobbered the global function "dir" without realizing.

Well no, if you can't declare any variable without 'var', the ' dir= "data" ' line is very explicitly assignment and not a variable declaration. You can still make the mistake but the piece of code would stand out.

The problem in javascript is that "a=12" does create a new variable, even if it's a global, and that many people just go through their code declaring only globals.

If you did get a syntax error everytime you did that, you'd soon have the habit of putting var in front of every new variable you declare.

That's true. All things considered, I think I'd rather have the "var" implementation, crash when using an undeclared variable, and even with the ability to clobber globals if you forget to use "var".

Ah, that makes sense. Thanks for being patient with my questions.

Maybe, but if globals are rare the asymmetry to reduce noise may be worth it.

The common case should be short and implicit, and the uncommon case should be explicit. Here, the common case is wanting to create a new variable inside a function, rather than stomp on a global. Instead, JavaScript forces a programmer to explicitly state something that he wants 95%+ of the time, with a nasty failure case when he forgets.

or creates a new global variable otherwise

That is the worst possible result. Requiring an explicit local or global keyword is reasonable, defaulting to local is reasonable, defaulting to global is insane.

Good point. Though defaulting to any sort of variable declaration could be problematic. You might be creating a new variable when you think you're updating one. I like AndyKelley's crash-on-default proposal best. This would also help the compiler in optimizing.

I agree! If you say:

x = 5

I think there should be a "var x" either locally or globally -- SOMEWHERE -- and actually there should be a compile error if "var x" is missing.

> is preferable ... > or creates a new global variable

You're the first person I read who actually defends this feature of JavaScript. (Crockford lists it in his "terrible parts" I believe.)

I also dislike a lot of what's supposedly JavaScript's "nicer" parts. I guess I just have bad taste :)

I like it because it involves one less keyword in the grammar vs the Python 3 solution. Though tptacek makes a good point (I haven't myself had trouble with that, but I can easily see why others might.)

Right, and nonlocal has been added to Python 3 for just this purpose (and maybe it's in 2.7? I forget).

Does it actually reference the parent, or does it just make a copy? One is a closure, the other is an implementation artifact.

It doesn't make a copy. It creates a special object (which you don't see unless you're digging in the internals) called a "cell" which holds a reference to the object for as long as the function exists. To see for yourself, look at `somefunc.func_closure` -- it'll be `None` on regular functions, and a tuple of cells on a closure function.

From a quick test, it appears that it can reference everything the parent can reference and does not make a copy.

"I don't know whether Guido will ever add it for, er, closure, but I do know that I have _never_ _once_ needed this keyword in hundreds of thousands of lines of real Python code I have written."

This is the blubbiest non-ironic statement I can remember reading.

Could you give examples of useful closures (actually multi-line lambda) ? I am relatively proeficient in python, and am learning javascript, and I really don't like how non trivial anonymous functions are pervasive in the later. I understand it may just be habit, though....

OTOH, one nice aspect in say ruby is the combo "real" anonymous functions + the syntax f a equivalent of f(a), which is what makes DSL possible in ruby (and impractical in python in general). I think it comes at a price in readability, though.

I always wondered what closures actually were. I'd read up on them, but apparently Python didn't have them so I always felt that I was missing something.

Thanks for clearing this up.

A lot of people don't seem to understand the difference between closures and lambdas, and a lot of people get it backwards as soon as someone points it out to them. Good to know there's one clear demonstration out there.

Actually, if you could pass this on to people when you hear them refer to lambdas as closures, that would be pretty awesome.

The crappy thing about Python closures is that you have to define the function above where you call it. I think this kind of puts the cart before the horse and messes of readability.

JavaScript, on the other hand, allows you to define the inner function anywhere in the local scope and makes closures much more attractive, IMO.

Changing this may mean big changes to the interpreter and so may not be worth it. I wonder if lambda created closures are any different...probably not.

You can of course define single line lambdas inline, and I think those are quite attractive, but when you want to get multiline lambdas you get in quite a sticky situation with Python. For a simple example, see the top answer here: http://stackoverflow.com/questions/1233448/no-multiline-lamb...

In order to make it work Python would have to introduce a scope-blocking structure like braces, parens, or an 'endlambda', all of which I think are steps in the wrong direction, especially given Python's great feature of semantic whitespace. I'll take a clean function defined above over braces.

I do know of a way to hack a multi-line Lambda with existing syntax, though, but I have no guarantees about it. I don't think I've used it in serious code, since it's just real ugly.

  k = lambda x: eval(compile("""
  print x
  y = 4
  print x + y
  x = 8
  print x
  """, '', 'exec'))

  x = 5
  k(2)
  # 2
  # 6
  # 8
  print x
  # 5

By the way, what is the idiomatic way to do this in Python?

    (let ((x 1))
       (defun less () (incf x))
       (defun more () (decf x)))

Reply on news.ycombinator.com