Lambdas and Functions in Python | Hacker News Reader

Lambdas and Functions in Python

thepythoncorner.com

62 pointsmastro3544 comments

Python's lambda also has an interesting design flaw that anyone using it will come up against sooner or later [1]. The way it interacts with Python's scoping rules is quite broken. This hinders using it as freely as is possible in, eg, Clojure.

  >>> fs = [(lambda n: i + n) for i in range(10)]
  >>> [f(4) for f in fs]
  [13, 13, 13, 13, 13, 13, 13, 13, 13, 13]

[1] http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/

Python stores the captured variables in __globals__ which is shared with all the different lambdas for some odd reason:

>>> fs[0] == fs[1]

False

>>> fs[0].__globals__ == fs[1].__globals__

True

>>> fs[0].__globals__['i'], fs[1].__globals__['i']

(9, 9)

Kind of weird?

--edit--

  >>> def f(i):
  ...     return lambda n: i + n

  >>> fs = [f(i) for i in range(10)]

  >>> [f(4) for f in fs]
  [4, 5, 6, 7, 8, 9, 10, 11, 12, 13]

It works if it has somewhere to store the captured variables so just don't pollute the global environment and all is fine...

I think you're looking at Python 2, because in py3 the list comprehensions create a nested scope for the loop variable to be local in. -- i.e. 'i' would be in a local environment, not in globals. This doesn't change the behavior of these particular closures, since i is still mutated in the loop.

Python 3.6.5 (default, Apr 4 2018, 15:01:18)

...even weirder:

>>> fs[0].__closure__ == fs[1].__closure__

True

Since I don't have py3 installed at the moment, I can't check anything, but I'd guess this is two closures with distinct ids but the same environments, and == is doing a structural comparison rather than an identity comparison. Wish I could look into it right now.

  >>> fs[0].__closure__, fs[1].__closure__
  ((<cell at 0x7f4e9111b048: int object at 0x7f4e98e99500>,), 
   (<cell at 0x7f4e9111b048: int object at 0x7f4e98e99500>,))

So lambda isn't broken it's the list comprehension going all wonky with its local variables methinks.

All right, I looked into this using repl.it. The __closure__ field is not actually a closure in the usual sense of the word -- it's the environment of the closure, and fs[0] and fs[1] are expected to share the same environment. If you look at the identities of fs[0] and fs[1] themselves, they are different. It's as I said originally, just with a confusing name for the environment field.

A lambda is no different in scoping rules than a regular function def. If you put a def in a loop, you'll find the same behavior.

Closures are only created for functions defined inside other functions. Thus the term non-local. They're variables that are neither local nor global.

Don't call it broken when it works as intended.

“Works as intended” but also “unintuitive to most users” is still broken. Disclaimer: I have hated python ever since falling out of love with it 8 years ago for reasons just like this one so I’m definitely biased.

Since programming is a rapidly growing field and Python is the first language for many, I suspect that Python's scoping rules are in fact intuitive to most of its users. The ones who stumble are those with intuition built by some other language.

I'd argue that's the opposite. Python was the first dynamic language I learned. The scoping rules in Python didn't bother me because I didn't know any better, not because they're intuitive or well designed.

Fair anecdote. I'll offer that in my experience teaching hundreds of students, most don't have trouble with the cascade from locals to globals to builtins, and quickly grasp that globals are looked up for each call. That closures work at all confuses the ones who are thinking carefully about local scope. I may have a biased sample of students, but I think it's decent evidence.

As another comment said, this works the same way in def and lambda. This same thing happens in Javascript:

http://markheath.net/post/javascript-closure-loop-gotcha

“Isn’t it better?”

No!!!!!!!!

The original code looks by far the best to me. Why over-complicate?

Because it's almost the same thing copy-pasted repeatedly; textbook boilerplate. I know it's a matter of taste, but the last one also looks by far the best to me.

Well, actually I find the last code far better and easier than the original one. As I wrote in the article, if you want to add a factorial method you just need to change the catalog of the function provided by adding this line

“!”: lambda x: math.factorial(x),

However, it’s probably a matter of taste... but I do like the last code more. :)

I'm a big fan of first-class functions (or similar) and functional code, but lambdas are in the end just syntactic sugar. Why not just write:

"!": math.factorial,

The lambda wrapper isn't doing anything here.

I also thought that, but `math.factorial` is a builtin, and so you cannot get the signature of it, and the code can't determine how many operands the command needs. This is similar to the `self.stack.clear()` example. (`signature(math.factorial)` gives "ValueError: no signature found for builtin <built-in function factorial>")

Most of the code by volume in this post's final product (and most of the ugliness you're probably seeing in it) comes from dealing separately with the arity of different operations (`compute_operation_with_one_operand`, `compute_operation_with_two_operands`, etc) -- which retains a whole lot of copy-paste-but-change-one-small-thing-in-the-middle boilerplate.

That can be taken further and generalized out.

It's been a while since I did much Python metaprogramming-navel-gazing, but if you can't do something to automatically detect the arity of the operations' lambdas/functions, you can at least explicitly annotate the operations data with their arity -- which is already implicitly being done (as code instead of data) in `compute`'s 3 if-statements + the 3 "compute_with_N_operands" functions.

From there, you'll only have 1 case once, instead of 3 cases with each one manifesting in 2 different places. And just about half of the code disappears.

----

edit: since adjkant already covered the "automatically detect the arity of the operations' lambdas/functions" approach to removing all the cases, (https://news.ycombinator.com/item?id=17329772), here is the gist of what I meant by "explicitly annotate the operations data with their arity":

    import math, operator

    class rpn_engine:
        def __init__(self):
            self.stack = []
            self.catalog = {"+":    (2, operator.add),
                            "-":    (2, operator.sub),
                            "*":    (2, operator.mul),
                            "/":    (2, operator.truediv),
                            "^2":   (1, lambda x: x * x),
                            "SQRT": (1, math.sqrt),
                            "C":    (0, self.stack.pop),
                            "AC":   (0, self.stack.clear)}

        def compute(self, operation):
            (arity, op) = self.catalog[operation]
            operands = reversed([self.stack.pop() for _ in range(arity)])
            return self.stack.push(op(*operands))

Someone replied with and then unfortunately deleted an extremely good suggestion.

Instead of each operation taking a variable number of numbers and then either returning a value or directly changing the state of the stack (which was also a bug in my previous gist, in pushing the result of C and AC): homogenize the types of the operations by making all of them take a stack and return a stack.

You could also wrap all of the basic arithmetic functions with a higher-level function so that, eg, `add` would still not actually have to be aware of the stack:

    import math, operator

    def stackFunction(arity, function):
        return lambda stack: stack[:-arity] + [function(*stack[-arity:])]

    class rpn_engine:
        def __init__(self):
            self.stack = []
            self.catalog = {"+":    stackFunction(2, operator.add),
                            "-":    stackFunction(2, operator.sub),
                            "*":    stackFunction(2, operator.mul),
                            "/":    stackFunction(2, operator.truediv),
                            "^2":   stackFunction(1, lambda x: x * x),
                            "SQRT": stackFunction(1, math.sqrt),
                            "C":    lambda stack: stack[:-1],
                            "AC":   lambda stack: []}

        def compute(self, operation):
            self.stack = self.catalog[operation](self.stack)

        def push(self, number):
            self.stack.append(number)

The last version looks much better to me, and much more extensible.

This code can look a lot better by replacing the compute function with this:

https://pastebin.com/fMtc8Git

Yep! Good call! ;)

It looks like that reverses the arguments? You could do

    args = self.stack[-n:]
    del self.stack[-n:]
    self.push(function_requested(*args))

or maybe

    self.stack[-n:] = [function_requested(*args)]

Yeah oops coded that up quick. Either way you fix it, it’s far better than a function for calling each number of args and keeps the code readable.

For fun, an updated version that cleans up other things as well as the ordering, not relying on signatures of lambdas, etc.

https://pastebin.com/HF41J8KQ

It's interesting to note that a dictionary ("catalog") of values is essentially the same concept as an object. This is clearer in Javascript, where `{...}` notation is called an "object" rather than a "dictionary" and the `.foo` and `["foo"]` notations are mostly interchangable.

Storing functions in a dictionary is very close to storing methods in an object. The only differences are sharing state via a `self` (or `this`) argument, and inheritance chains (either from classes or prototypes).

Also interesting to note that many of the author's functions, like `lambda x, y: x + y`, only make sense because the underlying language (Python) thought that constructs like `+` shouldn't be functions for some reason. Many languages seem to do this, e.g. here's a bug report I raised for PHP https://bugs.php.net/bug.php?id=66368

When we introduce arbitrary distinctions, like dictionary vs object, named vs anonymous functions, functions vs operators, etc. then we end up with all of the choices and complications like those shown in this article.

One of the principles I try to follow when programming is to avoid making distinctions, or breaking symmetries, unless my hand is forced by the underlying dynamics. Experience has taught me that introducing distinctions in one place just introduces complication elsewhere (e.g. more cases to handle when metaprogramming).

I don't know why operators are the way they are in Python, but to me it looks like an oversight, which ultimately let to the "operator" module, exposing all operators as functions

At the risk of oversimplifying/getting the subtleties wrong, operators are syntactic sugar to call magic methods. You could still call the magic method itself:

  >>> a = 1
  >>> a.__add__(2)
  3
  >>> b = []
  >>> b.__len__()
  0

It's rare I have to use the "operator" module, and I find Python pretty well thought out w.r.t. consistency. All in all, seems fine to me.

> You could still call the magic method itself

Only if you have the first argument available. As a silly example, we might want to do the following (in analogy to the built-in `sum` function):

    product = lambda vals: reduce(operators.__mult__, vals, 1)

We can't use a bound `__mult__` method for situations like this.

More generally, looking up methods from particular objects forces our code to be "first order", i.e. we have to explicitly deal with intermediate results which don't appear in higher-order programming. For the `product` case we could do:

    def product(vals):
      result = 1
      for val in vals:
        result *= val
      return result

3/5 of these lines deal with `result`, which doesn't appear at all in the `reduce` version. The same thing happens with stuff like composition too, e.g. the `x` in `lambda x: foo(bar(x))` when we could just `compose(foo, bar)`.

When code is bound up in methods, we're often forced to expose these intermediate values, in order to call their methods. Of course, we can abstract over such boilerplate too, with functions like `lambda o, m: o.__getattribute__(m)`, etc. but at that point we're fighting against the language.

PS: Yes, I know Guido doesn't like `reduce` and moved it out into `functools` for Python 3.

>Also interesting to note that many of the author's functions, like `lambda x, y: x + y`, only make sense because the underlying language (Python) thought that constructs like `+` shouldn't be functions for some reason

Actually python's + (and most operators) are available as functions, and allow for overloading as well:

  >>> import operator
  >>> operator.__add__(1,2)
  3

>Many languages seem to do this

"Many"? Unless you ever only touched Smalltalk, Lisp or some fringe languages, this is the norm for most languages -- operators are distinct from functions (same as expressions are different than statements). Whether this is a good idea or not, I wont go into, but it's not like this is something surprising, or something that merely "many" languages seem to do. Not just many, but almost ALL mainstream languages (C, C++, Java, C#, etc) do it.

>operators are distinct from functions

>c#

As python, C#'s operators are backed by functions - they're just static ones on the class.

personally I always always hated operation segregation

Lisp operators are distinct from functions. It's just that arithmetic operations are functions; Lisp dialects use operators for various other constructs that cannot follow the function call evaluation rules.

C++ sort of makes operators functions. Any overloaded operator is a function. If a is a class object then a + b can be spelled otherwise as a.operator+(b). This doesn't work for the built-in definitions of the operator, as in 2.operator+(2).

The built-in + operator in C++ can also do things that used to be impossible in C++ until its template system became more powerful. Namely, the result type of the built-in + is deduced from both operands and can be different from both of them. If C++ were being redesigned clean-slate, it would likely be that operator overloading is allowed for the non-class types like int, and all the built-in operators are defined in terms of templates, whose deduction/instantiation reproduces all the rules.

The thing that makes it beautiful in Lisp is multiple dispatch. In higher order functions, + is still just +, not some function that's already bound to the left hand side.

> Actually python's + (and most operators) are available as functions, and allow for overloading as well

Yes, operators like `x + y` desugar to method calls like `x.__add__(y)`, which is yet another somewhat-arbitrary distinction (operators, functions and methods). The fact that `operator.__add__`, etc. are available doesn't remove the distinction between operators and functions; we can do `operator.__add__(x, y)` but we still can't do `+(x, y)`. Those `operator` functions are just a way to avoid defining our own `lambda`s, but even then it doesn't save very many keystrokes.

> Not just many, but almost ALL mainstream languages

In other words, "many" ;)

I don't think it's useful to group languages into "popular" and "fringe", since the cost/benefit of a language construct depends more on the paradigm than the popularity. For example, you mention C, and one way to think of C is as a (mostly) "portable assembly"; in which case it can make sense to distinguish operators like `+` (which are essentially machine instructions) from functions (which are more complicated subroutines involving jumps, stack manipulation, etc.).

Highly dynamic languages like Python don't emit machine instructions so that argument doesn't hold (they do emit opcodes for the interpreter, but the language designers get to choose what those are); other arguments can be made either way, but they depend on the paradigm. The indirection given by `__foo__` methods fits the language quite well, but it doesn't preclude those symbols being used as functions as well.

I don't hold much opinion on the use of operators in Java and C#: on one hand they allow high-level OOP with dynamic dispatch, first-class functions, etc. but on the other hand the body of a function/method mostly harks back to structured programming (if/else, for/while, etc.).

You also mention distinguishing between expressions and statements. That's another thing that seems to make (a little) more sense in something C-like, but less so in a high level language. Python suffers for this, since it makes `lambda` much less useful. On the other hand PHP gets this right by simply making statements a type of expression, although it shoots itself in the foot when it comes to precedence, e.g. `$x = $y xor $z;` is the expression `($x = $y) xor $z`.

> it's not like this is something surprising

I never said it was. Although I do like to view common things from new perspectives: in this case that operators are distinct from functions and methods, which we often take for granted, but (a) doesn't have to be that way (b) can cause unnecessary work (like this whole article) without us necessarily realising where it stems from and (c) has no real reason to be so widespread, and mostly comes down to self-reinforcing logic like "others are doing it, so we'll do it too".

> Unless you ever only touched Smalltalk, Lisp or some fringe languages

I wouldn't say people who've only used such languages would think in this way; rather, I'd say that even brief exposure to different ways of doing things can permanently alter the way one thinks about programming, and this is usually for the better (e.g. it's probably a bad idea to program in Python as if it had backtracking, or return-type polymorphism, or delimited continuations, or whatever; yet it can be invaluable for gaining a big-picture perspective and avoiding "everything looks like a nail")

Has anyone ever experienced a time where assigning member variables on a function made sense?

The thing I love about python is the freedom to do whatever I want and the responsibility to almost never do it. I crave examples of exceptional cases.

The most common cases are decorator functions, e.g. one that counts the number of invocations: https://stackoverflow.com/questions/44968004/python-decorato...

That code seems suspect. Throwing and catching BaseException is bad/has some weird consequences and probably unintentional - I assume the author meant to use Exception? Pathologically avoiding `elif`. Misspelling `inspect` in the code examples which are ostensibly verbatim Python sessions.

In general, lambda functions are less necessary in Python because functions are first-class. The lambda syntax is useful for extremely trivial functions, e.g. `sorted(a, key=lambda value: value.foo_bar())`, but otherwise "normal" functions work just fine/better - especially since you can define a function wherever you want. (This is how it's supposed to be. E.g. in Java or Javascript, code using many lambdas can be a royal pain to debug.)

Maybe related, I dislike the use of `map` and `filter`. Almost all real-world cases are more readable using list/dict/set comprehension or generator expressions. The only thing missing now is assignment expressions (PEP 572, [0]) so I can do a map + filter in one comprehension.

[0] https://www.python.org/dev/peps/pep-0572/

Moving from Ruby to Python, for me one of the initial frustrations was being unable to define multi-line anonymous functions, though eventually I accepted the wisdom of Python's conservative philosophy over Ruby's "Do what makes you happy".

But I'm glad that Guido et. al decided to keep lambda around. It feels like the right tradeoff with syntactic sugar, as one-line anonymous functions offer just enough utility in a limited scope. In contrast, having to formally define trivial functions feels like too much overhead and clutter.

Hi guitarbill, the article is intended to point out how to review the original code. I figured out a bad code written by anyone else and I tried to make it better.

About the lambdas you are right, they’re good just if you have trivial functions, like in other languages. If you need more “power” just use normal functions.

PS: if I mispelled “inspect” or anything else... sorry, I’m not a native speaker, I wish I could speak and write better but the truth is that my English sucks... :(

> I’m not a native speaker, I wish I could speak and write better but the truth is that my English sucks

For what it's worth, I actually would never have guessed that; in fact, I think your English is better than that of many native English speakers I know. Keep at it!

Fair enough, I wasn't sure if the review was just fiction - a good idea to base the article on.

However, I do want to refute that English skill has anything to do with the things I'm pointing out. Your English is fine, plus I don't care, as I also had to learn English as a second language. The issues are purely technical.

First, `BaseException` is introduced in the refactored versions, not in the original code - so it's hardly Peter's fault. Second, the misspelling is in code, and it stops the REPL example from working ("from inpect import signature") - the REPL code looks like it's copy & pasted from the console, but it can't be.

So "suspect" was probably the wrong word to use here, even if that code has been altered. I hope you can use the feedback to improve the article, even if the feedback is worded poorly!

For the BaseException you’re right.

About the mispelled word I didn’t notice it... that’s weird, it IS a copy and paste, probably when I pasted it I lost that line and I rewrote it?

Can’t remember, I posted the original article on Medium on December 2017. However I will surely modify the article next week to address that typo, thanks!

Reply on news.ycombinator.com