The fate of reduce() in Python 3000 (2005) | Hacker News Reader

The fate of reduce() in Python 3000 (2005)

artima.com

70 pointstosh37 comments

> Also, once map(), filter() and reduce() are gone, there aren't a whole lot of places where you really need [lambda].

By far the most common use of lambda for me is the "key" argument for sort() or sorted(). As in something like:

foo.sort(key=lambda x:x[2])

Yes, there are itemgetter and attrgetter in operator module, but I always have to look up their names and writing the lambda expression is simply faster.

itemgetter and attrgetter are compiled C functions called by cpython https://github.com/python/cpython/blob/master/Modules/_opera...

They mean faster to write, not that it's necessarily faster to execute (though the difference is likely immaterial).

Right, though for those uses something like the swift {$0[0]} or haskell (!!0) just seems a lot more of convenient.

Yeah, I don't get the "hate" for the functional operations, though comprehensions have better syntax

And no, I don't want to 'def' every small function I might need, pep-8 recommendations aside. Key as mentioned is one use, but there are other cases where it's especially useful

> I think dropping filter() and map() is pretty uncontroversial; filter(P, S) is almost always written clearer as [x for x in S if P(x)]

Performance reasons aside, I strongly disagree with this statement. Anyone can see that `filter` is going to apply the predicate to the iteratable in the first statement. The second statement is super awkward, declares a temp var `x` which to my limited Pythonic eyes looks like it's just calling the identity function on `x` that comes back from the "filter".

Again, I have no doubt Python is optimized to perform better with the second form, but that seems a limitation in the interpreter. The syntax to non-Python devs is pretty raunchy imho.

I am used to haskell's syntax, does this become more readable once you get used to it? The haskell equivalent would be

    [x | x <- s, p x]

In many senses this is a style thing, but:

There is going to be a pretty sharp break between people who accept that working with a collection deserves special syntax support and those who don't. The readability issue for me isn't that the syntax doesn't look nice, it is that I've invested years into making a very efficient little mental space that deals with "data2 = fn(data1)" style constructs and I want to use it here like I do everywhere else.

Some silly-high percentage of for loops can be cast into this data2 = fn(data1). Why am I being told that it isn't readable? If this isn't easy to read, then something is wrong with the function syntax.

A chaining operator, to recast fn3(fn2(fn1(data))) => data > fn1 > fn2 > fn3 with reduce is much more readable, debuggable and easy to reason about than having to remember what syntax todays language uses to talk about collections and how that works in to the flow of data in my program.

I agree with you for the example with a named function P, but as Guido points out, filter predicates are typically simple expression which you don't want to create a named function for. So it would be say `filter(lambda x: x < 17, S)` versus `[x for x in S if x < 17]`. I think the second one is nicer even if the repeated x is ugly. Neither is great though. The comprehension syntax assumes filters are accompanied by maps. A filter without a map is a somewhat ugly special-case.

The real issue is the lack of a lambda syntax with a pythonic feel.

The list comprehension syntax is just set notation with Python keywords instead of symbols. Haskell, JavaScript, and Julia have similar syntax. It's also similar to the equivalent loop.

Which of these is clearer?

    [x * y for x in S if P(x) > 0]
    map(lambda x: x * y, filter(lambda x: P(x) > 0, S))

reduce() is still available in the functools module.

I highly recommend to become familiar with the functools, itertools and collections modules to anybody using Python and not using them regularly.

I’ve really come to miss reduce in Python’s builtins since std::accumulate has become a staple in my C++ development. It seems surprising that with Python 3’s increased support for functional concepts, it would relegate reduce to the standard library.

Not really, increasing support for functional concept was never a goal of Python 3 (or at all), it was mostly incidental and it did regress in other ways than just shoving the (relatively little used due to the existence of min/max/any/all) out of the builtins e.g. while extended unpacking was added, unpacking iterable function parameters was removed.

+1 for the std::accumulate reference, and I'll add an honorable mention of std::transform which since C++17 also runs in parallel.

This is broadly true across language. In Java, it was Collections, List, Elixir it's Enum, Python is your mentioned stdlib namespaces. Knowing your collection-ey stdlib functions will only help your code be more idiomatic and likely performant.

Indeed, those modules are essential for doing anything in Python beyond small scripts.

I spent some time comparing Python's [0] map/filter and comprehensions against Snigl [1].

Comprehensions are just plain faster, especially if you perform say filter and map at the same time.

That and a lambda implementation that only a mother could love, lack of tail call optimization etc.

What bothers me about comprehensions is they come with a separate set of rules, like LOOP in Common Lisp or templates in C++; but I'm pretty sure the restrictions are part of what makes them fast.

[0] https://gitlab.com/sifoo/snigl/blob/master/bench/seq.py

[1] https://gitlab.com/sifoo/snigl/blob/master/bench/seq.sl

Agreed, this made my eye twitch:

> filter(P, S) is almost always written clearer as [x for x in S if P(x)]

sure, Guido, if you think it's clear that the variable x is in the enclosing scope...

Python 3 changed this, iirc. Comprehension no longer leak.

It's still an assignment operation, and you can still get weird effects in the enclosing scope. I did something like this by accident:

    Python 3.7.1 (default, Nov 28 2018, 21:47:25)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: lookup = {}

    In [2]: [x for (x, lookup[x]) in ('ab', 'cd', 'ef') if x in lookup]
    Out[2]: ['a', 'c', 'e']

    In [3]: lookup
    Out[3]: {'a': 'b', 'c': 'd', 'e': 'f'}

The whacky thing there isn't the list comprehension though, it's the implicit assignment of `lookup[x]`. If you desugar it some (which isn't really possible, but

    [(x, lookup.setdefault(x, y)) for (x, y) in ('ab', 'cd', 'ef')]

comes close), what's happening makes a lot more sense. It works exactly the same way as any other scope: first checks locally, then nonlocally or globally. You're modifying a nonlocal. The syntax you're using to modify the nonlocal is surprising, but that's the surprising part, not the ability to modify a nonlocal.

I think I was personally surprised by two assumptions that I implicitly held, both of which disappear when you write them down:

1. The syntax of comprehensions led me to think that they introduce some kind of a variable binding, such as `let` in many Lisps, and that binding `lookup[x]` would have been a syntax error. Of course Python didn't introduce anything like that for comprehensions, and it's just a normal assignment operation, with an extra scope inserted in Python 3 to stop `x` leaking into the enclosing scope.

2. Because of the swap idiom `x, y = y, x` I thought tuple assignment happens somehow in parallel, so my code should have caused an undefined variable error, even if it is not a syntax error. But of course it just evaluates the right hand side first and then assigns to the left hand side in left-to-right order, which the language reference https://docs.python.org/3/reference/simple_stmts.html#assign... notes "sometimes result[s] in confusion" (near the end of the section).

I'd desugar it like this:

    >>> lookup = {}
    >>> for (x, lookup[x]) in ('ab', 'cd', 'ef'):
    ...     pass
    ... 
    >>> lookup
    {'a': 'b', 'c': 'd', 'e': 'f'}

This makes is much more clear that there is just a double assignment, and that comprehension are pretty close to regular `for` statement.

(this was a great example btw -- I write python3 a lot, and I was staring at it for a long time before Icould figure it out)

It appears to work without list comprehensions as well:

    Python 3.7.1 (default, Nov  6 2018, 18:49:54)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.1.1 -- An enhanced Interactive Python. Type '?' for help.

    In [1]: lookup = {}

    In [2]: a, lookup[a] = 'xy'

    In [3]: a
    Out[3]: 'x'

    In [4]: lookup
    Out[4]: {'x': 'y'}

I wasn't previously aware that this was possible, but I guess it's no different from

    a = 'x'
    lookup[a] = 'y'  # i.e. lookup['x'] = y

Okay that's pretty sweet actually. I write a lot of legacy-compatible code, so I'm kinda behind on Py3. Maybe that's my 2019 project.

> Comprehensions are just plain faster, especially if you perform say filter and map at the same time.

Comprehensions are often slower if you're applying a function regardless, definitely so if that function is a builtin.

Discussed in 2008: https://news.ycombinator.com/item?id=225764

Why were map and filter kept? What can you do that cannot be as easily done with generator expressions?

Coconut has many examples of functional programming techniques that cannot be easily done with python: https://coconut.readthedocs.io/en/master/DOCS.html

Strangely FP is not popular among python developers.

It's a pain in the ass to write functional code in python because the syntax is not optimized for it. A functional programming language makes lambda and (partial) application as cheap as possible, which Python does not.

is coconut used in production? seems like a really bad idea?

It's python after all, so why not?

I think you can get 95% of the benefit of FP just by favoring abstractions which present a functional interface. Python hasn't really gotten in my way, in that respect, though it would be nice to have first-class persistent data structures with structural sharing.

I'm sometimes tempted to use it in loops, when

  for x in map(func, iterable):
      ...

is cleaner than

  for x in (func(y) for y in iterable):
      ...

and shorter than

  for y in iterable:
      x = func(y)
      ...

But that mostly only makes sense if you already have a suitable named function (e.g. str), and I usually end up going with the third variant anyway.

There's an analogous case with filter and `if not cond: continue`.

map and filter calls are trivially rewritten to generator expressions, but it's not necessarily an improvement.

> What can you do that cannot be as easily done with generator expressions?

Technically nothing, but at the same time if you already have a function (especially a builtin) map/filter is more convenient and can be faster. And they're first-class objects which comprehensions are not, which can be convenient.

The goal is not necessarily to prune all redundancies: reduce already existed when sum/min/max/any/all were added, despite the latter being fairly simple special cases of the former (not quite entirely for min/max, but not far).

Why bother having any control logic other than conditionals and gotos? Because they signal intent directly.

Does anyone know the origin story of reduce/filter in Python? It seemed interesting. I remember watching a talk by Guido. He mentioned that someone submitted a PR with it, and that it was very complete, so he merged it. But I'm curious what the decision making process was like for Python back then, or if there even was a formal process.

Programming language archaeology is a field that ought to get more attention.

Reply on news.ycombinator.com