Hacker News Reader

Comment by zmmmmm | Hacker News Reader

The worst part of python is the lack of utility functions for munging collections. But it sits at a slightly higher level than this - things that are idiomatic in other scripting languages like groupBy(fn), sortBy(fn), countBy(fn), collate, are all inexplicably like IKEA self assembled furniture in Python instead of first class entities. It makes lots of routine data munging and algorithmic work much more annoying than it needs to be (comparing to Groovy, Ruby, etc).

View on HN

Not sure what collate is supposed to be, but these idioms are hardly difficult if you're used to Python:

- groupBy is itertools.groupBy(lst, fn)

- sortBy is just lst.sort(key=fn)

- countBy is collections.Counter(map(fn, lst))

- Sibling comment mentioned flatten, which is just [item for sublist for sublist in lst]

More esoteric needs are usually met by itertools.

> - sortBy is just lst.sort(key=fn)

You mean sorted(). list.sort only works on list and in-place.

Annoyingly, itertools is one of those packages in the Python standard library with wordsruntogether-named functions, so it's really known as itertools.groupby instead.

The real issue is that python doesn’t really make it easy to define anonymous functions. If you want one with more than one line, you need to name it and move its definition outside the point of use. Quite annoying.

I always thought of that as a feature.

Excessive usage of anonymous functions is a major pain. JavaScript developers for some reason almost refuse to name their functions, even though it would make reading and reusing code much easier.

You can make lambdas in Python, but naming function is in my mind very much a "feature".

Hm, why is it more important to name a function than, say, an if statement?

Code readability, specially if it spans multiple lines and has conditional code into it.

Same applies to then and else branches.

In a code review I don't what to read a thesis inside of a conditional branch.

If you just have a short one or two lines of code in a callback, maybe it's not so important. Where I have an issue is when you have five nested anonymous functions. Why wouldn't you break those out into named functions?

Assuming you have five levels nested if, ignoring the fact that you might be doing something wrong, then you should name those, but that requires you to extract those code paths and put them into separate functions.

Because an if statement is very constrained in what it can do. Every if statement has two branches (maybe one is a no-op) and one of them executes. If someone sees an if statement, it is implementing a fork in the code path.

A function can do literally anything. Seeing a function call, it hasn't narrowed down the realm of what might be happening even a little bit.

"lambda f:" gives away very little information about what is about to happen. "if f:" gives away some information about what is about to happen.

Not a perfect rule of thumb, convoluted if statements do exist. But that is why functions are more in need of hints and comments than other control structures.

> "lambda f:" gives away very little information about what is about to happen. "if f:" gives away some information about what is about to happen.

I think that's an unfair comparison:

If 'the thing that happens' for "if f:" is 'branch on the truthyness of f', then I agree that's pretty clear. However, if we're going to ignore the content of the branches then we should also ignore the body of the function, so "lambda f:" is also pretty clear: 'parameterise with f'.

If we're worried that "lambda f:" could do pretty much anything in the body, then we should also worry that "if f:" can do pretty much anything in the branches.

> If we're worried that "lambda f:" could do pretty much anything in the body, then we should also worry that "if f:" can do pretty much anything in the branches.

That is the part common to both. If you take that common part out, you will be left with more information in the if statement than the function call.

In the function, you know what arguments it (might) be interested in. You know they are evaluated. But in the if statement, you know what values it is interested in, usually get the same amoutn of information about whether they are evaluated, much clearer guarentees about how they are used and that the aspect of them that matters is their truthiness. Also the code might be about to branch.

Put it this way - if(...) is a specific - and common - named function. It is a very easy argument to make that it is carrying more information about what is going on than a general anonymous function.

If you take that common part out, you will be left with more information in the if statement than the function call.

I don’t think this argument generalises well. Once you start working with structured data and more complicated control logic, a generic loop with an arbitrary body often provides less immediate information to a developer reading the code than a function that explicitly names a specific pattern of computation like map or filter.

There are many recurring patterns of computation that are much more specific than a generic map or filter, too. While imperative languages tend to have only a small number of built-in control structures, if we’re using higher-order functions then we can name as many of these patterns as we like.

For example, suppose we want to generate a list of the first few triangle numbers, which is a sequence of the form:

    [0, 0+1, 0+1+2, 0+1+2+3, ...]

Writing this in imperative Python code with a typical loop would give us something like this:

    triangle_numbers = []
    total = 0
    for n in range(6):
        total = total + n
        triangle_numbers.append(total)

Here’s an idiomatic functional version, which is real Haskell code using a standard library function that captures this pattern of sequence generation in just five characters:

    triangle_numbers = scanl (+) 0 [1..5]

Assuming you’re equally familiar with both programming styles, I think it’s fair to say that “scanl” tells you much more about what this code is doing than “for”.

The inner function here is just addition in both cases. However, the code within the Python loop is allowed to do whatever it wants, so the reader has to recognise the pattern surrounding the addition. In the Haskell version, the function you pass to scanl must take an accumulated value so far and a next value as parameters and it must return the next accumulated value to add to the sequence being generated, and that is all it is allowed to do. So you just need to write (+), which is Haskell’s notation for what Python calls operator.add or an equivalent lambda, and that tells the reader exactly how the next accumulated value is being derived at each step to build up the sequence.

Well, ok. But you've named all your functions here so I'm not sure which part of this you're arguing about.

    triangle_numbers = scanl(f, 0, [1,2,3,4,5])

gives me scant little information about what the value of triangle numbers is at the end of the operation.

    triangle_numbers = if(f, 0, [1,2,3,4,5])

    triangle_numbers = if(true, 0, f)

tells me that whatever f is something weird is going on. Commenting if statements to point out that they are branching code is foolish.

But you've named all your functions here so I'm not sure which part of this you're arguing about.

My intended point was that comparing lambda: to if: is apples-to-oranges.

Both styles in my example have an outer structure that builds a sequence, wrapped around an inner calculation that says how you build the next value in the sequence. The inner calculation here was simple addition, a one-liner in the for loop and (+) in the Haskell version. However, it could equally have been something more complicated.

If it were, a reader would still have to figure out what the body of the for loop was really doing. In contrast, the (possibly anonymous) function passed into scanl would still have a more constrained purpose, which is immediately known because scanl represents one specific pattern of computation. The implicit algorithm it represents has a hole that is a certain shape, and you can only pass it a function with the right shape to fit into that hole.

Put another way, if you’re programming in this functional style, it’s not writing the word lambda that tells you what the next piece of code is for, it’s passing that lambda expression into a higher-order function like scanl. Passing an anonymous function is like writing the nested block for a branch of an if statement or the body of a for loop. It’s the call to scanl that is roughly analogous to writing the if or for statement itself.

Lambdas are constrained by their type. If you see a lambda being passed to, let's say `map`, then (assuming you know the type of `map`) you know the type of the passed lambda.

Of course this assumes the lambda is pure, but in the kind of code we're talking about (pseudofunctional code with lots of higher-order functions) they should be pure for the most part.

A pure function (which a lambda typically is) answers a question. It's helpful to have words describing what question it answers. The next person to read the code will actually be able to read the words.

An if statement is a lower level mechanism for directing the code path within a function. If it needs a name, wrap it in a function. I often do.

Control statements rarely "need" a name, but if they are significant enough to be named they can be named by extracting meaningful and somewhat reusable functions.

An if statement has a name, it's "if".

It also has a condition and a body to execute conditionally.

    if (condition)
      body

is just a goofy way of writing

    if(condition, body)

Fully agree, if you want to see that taken to the extreme, read Android code where people apparently only know about anonymous Java classes.

Many Swing coders can also apply.

It is so hard to actually pack the functionality into a separate piece of code.

"Yeah, but then I have to look elsewhere"

Really? What about when a method is full of stuff like that?

Its readability is lost beyond hope.

Have you actually used other languages like clojure, haskell, kotlin, go or ruby? Python is the weird one here for not allowing multi-line lambdas. There are legitimate and actual useful cases for multi-line lambdas, disallowing them does more harm than good.

If the idea of multi-line alone compromises readability, then why even have multi-line blocks at all? Function, loop and conditional blocks should all at most have one line each. I hope you can see the absurdity of this.

Being weird isn’t sufficient evidence for being worse which is what you seem to be arguing.

Why is having to name a function doing more harm than good?

Your second paragraph is a straw man.

Weird as in it goes against well-established practices. If there are good, compelling reasons to disallow multi-line lambdas, other languages would have adopted it. But no, the opposite is happening, newer and recent languages actually have multi-line lambdas. Get this, even java adopted multi-line lambdas. Crazy right? What's wrong with everyone?

There's nothing wrong with having to name a function. But there is something wrong with not having a choice and being forced to name a multi-line function. Just as you don't want to name every sub-expressions, you don't want to name every blocks of code. Unnecessarily naming things pollutes the namespace.

Where's the strawman? The premise was that functions that spans more than a line should be given a name because they are "unreadable". I have showed that, by this logic, anything that spans more than a line is "unreadable". You can't make special exceptions and say this only applies to lambdas. Lambdas (aka higher-order functions) are blocks of code just as much as anything else.

Indeed, there are many such explicit omissions in python (switch statements, assignment expressions (before 3.8), increment/decrement operator, etc) that may seem to some as a bug, but it really is an explicit decision and part of what makes Python the clean and readable language it is.

do python lambda expressions not qualify?

https://docs.python.org/3/tutorial/controlflow.html#lambda-e...

Lambda expressions in python are limited to a single expression. The poster explicitly calls out wanting more than one line.

It's possible to execute statements in a lambda through exec. But it's more of hack than what it worth.

  print((lambda x: [exec('x = x + 1; result = x'), eval('result')][1])(1))

I don't understand why you wouldn't just define a function if you needed multi lines.

I don't understand why you wouldn't just define a function if you needed multi lines.

For the same reason that you wouldn’t necessarily define a function every time you wanted more than one line in the body of an if statement or for loop.

In a functional programming style, typically most of your control structures are represented as higher-order functions and you pass them other functions where you might use nested blocks of statements in an imperative style.

That is, where imperative pseudocode might look like this:

    for n in [0..10]:
        output_array[n] = n * n

some corresponding functional pseudocode might look more like this:

    output_array = for_each [0..10] (n -> n * n)

In some functional languages, it’s even idiomatic to write the supporting function so it reads more like this (borrowing Haskell’s $ notation, which avoids the awkward parentheses):

    output_array = for_each [0..10] $ \n ->
        n * n

Now you might want a multiline body for the supporting function in much the same situations that you might want a multiline body for the imperative for loop:

    for n in [0..10]:
        location = get_location(n)
        route = fastest_route_to(location)
        times[n] = average_journey_time(route)

    times = for_each [0..10] $ \n ->
        location = get_location(n)
        route = fastest_route_to(location)
        average_journey_time(route)

Similarly, if the logic you’re using in the inner part of the code is a self-contained concept, you might want to factor it out into a named function of its own in either case.

This functional style can be tidy and very flexible in languages that are designed to support it. However, I wouldn’t necessarily encourage it in a language like Python, precisely because Python’s syntax and language features don’t make it natural and concise to write like this.

IMO multi-line lambdas are unnecessary; they can't be easily documented or better maintained than a named function. If your anonymous function needs more than a statement it makes perfect sense to define it as a separate named function. Faster to find, document and maintain.

Do you have the same thoughts regarding JavaScript?

> move its definition outside the point

You can define a named function inside a named function and thus limiting its scope.

  def outer_func():
      def inner_func():
          print("hi")
  inner_func()

I honestly prefer that, anything function longer than a line probably could use a name that gives it some context, even maybe a short docstring, to make the code more readable.

It's not so much "lines" that are a problem; it's that lambdas cannot contain any statements.

For example, here's a multi-line lambda, containing only expressions:

    >>> lambda f, pred, xs: [
    ...   f(x) for ys in xs
    ...        for x  in ys
    ...        if pred(x)
    ... ]
    <function <lambda> at 0x1113e80d0>

Here's a one-line lambda, attempting to use a statement:

    >>> lambda x: (x += 1)
      File "<stdin>", line 1
        lambda x: (x += 1)
                     ^
    SyntaxError: invalid syntax

Note that the parentheses are to avoid parsing a '+=' expression with 'lambda x: x' on the left and '1' on the right:

    >>> lambda x: x += 1
      File "<stdin>", line 1
    SyntaxError: cannot assign to lambda

This restriction is problematic because of the following combination:

- Python statements don't compose as well as expressions; we can write one statement followed by another, and we can nest statements (or blocks) inside control statements (if, else, with, etc.), but that's about it. We can't assign statements to variables, we can't call a function on a statement, or return a statement from a function, etc. i.e. we can't abstract over statements (unless we try wrapping them in a named function and using that as an expression, but that can break the underlying functionality of the statement, e.g. 'def ifThenElse(cond, true, false):' will evaluate both branches when called).

- Python uses statements for a whole bunch of stuff. In particular it has a 'with foo as bar: baz' statement, rather than e.g. a 'with(foo, lambda bar: baz)' expression; the recent pattern matching functionality is a statement rather than an expresssion; defining a named function is a statement; etc.

Taken together, we can often find ourselves needing to introduce a statement somewhere in our code; that requires us to turn a 'lambda' into a 'def'; but 'def' is a statement, so we have to turn any enclosing expression into a statement too; and this propagates up to disintegrate whatever expression we had; leaving us with a big pile of statements, full of boilerplate for managing scopes and names.

itertools.groupby isn't really the groupBy operation that people would normally expect. It looks like it would do a SQL-style "group by" where it categorizes elements across the collection, but really it only groups adjacent elements, so you end up with the same group key multiple times, which can cause subtle bugs. From my experience, it's more common for it to be misused than used correctly, so at my work we have a lint rule disallowing it. IMO this surprising behavior is one of the unfriendliest parts of Python when it comes to collection manipulation.

https://docs.python.org/3/library/itertools.html#itertools.g...

Yes, for itertools.groupby to work as most people would expect, the data needs to be sorted by the grouping key first. That may obviously cause a significant performance hit.

It's common in other languages as well (at least Haskell) and a bit surprising at first. However, a `.sortBy(fn).groupBy(fn)` is easy and of similar efficiency and when you actually need the local-only `groupBy()` you're happy it's there.

A bit more expressive overall.

At least it is better than lodash' useless groupBy which creates this weird key value mapping, loses order and converts keys to string and what not.

yep, that's a good example of what I refer to as IKEA assembling your groupby. You need to put something like 3 parts together before it does what you want, and they aren't that intuitive (or they only are in retrospect).

The resulting groups are also iterators which are exhaustible. It's good if you're running group by on a huge dataset to save some memory, but for everyday operations it's another trap to fall into.

Toolz seems to have it.

https://toolz.readthedocs.io/en/latest/streaming-analytics.h...

I think you've kind of proven the op's point, these all have different patterns and are in different places.

Q.E.D.

By collate, OP might be referring to the behaviour provided by the builtin `zip`.

> Sibling comment mentioned flatten, which is just [item for sublist for sublist in lst]

I never get this one right first time, but surely that's not it.

It's a package, not a built in, but pytoolz [0] is a very complete solution to the type of functional programming munging you are taking about. I wish more people knew about it (no one here seems to have mentioned it). And it has been Cython optimized in another package called cytoolz [1]. The author explains in the Heritage section how it is an extension of and plays nice with itertools and functools [2].

For Python coders new to functional programming, and how it can make working with data easier, I highly recommend reading the following sections of the pytoolz docs: Composability, Function Purity, Laziness, and Control Flow [0].

[0] https://toolz.readthedocs.io/en/latest/

[1] https://github.com/pytoolz/cytoolz

[2] https://toolz.readthedocs.io/en/latest/heritage.html

Yes - the description of writing Python as like assembling Ikea furniture is absolutely spot-on. Yes, you can do it, and the results can be nice, but by God it is sometimes such a pain.

The comment showing where groipBy, sortBy etc can be found just shows the problem - they are all in different libraries.That's just plain annoying! And don't get me started on the pain of trying to build an Ordered Dictionary with a default initial value!

Like defaultdict? All dicts have been ordered since 3.6.

Small note: they've been sorted as an implementation detail of cpython since 3.6, but as language spec since 3.7.

> And don't get me started on the pain of trying to build an Ordered Dictionary with a default initial value!

  >>> from collections import defaultdict
  >>> from datetime import datetime
  >>>
  >>> d = defaultdict(datetime.now)
  >>> d[1], d[2], d[3], d[4], d[0]
  (datetime.datetime(2021, 7, 9, 15, 50, 52, 87605), datetime.datetime(2021, 7, 9, 15, 50, 52, 87613), datetime.datetime(2021, 7, 9, 15, 50, 52, 87614), datetime.datetime(2021, 7, 9, 15, 50, 52, 87615), datetime.datetime(2021, 7, 9, 15, 50, 52, 87616))
  >>> for k,v in d.items():
  ...   print(k,v)
  ...
  1 2021-07-09 15:50:52.087605
  2 2021-07-09 15:50:52.087613
  3 2021-07-09 15:50:52.087614
  4 2021-07-09 15:50:52.087615
  0 2021-07-09 15:50:52.087616
  >>>

Seems pretty good to me?

> ...like groupBy(fn), sortBy(fn), countBy(fn), collate...

What leaps out at me is that these 3 functions are all straight out of relational algebra style worldview.

Python the language doesn't support relational algebra as a first class concept. The reason it feels like IKEA self assembly is probably because you are implicitly implementing a data model that isn't how Python thinks about collections.

Exactly. Pythonic versions:

    groupBy(fn): {fn(x): x for x in foo}
    sortBy(fn): sorted(foo, key=fn)
    countBy(fn): Counter(fn(x) for x in foo)
    flatten: [x for y in foo for x in y]
    filter(fn): [x for x in foo if fn(x)]

All the "for x in blah" can seem like a lot of boilerplate for a non-Pythonista, but actually it becomes subconscious once you're used to it and actually helps you feel the structure (a bit like indentation isn't necessary in C but it still helps you to see it).

For compound operations (e.g. merge two lists, filter and flatten), I find the code to be a lot more easier to "feel" than if you'd combined several functional style functions, where you have to read the function names instead of just seeing the structure.

Speaking of the list and set comprehensions, they also mimic formal mathematical notation e.g. {f(x) | x>0, x in reals} or whatever

That groupBy doesn’t look like it does what you want to do.

For example if

“def fn(i): return i//2”

You probably want to get an output like {0: [0,1]} but you’ll only get a scalar for the value (1) and it will be the last one evaluated.

It would be nice if there was a way to do this on one line but you’ll have to do

ret=collections.defaultdict(list) For i In foo: ret[fn(i)].append(i) Return ret

I suppose it isn’t that bad as it will be in the groupBy method.

Or you could add it to a method on a list. I think that’s easier to do with Rust.

Hmm, you're right... that one is actually quite hard to do in a Python-ish way. There is a one liner, just for the record, but it's absolutely horrific code because it redundantly re-evaluates the same conditions over and over:

    g = {fn(x): [y for y in foo if f(y)==f(x)] for x in foo}

As you said, you'd have to use a for loop. I think sometimes people are unnecessarily afraid of for loops in Python (not everything has to fit on one line!) but this is a case it's a pity to take up so many lines. Or, back to square one, probably this is a case it's best just to use itertools groupby (after a sort) after all.

> All the "for x in blah" can seem like a lot of boilerplate for a non-Pythonista

It's not the boilerplate that's the problem, it's that using the same construct for everything doesn't convey its intention.

If I see a chain of filter, map, group etc., I instantly know what each part does. I know that filter does a filtering, and then I can look at the function passed to it do know how exactly it's filtering. If I see a list comprehension, I have to first understand if it's filtering or mapping or whatever (or even all at once), before I can grok it.

    filter(fn): [x for x in foo if fn(x)]

Python has filter(predicate, iterable) -> iterable built in.

Absolutely but it's not "Pythonic". Guido said:

> About 12 years ago, Python aquired lambda, reduce(), filter() and map(), courtesy of (I believe) a Lisp hacker who missed them and submitted working patches. But, despite of the PR value, I think these features should be cut from Python 3000.

("Python 3000" was the code name for Python 3.) He eventually stepped back from this hard line view but it remains a fact that it's a bit of a historical accident that it's present at all.

[1] https://www.artima.com/weblogs/viewpost.jsp?thread=98196

The thing is, filter is present in almost all languages now while list comprehensions are in Python, Julia and Haskell. Maybe it's not "pythonic", but Python doesn't exist in a vacuum and is influenced by other languages, as proven with the optional typing and the pattern matching. Maybe it's time to review the definition of pythonic?

One of the harder things for beginners to grasp when learning a second or third programming language is that language is not just syntax. You see it all the time in Python in particular: people writing code like it's C or Java. It's no different to grabbing a French dictionary and transliterating your English sentence to French word by word. It just doesn't work and, in some cases, is completely wrong. Grokking a language means so much more than knowing the key words for doing all the stuff you used to do in your old language.

Following your analogy, you will notice that grammar is similar in many European languages. You can rely on subject-verb-object for instance.

Most popular languages these days have aligned patterns for common collections operations. Python stands out IMO.

I wrote a bit more details in a parent comment (https://news.ycombinator.com/item?id=27782030)

Most popular languages offer similar operations on collections these days. I've been using Java, Scala, JS/TS, purescript, Ruby, C++, Rust in the past, and these days all have a similar offering to iterate over collections using idiomatic functions such as map/filter/sort/etc.

I'm working on some python these days and I find quite unpleasant the 1-line lambdas, and general messy options to implement common collections operations.

It will take Python 10 more years to allow a flatten()...

more_itertools ftw

10 years is very optimistic: flatten has little use without a functional pipeline (something like clojure’s threading macro) as in other contexts it’s a trivial comprehension.

Do you need the recursive form? Because a single flatten is itertools.chain.from_iterable

I have written this in serious code before, and I don't regret it:

    from itertools import chain

    flatten = chain.from_iterable
    def lflatten(x): return list(flatten(x))
    def sflatten(x): return set(flatten(x))

I know it exists, but the universally known name is 80% of the utility of having a standard flatten() function.

I could tolerate a slightly different name, but `itertools.chain.from_iterable` is absurd to me.

I think the motivation there historically was to encourage use of generators and generator expressions — it’s definitely intentional, because things like `apply` and `reduce` used to be builtins but now they’re imports.

If you reach for itertools imports often in an interactive REPL, you might be interested in Pyflyby: https://labs.quansight.org/blog/2021/07/pyflyby-improving-ef...

Also many python std library collection methods modify their inputs.

Some of those are given as examples in the functools / itertools doc pages.

And it's simply amazing that functools doesn't fill this gap.

Reply on news.ycombinator.com