Hacker News Reader

Comment by soVeryTired | Hacker News Reader

Honest question: why would anyone make a list comprehension that's multiple levels deep? Isn't the purpose of comprehensions to provide quick-and-dirty inline for loops, where a full for loop is too verbose?

View on HN

Multiple for loops in list comprehensions (if that's what's meant by "multi-level") are pretty straightforward once you realise the simple rule that translates them into loops: write everything to the right of the expression in the same order with nesting. For example this:

    l = [f(x, y) for x in X for y in Y if x > y]

is the same as this:

    l = []
    for x in X:
        for y in Y:
             if x > y:
                 l.append(f(x,y))

If a list comprehension doesn't fit on one line I try to highlight this to any future reader by using one line per thing that would get nested:

    l = [
        f(x, y) 
        for x in X 
        for y in Y 
        if x > y
    ]

Needless to say, there are still list comprehensions that are complex enough that they ought to be broken out into loops. But being nested isn't enough by itself.

People who come from math backgrounds often becomes infatuated with complicated list comprehensions when they realize that they can be used as set-builder notation. {(x,y) for x in range(10) for y in range(10) if y == 2*x}

That looks exactly like my notes from math courses and I'm overjoyed that it's meaningful in Python.

I use stuff like that. But my rule of thumb is that I try and make it readable by breaking it up into multiple lines and using indentation. If I can't make it readable, then I will consider another way of writing it. This is mainly for self preservation, since I'm likely to be the person who has to read it later.

Comprehensions have some other benefits too, for example, they're much less likely to be accidentally stateful. My general experience is that for complex logic, a comprehension is harder to write initially, but much less likely to have bugs when I get it right.

That said, I usually won't use deeply nested comprehensions, but sometimes it actually is the clearest way to parse something (e.g. extracting a field from nested JSON.)

For me it’s very rare and used where I would write something like concatMap in Haskell.

    citiesByState = {
        'WA': ['Seattle', 'Spokane', 'Olympia'],
        'OR': ['Portland', 'Salem'],
        'CA': ['San Francisco', 'Los Angeles', 'San Diego'],
    }
    cityToState = {city: state
                   for state, cities in citiesByState.items()
                   for city in cities}

I would generally avoid multiple levels in a comprehension but there are some very simple two-level cases like this that I use occasionally. I feel that the code is easy to read, at least.

In Haskell, these direct equivalents aren’t bad at all:

    cityToState = Map.fromList
      [ (city, state)
      | (state, cities) <- Map.toList citiesByState
      , city <- cities
      ]

    cityToState = Map.fromList $ do
      (state, cities) <- Map.toList citiesByState
      city <- cities
      pure (city, state)

For multiple “nested loops”, I generally prefer do-notation over both list comprehensions and combinators such as concatMap, unless the structure is simple enough that the combinator version is much shorter.

And for those following who don’t know Haskell, both of these are basically sugar for concatMap.

(>>=) = flip concatMap

While supported, list comprehensions that are multiple levels deep isn't considered good style. There is an example of this in Google's python style guide: https://github.com/google/styleguide/blob/gh-pages/pyguide.m...

It used to be for performance reasons, the way python runs comprehension is faster than plain loops, so the more you could cram in a single comprehension, the faster it ran.

These days though, you can split it in generator expressions and make it run at the end inside a comprehension, and you get best of both worlds: splitting helps readability, running it all at once in a single comprehension at the end keeps it performing.

It's useful for working with permutations. E.g,

    [f(a, b) for a in some_list for b in some_list]

or even

    [f(a, b) for a in some_list for b in some_list if a is not b]

itertools has functions that make that sort of nesting unnecessary (and some would say make the code clearer), eg:

    import itertools as it
    [f(a, b) for a, b in it.product(some_list, some_list) if a is not b]

Oh, or more directly:

    [f(a, b) for a, b in it.permutations(some_list, r=2)]

And then you could also use itertools.starmap instead of using a comprehension at all, if you wanted.

Reply on news.ycombinator.com