Comment by danso | Hacker News Reader

Comment by danso | Hacker News Reader

Coming from Ruby to Python...I could not believe how damn hard it was to comprehend the beautifully explained and simple code (just 21 lines!) in Peter Norvig's writeup of how to create a spelling corrector [1], purely because of the list comprehensions (and a few other visual differences, such as partial list notation)

For example:

     inserts = [a + c + b for a, b in splits for c in alphabet]

It's no different (conceptually, at least) than doing:

     inserts = []
     for a, b in splits:
        for c in alphabet:
            inserts.append(a + c + b)

Or in Ruby (OK, maybe this isn't perfect Ruby):

     inserts = splits.reduce([]) do |x, (a, b)|
         alphabet.each{|c| x << a + c + b }
         x
     end

But the list comprehension was so difficult for me to read that it might as well have been a brand new concept. But I found I got quickly over it after rewriting all of his comprehensions as for-loops, and then trying to convert them back into comprehensions. It was a good reminder that sometimes the barrier is just plain muscle/memory reflex (and probably much more so for beginning programmers...)

[1] http://norvig.com/spell-correct.html

I most frequently use List Comprehensions in a nested manor, whenever I do that I /ignore/ the normal python syntax and do this for readability.

    inserts = [ a + c + b
                for a, b in splits
                for c in alphabet ]

It has the effect of showing the nested loops stacked like they would be in code, but without intending.

> I /ignore/ the normal python syntax

Is it not normal to do that? I always lay out the nested ones, and even single comprehensions if the conditional is complex.

Yes, that's very common formatting (except for the whitespace immediately inside the square brackets, which violates PEP 8, but that's incidental to the question of line breaks).

For the projects I manage, I would have the dev rewrite that as a proper for loop.

That's way too confusing and hard to parse. List comprehensions are great for simple operations, but once you start nesting, you're just being a jerk to the next dev that has to read your code.

Code should be clear first, elegant second.

> Code should be clear first, elegant second.

I disagree: code should be correct before it is clear. And because it's so easy to mess up a for loop (for me at least), I choose list comprehensions where reasonable.

> you're just being a jerk to the next dev that has to read your code

That's not very charitable to either party. You're assuming that the motive of the author is to be a jerk, and you're assuming that the reader won't understand. If I had a dev on my team who couldn't read a list comprehension, I'd (a) wonder how they were hired, and then (b) teach them.

I've written a fair share of Haskell, and I still get tripped up by list comprehensions in python - even my own. It's not that it's impossible to understand, and when there's only one predicate, I think it's fine:

  for row in [[i*j for i in range(1, 8)] for j in some_list if j % 2 == 0]:
    some_op(row)

vs

  for j in some_list:
    if j % 2 == 0:
      row = [ i * j for i in range(1,8) ]
      some_op(row)

I would compare it to sentences and paragraphs. The former feels like a run-on sentence, while the latter is more obvious, cause it has one predicate per line. Also, list comprehensions are a bit like yoda-speak - it introduces the verb before the subject. You have to untangle the order of operations, rather than having the order read top-down and left-right.

Of course, I'm a rubyist, so I'd prefer:

  some_list.select { |j|   j % 2 == 0 }
           .map    { |j|   (1..8).map { |i| i * j } }
           .map    { |row| some_op(row) }

Though definitely need to do something about that second line.

Haskell list comprehensions are a bit easier to parse because they have symbolic delimiters, the fact that Haskell is naturally more terse, and because you can always check the type of the list

  [ [i*j | i <-[1..8]] | j <- [1..4], j % 2 == 0 ]

Any list comprehension that is

1. Longer than ~80-120 characters

2. Can't be explained in 1 comment line

Should be written as a nested for loop IMO.

Yep. But even more: as any piece of code may have to grow and become more complex, any list comprehension except in throwable scripts should be rewritten as a for loop.

Of I think 95% of the list comprehensions I wrote have been refactored to for loop by myself while debugging or extension the code later. So now I just never use them anymore, except for throwaway code.

I don't get this. The list comprehension is much clearer for me. Do you have a mathematical background ? If you can easily think about sets I would expect the list comprehension to be far clearer.

List comprehensions are much easier to read just because they're so much shorter : there's less to remember while reading it.

This is an issue one always hits when you try to implement stuff that's more complex. List comprehensions help because they let you talk directly about more complex sets. They raise the level of abstraction. It's hard to get comfortable with but it really helps.

Yes, I happend to have a strong mathematical background.

the difference in our points of view might be that I work on a code base with some ten million lines of code. Anything in there that is not purely flat stupid python, anything clever in fact, will be an annoyance some day. For example, someone started to use a "clever" `template.render(locals())` 5 years ago, and now we have no way to tell where the variables come from.

Inside a list comprehension, how do you print? branch? comment? emit logs? (I know you "can" do that, but then it becomes a mess not worth it). Moreover, if in you 5 line long-list comprehension, if there is an error, will the traceback tell you where it is?

I understand that list comprehensions are only for simple list construction. So you don't print or branch. Comment is pretty simple though.

My perspective is that in large codebases I have far more trouble remembering function names and what they do than I ever have trouble decoding statements. So the increased code size, long functions and ... that comes from those long list construction statements and loops always seems to take more out of me than difficult list or dictionary comprehensions. Also there are very much fewer places for bugs to hide.

There's also the additional argument that list comprehensions are more efficient because they don't actually construct the list. They generate iterators. It is not a huge difference like in Haskell but it definitely helps.

And there are special cases where you have to use list comprehensions : infinite lists can work as a comprehension, yet the equivalent code to generate them is pages long.

Huh. You seem to think nobody could possibly find comprehensions as easy to read as loops. That surprises me because I think the reason they got implemented in many programming languages is precisely that many people find them easier to read!

I think he's talking about more complex list comprehensions then just the traditional for-loop + map setup. I haven't been in Python land for awhile but I remember some atrocious contraptions of 80+ line comprehensions that could of been infinitely more explicit as a literal for loop.

This is Python after all.

Well, rewriting a 80+ line comprehension as an 80+ line series of nested loops and conditionals wouldn't increase readability in my opinion. What would make it easier to read is to divide the computation into several smaller steps, which could be done either with several comprehensions or several loops.

I'd do the ruby version as:

    splits.flat_map do |(a, b)|
      alphabet.map{|c| a + b + c}
    end

Which feels pretty close to the list comprehension to my eyes.

Depends on a programmer I guess. I'm pretty confused by the Ruby version and I can do Ruby in anger. I'd also use maps rather than each if I was writing it.

Reply on news.ycombinator.com