Refactoring with Loops and Collection Pipelines | Hacker News Reader

Refactoring with Loops and Collection Pipelines

martinfowler.com

23 pointsingve18 comments

Did Martin really need to invent the important-sounding term "collection pipeline" for this?

well, it takes what it takes to sell consultancy.You have to be a bit "creative", make the old stuff new, marketing is at the center of any consultancy /s

I mean that's how things work, you don't have to invent anything, just make up some new expressions.

> I mean that's how things work, you don't have to invent anything, just make up some new expressions.

Let's be somewhat more charitable, and say "you don't have to invent anything, just provide an insight". Which is pretty reasonably, I think.

agreed.

Agreed. The real title should be "Refactor loops with map, filter, and reduce"

There are languages that have map, filter and reduce as built-ins but aren't very accommodating of -- or discourage 'fluid interfaces' that allow the piping style in his examples.

E.g. in python, you can't really 'pipe' to compose its filter and map built-ins, you have to do:

      map(lambda x: x * 2, 
          filter(lambda x: x % 2 != 0, [1, 2, 3]))

(Or use list comprehensions which also are not pipe-like).

He's also specifically talking about 'collections' as in 'lists of dicts' in Python parlance, or 'arrays of objects' in other languages.

The idea that you pass this 'collection' through a 'pipe' of operations is pretty powerful and obviously maps naturally to query-like behaviour. So to give Python back some dues, the place where you do of course see some 'collection piping' is when using the Django ORM.

So yeah it may sound a bit grandiose but I think he's highlighting those terms on purpose because it is about more than just map, filter and reduce.

Thanks for this; I am starting Python coming from C# and I kept looking for LINQ equivalents. Apparently I can stop looking, it was driving me nuts :)

How about this one?

mcoll - Modern API for python collections: https://github.com/nvbn/mcoll

That looks pretty neat, thanks!

I'd only previously come across underscore.py [0] (python port of underscore.js) which looked a little unmaintained though perhaps because it's 'done'.

[0] https://github.com/serkanyersen/underscore.py

It's not using pipes, but you can refactor the example with this Python list comprehension.

    def filter_authors(authors, company):
        return [author['twitter_handle'] for author in authors if author['company'] == company and author['twitter_handle']]

EDIT: HN code formatting seems to be broken so you may need to scroll right to see the entire comprehension.

You can do other SQL-like operations with list comprehensions too.

http://code.activestate.com/recipes/159974-sql-like-set-oper...

The term is older than this article.

http://martinfowler.com/articles/collection-pipeline/

I also don't think he invented the term. A quick google search shows it's connected with LINQ.

I don't wish to be argumentative, but I googled the term before I posted, too, and I don't see many references to it that predate Fowler's first article on the subject, or, indeed, many articles that don't actually credit Fowler for the term.

Maybe he didn't invent the term, I don't know. It just feels to me like it's two words which, when used in combination, don't really add a whole lot of useful meaning to something that many of us will have been using for quite some years.

I concur. I have noticed that people that initially come into the industry as "object bigots" (Fowler's phrase to describe himself, not mine) end up discovering patterns inherent in functional programming or other paradigms and declaring them as novel ways to do stuff in the object world or otherwise. Its probably not optimal to keep adding terms to what is essentially one single concept.

This is what design patterns are all about. You take some common practice, give it a name (or pick a name from commonly used names), and write documentation clearly describing it. Then we can talk about about more easily.

Maybe you don't like the name, but there's nothing unusual about this.

Fowler has it backwards, list comprehension is a subset of LINQ syntax which is full monadic comprehension (among other things)

I'm all for threading and pipelining when appropriate, but I don't find these any easier to read. The nested example in particular is harder to follow.

This is in sharp contrast to Python list comprehensions, which are very straight forward and understandable.

What about the performance implications, i.e. depending on language, you are iterating the list multiple times, which can be slower sometimes than just one iteration through.

Is there some advantage to not having the "handle" null check the first "Where"?

Reply on news.ycombinator.com