Functional Programming in Python [pdf] | Hacker News Reader

Functional Programming in Python [pdf]

oreilly.com

153 pointsmseri59 comments

Even knowing a lot about FP, I still found this worth skimming for the esoteric Python syntax. When it comes to constructing dictionaries with list comprehensions, I would always do something like this:

  dict([n, 2 ** n] for n in range(5))

But they pointed out an actual "dict comprehension" that I didn't even realize existed:

  { n: n ** 2 for n in range(5) }

And there is a similar "set comprehension":

  { n ** 2 for n in range(5) }

Always amazes me how you can use Python for so many years and still encounter new features in the language.

This is part of the generalized comprehension syntax. You can also do lazy generator comprehensions.

    a = (i for i in range(10) if i % 2 == 0)
    print(list(a))

You can omit the parenthesis, and use them in calls which expect an iterable.

    b = max(i for i in range(0, 10) if i % 2 == 0)
    print(b)

It's not exactly the same, which is what I thought was interesting. My first example did use a generator expression inside the dict() constructor, but in that case you need to specify the key and value in a tuple or a list.

With the dictionary comprehension you can just separate the key and value with a colon, which is more natural. It might just be sugar on top of a generator expression but it is definitely a special case, syntactically speaking.

Yes, in Python 2.7 onward:

  s = {i**2 for i in range(10) if i}

and the colon is what makes the dict comprehension different from the set comprehension:

  d - {i:i**2 for i in range(10) if i}

Although Raymond Hettinger has also called them generator "comprehensions" in the early proposals, the current documentation calls them "generator expressions". And good examples.

Here's how you make those set/dict whatever comprehensions in Python 2.6, before the native syntax is used:

  s = set(i for i in xrange(10) if not i%2)

Seems like a comprehension is just a particular form of Python expression.

Here's the grammar file: https://docs.python.org/2/reference/grammar.html

Look for dictorsetmaker (for {}) testlist_comp (for generator expressions) and listmaker (for [], i.e. list comprehensions).

it was added in python 2.7 / 3.0

The #1 and #2 expressions yields different results.

>>> dict([n, 2 n] for n in range(5)) {0: 1, 1: 2, 2: 4, 3: 8, 4: 16} >>> { n: n 2 for n in range(5) } {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Nice, short little book!

`compose` can be simpler:

    def compose(fn, *fns):
        def _composer(f, g):
            return lambda *args: f(g(*args))

        return reduce(_composer, fns, fn)

This little function is really, really cool because it allows you to build up more interesting functions by piecing together a bunch of small, useful ones.

    def upper(s):
        return s.upper()

    def exclaim(s):
        return s + '!'

    # instead of this
    really_angry = lambda s: exclaim(exclaim(upper(s)))
    really_angry('napster bad') # NAPSTER BAD!!

    # we can do this
    really_angry = compose(upper, exclaim, exclaim)
    really_angry('fire good') # FIRE GOOD!!

    # and
    import operator as op
    from functools import partial as p
    max(map(compose(p(op.add, 1), p(op.mul, 3)), (1, 2, 3, 4)))

`compose` is a neat function and worth exploring. This is a cool book and I always hope Python gets more light shone on its FP-friendly features.

I started with something like that when I was missing function composition in Python. Eventually I ended up with a library [1] including a bunch of other stuff for getting rid of some of the duct tape code you usually need when you just want to compose some functions.

  from pipetools import pipe, X, foreach

  really_angry = pipe | upper | exclaim | exclaim

or...

  really_angry = X.upper() | "{0}!" | "{0}!"


  (1, 2, 3, 4) > foreach((X + 1) | (X * 3)) | max

You can write some pretty neat looking concise code with this, but also may regret it later when it comes to debugging, especially when lazy evaluation is involved (which is usually the case). The stacktraces tend to be not so helpful...

[1] https://0101.github.io/pipetools/

For compose to really shine, you need to be able to curry/partially apply functions. This part of things is made much more difficult than necessary because of Python's unnecessarily neutered lambda syntax (in fact, I don't think one can claim that Python is FP friendly until this decision is corrected).

It's also worth noting that reduce() was removed as a builtin for Python 3.

I'd love let-like syntax in lambdas, something along the lines of

    lambda x: f(y) + g(y) for y = expensive_computation(x)

In Python 3, `reduce` can be trivially imported from `functools`.

The ugly hack around this would be

    lambda x: (lambda y: f(y) + g(y))(expensive_computation(x))

It's possible to hack currying into python with a decorator like:

https://gist.github.com/grantslatton/9221084

(I made this for fun, use at your own risk)

Funny how things turned out. I still remember this post[1], it was profoundly disappointing to see Guido's way of thinking. Much of the damage was reversed but it still left an indelible impression that there's a lack of vision for what's going to be important if the language is to stay relevant in the future.

[1] http://www.artima.com/weblogs/viewpost.jsp?thread=98196

There should be one -- and preferably only one -- obvious way to do it. [1]

You may not agree with it, but it's a vision.

[1] https://www.python.org/dev/peps/pep-0020/

Removing features does not mean there is a lack of vision.

The purpose of Python is not the same as, say, Haskell.

Design decisions should therefore be viewed as such - it's not about a steady increase of functional features and a decrease of procedural features.

I actually find his arguments rather convincing, even though I'm partial to a functional approach to programming, myself.

Some are still pretty bullshit though.

> Why drop lambda? Most Python users are unfamiliar with Lisp or Scheme, so the name is confusing

To which I'd reply: Why drop class? Most Python users are unfamiliar with C++ or Java, so the name is confusing.

Are they? I'd probably argue just the opposite, that most python developers are very familiar with C++ or (most of them) Java.

I remember seeing that post and having it be the first time I thought "huh, does van Rossum know what he's doing" but it certainly wasn't the last.

https://en.wikipedia.org/wiki/Python_Bridge

"Python Bridge, officially known as High Bridge, is a bridge that spans the canal between Sporenburg and Borneo Island in Eastern Docklands, Amsterdam. It was built in 2001 and won the International Footbridge Award in 2002. The bright red bridge spans 90 meters and was designed by Adriaan Geuze of the architectural firm West 8"

Coincidentally, Amsterdam can be considered the birthplace of Python, where Guido used to work at the Center for Mathematics and Computer Science (CWI).

And now for some obligatory functional python. Run with

    python lambda.py 2>&1 | head -c 200

to avoid filling your screen with exhausted recursion depth. Notice any pattern in the output?

    import sys
    def c(j,t):
     sys.stdout.write(j('.')('P'))
     return t
    (lambda z:lambda y:z(z(y(lambda p:lambda n:(lambda s:lambda z:z(lambda x:
    lambda y:y)(lambda d:p(s)(y(s))(d)))(lambda x:lambda a:lambda s:lambda p:
    p(a)(lambda y:s(n(x))(y))))(lambda c:lambda a:lambda s:z(lambda y:s(c)(y)
    ))))(y(lambda p:lambda b:lambda t:t(c(b,p)))))(lambda s:lambda p:p(lambda
    x:lambda y:x)(s))(lambda f:(lambda q:q(q))(lambda x:f(lambda y:x(x)(y))))

This reads like Lisp.

Yes it's quite possible to write obfuscated Python. See the docs:

https://docs.python.org/2/faq/programming.html#is-it-possibl...

It's becoming more common to see "Functional Programming in X". Why don't we use functional languages like oCaml or Haskell more often? Are we making the jump in two steps instead of one? I've never written more than a few lines of either so I can't tell if something "better" is waiting for me in functional land.

I'd go one further.

Why are there so very few programs written in say, haskell, that you actually want to use?

This is a serious question that I haven't found the answer for. Around here people suggest pandoc, shellcheck and sometimes the xmonad window manager as pretty much the full list of things you can install, use and hack on written in haskell whose purpose isn't writing haskell code.

Given the popularity of haskell amongst hackers and the various claims about its benefits and strengths, the fact this list is so small (if it's larger and I'm missing a bunch - please DO let me know, I want to play with them and hack on them!) Is something I have difficulty reconciling with haskell being a useful general purpose programming language. Maybe they're being written now and they're on their way? But haskell has been around for more than a few years now. Maybe haskell hackers just mostly hate open source & free software unless it's GHC or a general purpose library? Seems unlikely. So maybe something else I don't yet understand. It is puzzling when I don't code in the language fluently enough to understand its weaknesses as opposed to my own in coding in the language and it's a point the many lovers of haskell never seem to address other than with extreme defensiveness which kind of misses the point of the question.

> Why are there so very few programs written in say, haskell, that you actually want to use?

I think a big part of it was struggling with cabal hell. I know that quite a bit of web development and API stuff is happening with Haskell since I've gotten paid to do some for multiple clients (some requesting Haskell).

I think that with the release of stack[0] (and it eventually being merged into the Haskell platform IIRC) many application developers will start to pick up Haskell and create those types of programs.

Not open source, but an application created in Haskell was bump[1].

0: https://github.com/commercialhaskell/stack 1: https://www.fpcomplete.com/wp-content/uploads/Bump-case-stud...

Probably because its easier to see the use of functional paradigms in languages you're familiar with. OCaml and Haskell are great languages, but they require a lot of new learning at once.

You are right about it being nice to see functional code in your language. However, from my learning experience I understood the functional style much better by using actual functional language. I started with Scheme which has very minimal easy to understand syntax. Many courses ask students to try forget whatever they know about programming before introducing functional style. If that helps (for me it did), I think starting with new language would be nice decision.

I made this comment earlier today about Elixir being more approachable for people coming from another dynamic language:

https://news.ycombinator.com/item?id=9942407

There are many steps between functional programming in, say, Python, and writing everything in Haskell. Many differences. Not everybody has the same opinion of all those differences.

I'm often missing a feature to make language X act in a purely functional way (i.e., to disable side effects completely in a relevant part of the code).

Also missing is a way to select between strict or lazy evaluation.

Yeah. Python has the functional programming features I expect of any modern language. However, I feel that Python has a lot of unneeded syntax. I always prefer apply() over * and map() and filter() over list comprehensions.

    func(*args)
    apply(func, args)

    [func(a) for a in collection]
    map(func, collection)

    [a for a in collection if func(a)]
    filter(func, collection)

I don't see why people use all of this special syntax.

In Python3, you are really looking at...

    (func(a) for a in collection)
    map(func, collection)

as equivalent. If you want a list (and not a generator), you would need to do this:

    [func(a) for a in collection]
    list(map(func, collection))

For me, the first set (comprehensions) of notation has a more mathematical feel to it, i.e. { x^2 | x \in 0...10 }. Just replace the bar with "for" and it's almost the same thing.

I believe the documentation for `filter` even mentions that it is equivalent to the comprehension[1].

[1] https://docs.python.org/3/library/functions.html#filter

I've always found the list comprehension harder to read for non trivial examples. It may just be a matter of which a person learned first.

> It may just be a matter of which a person learned first.

I think this is the case, for me map is much harder to read. But I also think that comprehensions go back to math sets, so I was familiar with this even before learning any programming. Therefore comprehensions clicked immediately for me and it's by far my favourite python feature.

I learned both around the same time, but even in good functional languages like Haskell, using a comprehension for anything more than simple problems results in an unreadable mess.

map or filter are much easier to read for complex data manipulations and as a bonus, their composition rules make it easy to increase performance. For example, if you see two map functions together, you can wrap them in a compose and only map over your elements once. This isn't as immediately obvious when you're using comprehensions.

Could you give an example?

Do you mean something like

> [ manager.name for manager in set([ person.manager for person in employees ])]

I assume with something like set() or unique() you need to create the intermediate iterable anyway, but without it I have trouble finding an example where doing a single list comprehension wouldn't suffice.

Not quite. I learned map and filter much earlier, and still I use more list comprehensions, both in Python and in Erlang.

> I don't see why people use all of this special syntax.

(1) It's easier to read, particularly for people who don't always think in terms of functional programming.

(2) It's faster, particularly if you have to construct a lambda for the function in apply, map, or filter.

Part that bothers me about Python is no special binding form, that is you could just do x = 3 to add x variable to my environment, and second is not-strict lexical scoping.

From other perceptive, these usually bother me when my functions are larger, and ideally functions should be small. So I take it as sign that I should probably break down my function.

And yeah, tail calls! I expect that from a modern language with functional programming features. Unfortunately it seems there are no plans to add them in Python

  [(foo(a), bar(a)) for a in collection if condition(a) or alt(a)]

  foo_bar = lambda x: (foo(x), bar(x))
  condition_or_alt = lambda x: condition(x) or alt(x)
  map(foo_bar, filter(condition_or_alt, collection))

As logic gets more complicated, wouldn't list comprehensions become easier to read straight through?

I don't see why people use all of this special syntax.

Comprehensions are one fairly easy way of thinking about sets of things, and transformations of those sets of things. It may not be your preferred way to think about them, but that doesn't mean it's unneeded or that people are wrong to prefer another way.

If you don't have the function handy and don't want to go thru the effort of making a lambda, list (set, dictionary, generator) comprehensions can be more convenient (and you can comprehend over nested lists too)

It's easy to read, though that is pretty subjective.

Hard to tell from a quick scan, but it appears to be slightly more informative than the classic document in the python docs: https://docs.python.org/dev/howto/functional.html

Just in case, a few years back I've seen an article on functional programming in python. Mostly arithmetic but the patterns were very pretty (think Euclid algorithm generalized). I never managed to find it again. If that rings a bell to someone, I'll be forever virtually indebted.

- http://www.ibm.com/developerworks/library/l-prog/

- http://kachayev.github.io/talks/uapycon2012/#/

- http://anandology.com/python-practice-book/functional-progra...

- http://maryrosecook.com/blog/post/a-practical-introduction-t...

- http://rosettacode.org/wiki/Numerical_integration#Python

Don't think this is the one you're thinking of, but I do like Mary Rose Cook's intro to fp using Python:

http://maryrosecook.com/blog/post/a-practical-introduction-t...

Indeed, this is quite newer, and a gradual introduction to FP idioms (with good successes).

Mine wasn't introductory and just threw out ways to decompose the problem into counter intuitive (think ~monad) blocks.

I'll edit my post to list those I've found so far. Thanks a lot anyway.

Use mochi if you really want to use FP in python.

https://github.com/i2y/mochi

Interesting, thanks for the link. Do you know if it has any multicore (i.e. parallel) support? I looked but all I could see was support for concurrency.

For parallel programming, you need either:

https://docs.python.org/dev/library/multiprocessing.html#mod...

http://www.parallelpython.com/

or use the GPU using many available libraries.

For anyone wanting a quick intro to fp in Python I stumbled into this awesome presentation (50 slides) about Functional Programming in Python. I especially like his short but clear examples: http://kachayev.github.io/talks/uapycon2012/#/ . Good intro before getting deeper into the linked book.

FP Library in Python: https://github.com/kachayev/fn.py

Mostly OT, but I like the short form books oreilly and packt are producing. I never liked the trend in technical books where every single one had to have 6 chapters of language tutorial, etc.

I am curious why lists do not have ".map" and ".filter" methods? IMHO it would be so better for chaining, now using a few maps and filters is inconvenient and looks unreadable. Plus, something shorter for "lambda"...

It's one of not too man aspects, where I prefer JavaScript (especially ES6) to Python.

That would require any new container classes to implement each of those methods again (and will probably still lack useful ones, like groupby) whereas currently they can just implement iteration and get the rest for free.

You could have both approaches, but that would go against one of Python's core principles ("There should be one-- and preferably only one --obvious way to do it.").

Functional programming? In a language with no tail calls and no function expressions?

I mean sure, you can do it, but you'd have an easier time in a language which supports those.

Reply on news.ycombinator.com