Hacker News Reader

Comment by BiteCode_dev | Hacker News Reader

Not only this will speed up comprehensions (which are already faster than map/filter), but it will likely solve this annoying weird edge case with the scoping of lambdas in comprehensions.

Last but not least, since it removes the additional frame that the comprensions used be wrapped in, it will fix the way you interact with them using pdb, which has always been rough thanks to this invisible indirection layer.

I assume JIT will also appreciate the change, so let's see if pypy moves on this.

Migrating to a PEG parser opened so many opportunities for Python. I never expected we would see so many good things come out of it, after years of defending LL1 to keep the reference implementation simple. E.G: the new error messages are awesome.

View on HN

> Not only this will speed up comprehensions (which are already faster than map/filter)

As someone coming from Haskell I find it odd that comprehensions are prioritized over generic maps (fmaps) and traversals in Python. Not only they are more verbose syntax-wise and don't allow for partial application, they also discourage people from discovering generators, as most of the tutorials go with [] and {} brackets instead of the lazy (). At least with map() people could get acquainted with generators earlier.

> I assume JIT will also appreciate the change, so let's see if pypy moves on this.

I wonder if there's a real difference between map and comprehensions performance when it comes to JIT inlining.

Guido hates higher-order functions. Once upon a time there was map, filter and reduce as builtins. Then for Python 3, they tried to remove all three, but failed due to community pushback. reduce() did get banished however.

The argument is that making it hard to use higher-order functions discourages point-free style. "Explicit is better than implicit" is a design principle of the language.

"Explicit is better than implicit" seems vacuous in the context of Higher Order Functions and it's applied functions.

It's pretty explicit. Nothing is implied, it only can work one way.

Well you can see Guido in his own words at https://www.artima.com/weblogs/viewpost.jsp?thread=98196.

The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking. As a result people have created arbitrary terminology for it like "lambda" and "reduce". When you've mastered the terminology it may seem like a good way of doing things. But there are easy alternatives for doing the same things that DON'T require mastering that terminology.

Python has a number of guiding principles, https://en.wikipedia.org/wiki/Zen_of_Python has a list, but one of them is, "There should be one– and preferably only one –obvious way to do it." And higher order functions do not win in Guido's mind.

Separately, while the operation of higher order functions is explicit, code that makes heavy use of them tends to make behavior implicit and dependent on setup elsewhere. I have heard Guido complain about that exact style of coding. I have grudging sympathy for his view. Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing.

> The simple version is that higher order functions do not correspond to anything that is part of our common sense thinking.

and what is common sense thinking here? are you talking about python or just programming in general?

> Functional code can easily be a joy to write but a confusing nightmare to debug. And we spent more time on debugging than writing

Citations needed. I'm not sure who WE is in this context. Python is an imperative lang and doesn't even do OO that well, let alone FP. but, if you are talking about programming in general, I do find it to be much easier to hunt down bugs in languages where the design philosophy of the language favors functional programing ergonomics.

There's even been studies on this. https://cacm.acm.org/magazines/2017/10/221326-a-large-scale-...

tl;dr: "For those with positive coefficients we can expect that the language is associated with a greater number of defect fixes. These languages include C, C++, Objective-C, Php, and Python. The languages Clojure, Haskell, Ruby, and Scala, all have negative coefficients implying that these languages are less likely than average to result in defect fixing commits."

> As a result people have created arbitrary terminology for it like "lambda" and "reduce".

But Python does exactly that (I'm not knocking it, terminology is important, but its all arbitrary)

In Python, we have "for" loops, we have "while" loops for lists and tuples(+ special syntax for "for" loops, when looping over a list of tuples!).

Any sufficient digging into "for loops" and their implementation, and we begin to realize it's all just iterators under the hood completely hidden from the developer, but I digress. We have a special syntax for list comprehensions, (special syntax for nested comprehensions) that extend to ranges. There's special syntax in the form of specific statements that exist only in the context of a loop; "continue", "break", "pass", etc.

There's all the implicit state passing into the bodies of these loops, and there's no referential transparency.

"map", "reduce", and "filter" are specific and have no special constructs, no magic, it does what it says on the box. I'd argue it has a very similar mental overhead to a for loop.

This is my experience, YMMV.

About common sense thinking.

The phrase "for x in y" has an immediate common sense meaning in English. What it means in Python works out to that. Sure, you find a lot of complexity under the hood when you dive into the implementation. But the whole point of programming languages is to allow the programmer to think in terms of human concepts rather than machine implementations. And the concept maps directly to one that beginning programmers already have.

By contrast I've yet to meet any non-programmer who has an intuition about what "map", "reduce", and "lambda" should mean. As great as they are as organizing concepts, they are something that programmers need to learn and gain experience with before it makes sense. For a programmer who has completely internalized the concepts, it may well be similar mental overhead to a for loop. But most programmers have NOT done so. And beginning programmers find for loops a heck of a lot easier to learn - because it corresponds to something that already makes sense to them.

And the implementation is less "completely hidden" than you claim. Heck, the official tutorial at https://docs.python.org/3/tutorial/ includes https://docs.python.org/3/tutorial/classes.html#iterators which shows you exactly how to poke under the hood and see how it works.

As for my comment about spending more time debugging than writing, that should be obvious. As https://www.scirp.org/journal/paperinformation.aspx?paperid=... notes, about 70% of total software costs come in the maintenance phase. Therefore as developers we spend over 2x the time maintaining software rather than developing it. The actual ratio varies by project size, industry, language, tooling, developer experience and so on. But that's a pretty good average. As your link notes, the contribution due to language is real, but modest.

My comment about functional programming being hard to maintain is based on anecdotal experience. I find my own functional code easy to maintain. But others have not. And my experience picking up other people's code has been at best mixed.

Oh finally, about Python, comments like "doesn't do OO very well" usually mean "doesn't do OO like I wish it did". I find them to be highly individual opinions, and not apples to apples when you compare people.

One more thing.

If you want to know why Python looks the way it does, you need to get into Guido's mind. You may disagree with him, but your disagreements will not help you understand why Python looks the way it looks.

And before arguing that he's wrong to think that, remember that for several years now, Python has added more developers in a year than Clojure, Haskell, Ruby, and Scala have *COMBINED*. So while he may be wrong-headed, he's wrong-headed in a way that works for a lot of people.

(Note, I think he's wrong-headed about a lot of things.)

You don't explicitly pass the arguments to the function, is the point. I am not sure I agree with it but it's a thing to consider.

AFAICT Guido has softened his stance on this somewhat. The main impediment to `map()` and friends at this point is the clunky `lambda` syntax. However here we see another benefit of syntactic comprehensions over higher-order functions: comprehensions can be inspected deeply and inlined or otherwise optimized as a special case, while a stack of higher-order functions might need a more advanced bytecode compiler or JIT to apply "deforesting" and inlining.

> don't allow for partial application

Unfortunately, partial application doesn't work in Python, the same way it works in Haskell. If a function has 3 args, calling `f(x)` is a guaranteed runtime `TypeError`. There is functools.partial to hack it into the language for cases where you absolutely need it but overall it's bad developer experience. In fact, when I do need to partially apply `f(x, y, z)` I much prefer doing `g = lambda y, z: f(fixed_x, y, z)` because it's clearer than using functools.partial.

I don't agree that the `lambda` is clearer than `partial`. This is a matter of personal style and preference, and IMO the top priority in cases like this is consistency within the codebase.

You can also refactor partial function applications into callable classes depending on your needs, which is basically what `functools.partial` does internally anyway.

Comprehensions vs. map/filter/etc. is a style thing.

If you think in terms of an expression language that composes functions, then map and filter are more natural.

If you think in terms of a statement language with special syntax for computation expressions, then comprehensions are more natural.

Even in Racket (a scheme), idiomatic code seems to prefer the `for/thing` family of macros, which have a variety of flavors and keyword arguments to support map, filter, reduce, mapcar, etc.

Whatever floats your boat.

So, out of curiosity: what does the PEG parser have to do with this particular PEP? It seems like this was just a somewhat inefficient (but simple) implementation choice from the start that is now being improved.

I don't know that it matters here specifically. But I think it was a kind of watershed moment in terms of how the development approach to CPython has changed.

Are you sure about the lambda scope bit? It seems to me there will still be only one loop variable, which gets re-used throughout the loop (unlike, for example, the 'do' loop in Scheme, where each turn through the loop explicitly gets new locations for all variables).

Reply on news.ycombinator.com