Going beyond the idiomatic Python
hackernoon.com
hackernoon.com
My first thought on reading that was to think of Darmok and Jalad at Tanagra. Then I realized that was allegory, not idiom. Then I realized I had no idea what the actual definition of "idiom" is, and looked it up: "a group of words established by usage as having a meaning not deducible from those of the individual words (e.g., rain cats and dogs, see the light)."
...and now I'm confused because nearly everything we call an idiom in programming isn't. What would be the correct term for whatever it is we are currently calling programming idioms?
For example, one from the idiomatic Python book that the article criticizes:
def contains_zero(iterable):
# 0 is "Falsy," so this works
return not all(iterable)
It seems that most of what we call idioms in programming are just using more advanced or more obscure language features or library functions to do something in a more compact or efficient way at the cost of immediate clarity to people who are not as advanced in the language. We've decided what these do is needed often enough that we agree that these are the way people should do these things, even if they don't understand what is actually going on.
I'm not sure programming languages can have actual idioms, except perhaps in the case of language bugs or bad/missing documentation.
The combination of language plus hardware could have idioms, like nested loops iterating a multidimensional array should have the outer loop on the first dimension, the second loop on the second dimension, and so on to minimize jumping around in memory and so hopefully interact better with the cache.
> "a group of words established by usage as having a meaning not deducible from those of the individual words (e.g., rain cats and dogs, see the light)."
Perhaps this is actually a really good illustration of 'idiom' meaning the same thing (roughly) in programming as it does in day-to-day use.
I'm not a Python dev, but I'm willing to bet that a Python dev knows the all() function, in the same way that a Javascript dev knows about something like .bind(this), or querySelectorAll, or a CSS 'dev', the horror, "margin: 0 auto;".
If that's not the case I might be wrong.
An iterable containing `[]`, `None`, or any other value that is false in a boolean context would return true. They've assumed that `(0 -> false) -> (false -> 0)`, which is simply not true.
Even if you're dealing with a context that cannot pass in an iterable with non-zero "falsy" values then it needs to be explicitly documented. At that point, why not "document" it by being explicit with the code. This is how I'd do it, but I feel like it could be made cleaner:
def contains_zero(iterable):
return any([i is 0 for i in iterable])
There's no such thing as "very Pythonic." It's a binary and the worst aspect of Pythonic is that people waste so much energy trying not to be unPythonic in order to meet the mores of the community.
[1]: Edit, Ok, I was confusing two experiences. It was reduce's removal that broke my code using the map->filter->reduce idiom. Map broke by returning a Map object rather than a list (but that was between Python and Python3). And so far as I know, filter has not been changed so as to break code.
Granted, If it weren't for Pycharm's command+click to open up where an object was defined I don't think I would like them quite as much. But being able to quickly walk the inheritance path, and finding simple clear code in small chunks is very nice.
"You may use the “harmful” examples as templates to search for in your own code. When you find code like that, replace it with the idiomatic version."
This is a really dumb idea, especially when paired with the incorrect implementations that the book author encourages and the blog poster critiques.
Incorrect code is never idiomatic, full stop.
This is why Python is so successful, and yet you've never heard anything about "Python fatigue." Being "Pythonic" (which as a proxy I will define as not needlessly flouting the principles that `python -mthis` prints out) is valued by most open source projects and most heavy Python users.
With respect to your complaints, they are not "arbitrary biases," they are all the same conscious decision on the part of Guido to favor the use of comprehension syntax over higher-order functions. See https://www.artima.com/weblogs/viewpost.jsp?thread=98196
You might disagree with that decision and prefer another language, but it is not arbitrary at all.
I suppose a true believer could look at double snake case as a convention to denote private methods in lieu of implementing modular code with actual private methods and see something beautiful and simple and explicit and uncomplicated and a good practice. There's nothing wrong with Python except the bullshit that surrounds it.
Imagine there wasn't such a unifying straw man concept or platonic ideal of being good Python. Would there still be single NumPy and SciPy implementations? Or would the work be scattered across smaller, less well-resourced efforts whose contributors disagreed on ultimately trivial things like whether the API makes use of higher-order functions, metaprogramming, or having many ways to do something?
Yes, that's exactly my point, there are many of them, none of which really have the same sort of universal buy-in, because people are constantly disagreeing about .forEach vs. for loops, or promises vs. callbacks, or whatever.
Boost has very strict guidelines about its approach to design and implementation[1] -- it even quotes the Zen of Python.
I didn't say that the concept of "Pythonic" is necessary to implement high-performing and widely-used libraries. I am merely contradicting your unsubstantiated assertion that the concept is harmful.
This never happened-- perhaps you're thinking of reduce? And reduce is still in it, it just got moved to `functools`.
GVR doesn't like functional-style programming, so he much prefers the iterative but compact comprehension alternatives.
It may just be the circles I hang out in, but most folks aren't map/filter versus comprehension purists. I'll do comprehensions usually, but map/filter when it helps make the line shorter or more readable.
My issue with 'Pythonic' is that it is used to express the idea that someone's thoughts are or are not properly structured.
[1]: https://mitpress.mit.edu/sicp/full-text/sicp/book/node35.htm...
It's a magnet for bugs (e.g. stock items with a price of zero being treated as items with no price at all).
E.g.:
`contains_zero` violates "There should be one-- and preferably only one --obvious way to do it."
`should_raise_shields` violate "Explicit is better than implicit."
`linsolve` and `contains_duplicate` violate "Readability counts."
def inverse_of(matrix):
return adjugate_of(matrix) / determinant_of(matrix)
Additionally, it seems like the word idiom is just re-skinned as a "rule" in this article.
The point of the article is just the classic 'foolish consistency is the hobgoblin...'
In a language without explicit typing and return values having an instant indicator of the functionality of a function can be a huge help in writing readable code.
[1] http://ancienthebrewpoetry.typepad.com/ancient_hebrew_poetry...
But the upsides... It's faster, it's different in a good way, in that it makes you think differently (OTP/processes) without the risk of being unproductive. It's very ruby-like but without the warts (although I'm sure it has a few of its own), and was even specifically designed with Ruby as an inspiration/starting point (AFAIK).
So far I've not come across articles that were negative about Elixir in relation to Ruby. So if you do, I'd be curious to hear what you think. And of course if anyone can point out things they feel are better in Ruby, I'd love to hear it.
It's just conventional.
The "!" is actually part of the function/method's name. When you want to sort an array, for example, you can either say "sorted_array = array.sort" or you can sort in-place with "array.sort!". So the exclamation distinguishes between the two methods and warns you are performing a destructive operation.
{being unPythonic} If it makes you happy to use '_p', do it. Explain it in the documentation if it might cause confusion. Be flexible enough that you'll change before letting yourself be voted off the island because it's not that important.
{rabble rousing} I think there is room in the world for an alt-python movement...https://github.com/altpython
I suppose it is more Pythonic to name predicates like "is_full" or something rather than "fullp". But whatever it is I've found I can't stand not using some kind of naming convention for predicates.
I think that consistently using "!" to mean "mutates in-place" would be a good convention for a language to use, but that's not what Ruby does. From Matz himself [1]:
The bang (!) does not mean "destructive" nor lack of it mean non destructive either. The bang sign means "the bang version is more dangerous than its non bang counterpart; handle with care". Since Ruby has a lot of "destructive" methods, if bang signs follow your opinion, every Ruby program would be full of bangs, thus ugly.
def contains_duplicate(array):
return sum([sum([1 if a_i-rot_ij else 0
for a_i, rot_ij in zip(array, rot_i)])
for rot_i in [array[i:] + array[:i]
for i in range(1, len(array))]]
) != len(array) * (len(array) - 1)
def contains_duplicate(array):
for i in range(len(array)):
for j in range(i):
if array[i] == array[j]:
return True
return False
def contains_duplicate(array)
return len(set(array)) < len(array)
def contains_duplicate(array):
any(len([y for y in items if x == y]) > 1 for x in items)
So I think the examples are contrived and the 'rule' is bullshit.
Interesting. I'm not sure if I agree or disagree.
I think the primary benefit of a comprehension is that it can be considered as a "unit", and you don't have to think about the iteration as fully to understand what's happening. As soon as "what's happening" gets more than minimally complex, that advantage is lost. Generator comprehensions should be much more clear and concise than their equivalent implementations using generator functions, though.
If I remember this evening, I'm going to go through my code library and see if I can find something that will satisfy your challenge :)
List comprehensions are less expressive than for loops hence there is less to reason about.
Hence, if what you want is achievable with a list comprehension, its intent will naturally be clearer than the for loop equivalent.
...provided you don't make it intentionally obtuse like the OP did.
def contains_duplicate(array):
return any(len([y for y in array if x == y]) > 1 for x in array)
The most pythonic version (your set solution) has a bug in that it does not apply to the general case. The second-most pythonic version (your one liner double for-loop slam dunk) is as clever as we can make it, and thereby fails Kernighan's debugging test[1].
I feel that one for loop inside another is fine. Everyone in every language can see it and know instantly what the run-time characteristics likely are, and what at each point it is doing. What's most important: if it ever seems to be broken, I can drop a debugger or a logger (maybe with a condition!) in at any point, with ease. There is a reason that pattern is nearly universally known, and imo it's that it is a good one.
1 - "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?", https://en.m.wikiquote.org/wiki/Brian_Kernighan
It's both shorter and less expressive than the equivalent for loop. If you find it harder to read that says more about your familiarity with list and generator comprehensions.
>The most pythonic version (your set solution) has a bug
Not in lieu of a hard requirement for counting non-hashable items, it isn't.
It's faster as well as simpler: O(n) instead of O(n2). Unless you're counting lists of lists or something it makes no sense not to use it.
>I feel that one for loop inside another is fine. Everyone in every language can see it and know instantly what the run-time characteristics likely are, and what at each point it is doing. What's most important: if it ever seems to be broken, I can drop a debugger or a logger (maybe with a condition!) in at any point, with ease
Writing your code in a more verbose and complex way in order to satisfy your desire to put a debugger in is by far and away not the most important aspect of clean coding.
[1, False, 3]
[1, '', 3]
[1, None, 3]
[1, [], 3]
The idiomatic implementation of `contains_zero` is the second one given, `0 in container`.
Just because someone claims something is idiomatic in a book doesn't mean they are correct about it being idiomatic.
> beware of writing if x when you really mean if x is not None -- e.g. when testing whether a variable or argument that defaults to None was set to some other value. The other value might have a type (such as a container) that could be false in a boolean context!
This should apply to the contains_zero function as well. It seems a little murky when PEP8 recommends checking whether a sequence type is empty,
Yes: if not seq:
if seq:
No: if len(seq):
if not len(seq):
LET(('a', 1,
'b', 2),
lambda o: [o.a, o.b])
lambda a=1, b=2: [a, b]
LET(('a', 1,
'b', lambda o: o.a + 1),
lambda o: o.b)
(lambda a=1: lambda b=a+1: b)()How would you write this, for example?
LET(('a', 2,
'b', lambda o: o.a + 1),
'c', lambda o: [i * i for i in range(o.b)]),
lambda o: o.c)
=> [0 1 4]1. List comprehensions are hard to read and understand, especially for beginners. I tutor online in Python, and when I show a student a list comprehension, it is one of the most difficult parts of the language to explain. Do you read left to right, or inside to outside? Postfix if at the end makes things even more confusing.
There are better alternatives in languages like Ruby and JavaScript. Which brings us to the next point:
2. No native map, filter, reduce, etc. methods. It is far easier to explain that you are just calling another method on a list/array than it is to get a beginner to trust the hocus-pocus apparent in Python's list comprehensions.
Edit: Python has built-in functions for map and filter. See correction by ubercore below, and my response.
3. The way a class is accessed from an instance in Python is not nearly as elegant as Ruby's. Do I use "type(obj)" or make a direct call with "obj.__class__"? Neither is as clean or intuitive as "obj.class".
4. Speaking of classes, why are built-in classes like "str", "int", "float", etc. lowercase while in everything else they are conventionally CapitalizedClasses?
5. Remembering that "sorted(my-list)" returns a new sorted list while "my-list.sort()" does the sort in-place requires experience, which someone taking an Intro to Programming course will probably not accumulate over 8 or even 14 weeks.
Note: I flirted with Ruby and Rails, but never built anything real with them. I vastly prefer Python to Ruby (and Django to Rails), but think Ruby got a lot of things right.
But having to provide a lambda for the simple case makes them less useful. Also, nested function calls are less readable than chaining method calls on the same object.
Of course, I'm not saying that a 'multi-paradigm' language for teaching is bad, but personally I'm inclined to think that it might be better to start with something more pure.
Ruby might actually not be a bad choice in this scenario. If you avoid the 'blocks, lamdbas and procs', it seems to me like a great introduction to OO programming. Despite this, for my day to day programming I would also choose Python over Ruby. But perhaps not for teaching.
2. They're all still there, but they're not as good as comprehension syntax.
3. Use type(self) unless you know why you'd want to use __class__ (Python 2 old-style classes).
4. That's legacy; probably if things were done from scratch today, they'd be Int, Str, Float. Back when the names were chosen there was a stricter line between builtins and unset-defined classes, and there was more consideration given to people migrating from C and C++.
5. Yes, luckily the .sort() method gives you at least some hint by not returning anything. It's a good application of command-query separation, but I forget it often myself.
I have been very guilty of this one, it's a running joke at work gatherings. In the flow I can quickly spit out comprehensions that are obtuse to everyone else. My general rule of thumb is that if I ever have to troubleshoot it I break it out into a for loop
(Or the well intentioned but ultimately misguided attempt to enlighten the masses.)
Pretty is better than boring.
Comprehensions are better than loops.
Short is better than long.
Clever is better than tangled.
Nested is better with line breaks.
Compact is better than meandering.
Appearances matter.
The point of a Python program is to teach people to write better Python programs.
Though if they're Java programmers just try to make them jealous.
Exceptions should never be explicitly checked.
Except StopIteration.
If you don't know how to make it Pythonic, refuse to take the obvious approach.
There should be one-- and preferably only one --Pythonic way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
But it's fine to take ramp it up over time as long as you get there.
If it doesn't make a good blog post, perhaps it doesn't make good code.
If it doesn't make good code, it definitely doesn't make a good blog post.
List comprehensions are one honking great idea -- let's do more of those!
To learn to write good code you have to write a shit-metric-ton of bad code.
def are_all_numbers(everything):
return all(is_float(e) for e in everything)