Python: Overlooked core functionalities
erikvandeven.medium.com
erikvandeven.medium.com
bar.baz()\
.filter(some_filter)\
.map(some_op)\
.min()\
.foo()
min(map(some_op, filter(some_filter, bar.baz()))).foo()
Any/all return a Boolean, so the chain would stop there.
I also personally think
any(x % 5 in range(y))
range(y).any(lambda x: x % 5)Even if you go far out of your way to format it similarly, it still forces you to do a lot of mental work to see the inner most starting point and then deduce what the sequence of operations that happens is backwards, eg:
foo(
min(
map(lambda x: ...,
filter(lambda: y: ....,
baz(bar)
)
)
)
But that's more of a math thing than an everyday coding thing, where dot chaining usually reads nicer.
Mixed is definitely the worst, like you said.
incidentally in your example, though data does flow from top to bottom, control does not, assuming the filter and map methods are lazy as they are in python; it ping-pongs back and forth up and down the sequence in a somewhat irregular manner, sometimes reaching as far as .min() before going back up, and other times turning around at .filter(...)
i wonder if you could implement the ide functionality you want with a 'wrap' menu of popular functions that are applicable to the thing to the left of your cursor, so when you had
filter(some_filter, bar.baz())|
min(filter(some_filter, bar.baz()))|
"y: %s" % y|
You mean Ruby? :P
(All Ruby iteratables mixin Enumerable, which is baaaaaaaasically inheritance.)
Lifetimes and async are a massive pain in rust. But the trait system is a work of art.
trait MyIterHelpers: Iterator {
fn dance(&self) {
println!("wheee");
}
}
// And tell rust that all Iterators are also MyIterHelpers.
impl<I: Iterator> MyIterHelpers for I {}
Not automatic, but you could use a decorator + the protocol as type annotation, I think
This is already implemented in IntelliJ for Java - they call it "Postfix Completion". For example you can type ".cast" after an expression to wrap what's before the cursor in a cast expression, so type "a + b.cast", then pick cast to "float", and pick how large a preceding expression you want to cast, and you can end up with "(float)(a + b)" and go from there. They have postfix completion that can extract expressions into variables, create if-statements and switch-statements from expressions, and so many more things that I wish I had when doing non-trivial Python coding in my IDE of choice (which is not by Jetbrains)...
Scala was really nice for this syntax when I used it for Spark.
min(map(some_op, filter(some_filter, bar.baz()))).foo()
min(some_op(item) for item in bar.baz() if some_filter(item)).foo()
items = (some_op(item) for item in bar.baz() if some_filter(item)) # or some better name given a context
min(items).foo()I’ll wrap the whole thing in a named function as a way of describing what I’m doing and make it a closure if it’s used only once:
def f(bar):
def smallest_baz():
bazs = (
some_op(b)
for b in bar.baz()
if some_filter(b)
)
return min(bazs)
return smallest_baz().foo() class WrappedList:
_fns = [map, filter, min, max, all, any, len, list]
def __init__(self, it):
self.it = it
def __getattr__(self, name):
for fn in self._fns:
if name == fn.__name__:
def m(*args, **kwargs):
result = fn(*args, self.it, **kwargs)
if hasattr(result, '__iter__'):
return self.__class__(result)
else:
return result
return m
def unwrap(self):
return self.it
WrappedList([1, 2, 3, 4]).filter(lambda x: x % 2 == 0).map(lambda x: x * 3).list().unwrap() # [6, 12]
WrappedList([1, 2, 3, 4]).map(lambda x: x >= 5).any() # False
self._fns = {fn.__name__: fn for fn in [...]}
[x * 3 for x in [1, 2, 3, 4] if x % 2 == 0]
I suppose we really ought to blame Euler for introducing the f(x) notation 300 years ago... Very practical when the function is the entity you want to focus on, often less useful in (procedural) programming, where we typically start with the data and think in terms of a series of steps.
Some languages like D and Nim have "UFCS", uniform function call syntax, where all functions can be called as methods on any variable. Basically, it decouples the implicit association between method dispatch and namespacing/scoping semantics. Rust also has something they call UFCS, but it only goes one way (you can desugar methods as normal functions, but you can't ... resugar? arbitrary functions as methods). Python couldn't implement this without breaking a lot of stuff due to its semantics, but it is definitely a feature I'd like to see more of.
Learn a better editor, and this will stop being a problem.
If you complain about doing this, this is because you don't know how to perform the basic functions necessary to write code. Heuristically, this is because you are either using a bad editor or didn't learn how to use a decent one.
I.e. your complaint is more comparable to Amazon reviews coming from people who don't know how to use the product and then write something asinine, like that one about a loo brush that feels too rough when used in the capacity of toilet paper (though I believe that one was actually a joke inspired by similarly stupid but less funny reviews).
To expand on this: examples of navigating by structure include moving by token / expression / definition. Examples of moving by syntax would be the search or "jedi" navigation (i.e. navigation where you enter a special mode requiring from you to type characters that iteratively refine your search results). Finally, simply moving up / down / left right by certain number of characters is the "screen geography" way.
There's no way to tell which method is better, because they apply better in different situations, however the "screen geography" method usually ends up being the worst, because it's the most labor-intensive and requires from the author to dedicate a lot of attention to achieve precision (i.e. move exactly N spaces to the left and then exactly M spaces down is very easy to get wrong, also, with larger N and M becomes really tedious).
Navigation by word is only slightly better than navigation by character, and often falls into the "screen geography" kind of navigation. It's easy to learn, it's quite universal and doesn't require understanding of the structure of the program or mastering better techniques (eg. "jedi jump"). That's not to say that it should be excluded from the arsenal -- quite the opposite, but a master programmer (in the sense of someone who writes programs masterfully) would be the one who's less reliant on this kind of navigation.
That never existed. Or if it did, it was long before any trace exists, and there's trace from quite a way back when e.g. the first commit in which I can find the len() builtin (https://github.com/python/cpython/commit/c636014c430620325f8...) also has calls to file.read and list.append, and the first python-level methods are created just a few commits later (https://github.com/python/cpython/commit/336f2816cd3599b0347...). Though there may be missing commits, this is 30 in, back when Python was an internal CWI thing (although nearly a year in, according to the official timelines of the early days).
This was years before magic methods were even added (https://github.com/python/cpython/commit/04691fc1c1bb737c0db...).
So no, I don't think it's a "holdover from" anything. Rather seems like it's GvR's sensibilities.
https://mail.python.org/pipermail/python-3000/2006-November/...
min(some_op(item) for item in bar.baz() if some_filter(item)).foo()
processed_items = (some_op(item) for item in bar.baz() if some_filter(item))
min(processed_items).foo()
The chaining example in OP’s first example is way better
I work in both python & js. The python reads like natural language:
Processed items is a set of some transformation of each item in bar.baz() where something is true for that item.
then Foo the smallest in that list.
JS-y stuff doesn't read like natural language, but I do think its more concise and fits the IDE function discovery workflow better.
Both models can be made into horrid messes or elegant solutions. Both are highly readable.
Now I like the python one because I find it natural to attach contextual "whys" or "because" comments to them.
Processed items is a set of some transformation of each item in bar.baz() where something is true for that item.
then Foo the smallest item.
# because foo is a slow function and we don't want to foo every bar and bazThe reason they are bad is that intermediate results are never named (and thus are never explained). In simple situations, it's possible to infer from the context what the author's intention was, but in more complicated cases, if you want to understand someone's code, especially if it's written in the way you did, you'd have to "disassemble" it into simpler operations, name the variables (after investigating or guessing the purpose of each operation) and then try to come up with the full picture of what's going on.
Also, as a style suggestion: avoid using backslashes. In your situation, you could just put dots at the end of the line and it will be enough to not need the backslashes. It adds noise to your code, i.e. characters that add no meaning, just sort of a "scaffolding" to hold your code together.
Naming intermediates is fine (and encouraged) if there are actually meaningful names to be given. But sometimes the expression itself is the shortest meaningful name for the expression.
Re backslashes, you can also just wrap the expression in parentheses.
same with give me `min(of_this)` instead of `of_this_want.min()`
The Rust x.min(y) to me is so asymmetric. min(x, y) conveys the symmetry of the operation much better, x and y are both just elements. (And the latter is how it can be used in Python. In Rust, you can call Ord::min(x, y) to get the symmetry back, but it is less favoured right now for some reason.)
from functools import partial
class Pipeable:
def __init__(self, fn):
self.fn = fn
def __ror__(self, lhs):
return self.fn(lhs)
def pipeable(fn):
return lambda *args: Pipeable(partial(fn, *args))
filter = pipeable(filter)
map = pipeable(map)
list = pipeable(list)
sum = pipeable(sum)
min = pipeable(min)
max = pipeable(max)
any = pipeable(any)
# Usage:
range(1, 100) | filter(lambda x: x < 50) | max()
# 49
[1, 2, 3, 4] | filter(lambda x: x % 2 == 0) | map(lambda x: x * 3) | list()
# [6, 12]
[1, 2, 3, 4] | map(lambda x: x >= 5) | any()
# False minimum = +Inf
for b in bar.naz():
if not some_filter(b):
continue
b = some_op(b)
minimum = min(minimum, b)
foo(minimum)
``` def myfunc(x=None): x = x if x is not None else [] ... ```
x = x or []
def listify(item, li=[]):
li.append(item)
return li
listify(1) # [1]
listify(2) # [1, 2] def func(arr=[])
# Look ma we mutated it.
arr.append 1
puts arr
end
The sibling comment is wrong too -- it is a local variable, or as much one as Python can have since all variables, local or not, are names.
If you were to do (the following is from memory, probably has typos):
def func(arr=[]):
print(locals)
def func(arr=[]):
print(locals)
print(func.__defaults__) # will print: ([],)
def func(arr=[]):
print(locals)
arr = 10
print(func.__defaults__) # will still print: ([],)
def func(arr=[]):
print(locals)
arr.append(10)
print(func.__defaults__) # will print: ([10],)
# if pristine and it has not been run before
def func(arr=[]):
print(locals)
arr.append(10)
print(func.__defaults__) # will print: ([],)
func([]) def f():
if not hasattr(f, "counter"):
f.counter = 0
f.counter += 1
return f.counter
print(f(),f(),f())
> 1 2 3 even 0 = true
even n = not (odd n-1)
odd 0 = false
odd n = not (even n-1) even 0 = true
even n = odd n-1
odd 0 = false
odd n = even n-1
I looked at python dis output before writing this, you can look at how it specializes in 3.11. But there's also 4 occurences of LOAD_GLOBAL f in the disassembly of this function, all four self-references to f go through module globals, which shows the kind of "slow" indirections Python code struggles with (and can still be optimized, maybe?)
You could scratch your head and wonder why even inside itself, why is the reference to the function itself going through globals? In the case of a decorated or otherwise monkeypatched function, it has to still refer to the same name.
Why not other places?
Using "yield" instead of "return" turns the function into a coroutine. This is useful in all sorts of cases and works very well with the itertools module of the standard library.
One of my favorite examples: a very concise snippet of code that generates all primes:
def primes():
ps = defaultdict(list)
for i in count(2):
if i not in ps:
yield i
ps[i**2].append(i)
else:
for n in ps[i]:
ps[i + (n if n == 2 else 2*n)].append(n)
del ps[i]A couple more along the same lines:
- Metaclasses and type. (This is admittedly dark magic, but useful in library code, less so in application code)
- Magic methods! Everyone knows about __init__, but you can override all sorts of behaviors (see: https://docs.python.org/3/reference/datamodel.html)
My favorite example (I have a lot of favorite examples :)) is __call__, which emulates function calling and is the equivalent of C++'s operator().
Why is it my favorite? Because as the old adage goes, "a class is a poor man's closure, a closure is a poor man's class":
class C:
def __init__(self, x):
self.x = x
def __call__(self, y):
return self.x + y
>>> a = C(2)
>>> a(3)
5I didn't have to use Metaclasses, either, though I have read about them, especially in Fluent Python. But I guess I belong to the 99% who haven't had to worry about them, yet :P
What is the benefit compared to having a method named "add" that also explains the behavior?
It may also give you a "clearer" (in quotes because subjective) presentation for something you're trying to do.
processor = SomeProcessor.load("path/to/config")
# with __call__
processed_inputs = processor(inputs)
# less awkward than
processes_inputs = processor.process(inputs)
If you don't know it already, it is really worth looking into. I am a python dev with nearly a decade of experience and I knew generators, and yet this was still an eye opener.
The concepts matter more than the chosen language in this deck.
I learned a lot! Looks like I can apply this to a PHP trace/profile parser project, especially the pipelined parsing and the query language idea.
is ps internal variable unique for each Thread or same?
is it safe to execute your primes() from different threads?
If you run it from different threads I guess it will be the same as calling the function multiple times, it will return a new started-from-the-top generator.
def sum():
yield 1
yield 2
print(repr(sum()))
print(next(sum()))
print(next(sum()))
<generator object sum at 0x7fc6f14823c0>
1
1You'll need to search, or try!
If you try to send g to a different process, you will get an error, because it doesn't serialize.
The best way to think of a generator is as an object implementing the iteration protocol. They don't really interact with concurrency, as far as multiprocess is concerned, they're just regular objects. So the answer is that it depends on how you plan to share memory between the processes.
> is ps internal variable unique for each Thread or same?
ps is local to the generator instance.
def f():
x = 0
while True:
yield (x := x + 1)
>>> f()
<generator object f at 0x10412e500>
>>> x = f()
>>> y = f()
>>> next(x)
1
>>> next(x)
2
>>> next(y)
1
For this specific generator, you would run into the GIL. More generally, if you're talking about non CPU-bound operations, you need to synchronize the threads. It's worth looking into asyncio for those use cases.
https://stackoverflow.com/questions/20579756/passing-value-t...
And don't forget "yield from" (same as yielding all values in a list, but keeps the original generator! You can send data back to the list if it is itself another generator!)
[
{"name": "/dev/loop0"},
{"name": "/dev/loop1"},
{"name": "/dev/loop2"},
{
"name": "/dev/sda",
"children":
[
{
"name": "/dev/sda1",
"children":
[{"name": "/dev/mapper/lubuntu--vg-root"}, {"name": "/dev/mapper/lubuntu--vg-swap_1"}],
},
],
},
{"name": "/dev/sdb", "children": [{"name": "/dev/sdb1"}, {"name": "/dev/sdb2"}]},
{"name": "/dev/sdc", "children": [{"name": "/dev/sdc1"}, {"name": "/dev/sdc9"}]},
]
def flatten(items):
for item in items:
yield {k:v for k,v in item.items() if k != 'children'}
if 'children' in item:
yield from flatten(item['children'])
[{'name': '/dev/loop0'},
{'name': '/dev/loop1'},
{'name': '/dev/loop2'},
{'name': '/dev/sda'},
{'name': '/dev/sda1'},
{'name': '/dev/mapper/lubuntu--vg-root'},
{'name': '/dev/mapper/lubuntu--vg-swap_1'},
{'name': '/dev/sdb'},
{'name': '/dev/sdb1'},
{'name': '/dev/sdb2'},
{'name': '/dev/sdc'},
{'name': '/dev/sdc1'},
{'name': '/dev/sdc9'}] def flatten(children=[], **other):
if other: yield other
for child in children: yield from flatten(**child) question_bank={'1+1' : '2', '2+3' : '5'}
def Quiz():
for question, correct_answer in question_bank.items():
answer = yield question
if answer == correct_answer:
print('Correct!')
else:
print('Wrong.')
yield 'Finished!'
question = Quiz()
q = next(question)
while q != 'Finished!':
q = question.send(input(q)) def get_api_results(query):
params = { "next_token": None }
while True:
response = requests.get(URL, params=params)
json = response.json()
yield from json["results"]
if json["next_token"] is None:
return
params["next_token"] = json["next_token"]
for result in get_api_results(QUERY):
process_result(result) # No need to worry about paginationI think you could simplify the rest like so:
def primesHN():
from collections import defaultdict
from itertools import count
yield(2)
ps = defaultdict(list)
for i in count(3,2):
if i not in ps:
yield(i)
ps[i**2].append(2*i)
else:
for n in ps.pop(i):
ps[i + n].append(n)It is. Itertools is a masterpiece of a module. It has a lot of functions that operate on iterators and will work both on standard iterables (lists, tuples, dicts, range(), count() etc.) and on your own generators. It forms a sort of "iterator algebra" that makes working with them very easy.
> I think you could simplify the rest like so:
Sounds good, but with a caveat: you do need to call "del" at the end for memory deallocation purposes. The garbage collector isn't smart enough to know you won't be using those dictionary entries any longer. Technically the code still works, but keeping everything in memory defeats the purpose of writing a generator.
The garbage collector doesn't understand "pop"? That seems...dumb? ¯\_(ツ)_/¯
if(x > 0): ...
if x > 0: ...
import pdb
pdb.set_trace()
breakpoint()This is, if anything, better. Because that way you can e.g. replace stray `breakpoint()` calls by warnings rather than break production :D
if not n in memo:
if n not in memo:The sent can only be interpreted one way.
- none of these functionalities are "overlooked", this is pretty basic python
- for fibonacci you have a decorator for memoization (functools cache / lru_cache)
- you don't need to use parenthesis for a single line "if"Fwiw I also thought it was pretty regular stuff, and then arcane library functions you've either needed or you haven't. Also, that's a generator, not a list comprehension.
You are right about the fibonacci operator, I thought I did refer to another article where I mention the lru_cache as well :) But I'll double check.
Good one about the parenthesis! I'll post an update soon
Even though most of the stuff OP writes about is worthless, I see it used a lot (if it's old) and less so, but still enough otherwise...
This article reads to me like as if it was written by someone learning Python, perhaps in their 3'rd-4'th month, when they finally decided to open documentation / some existing project code instead of implementing calculators and animal class hierarchies...
Here's one more trick: you can use `pdb` to run scripts! `python -m pdb foo.py` will run `foo.py` but trigger a breakpoint on the first error.
So can any other variable, using underscore is just a convention to make it obvious that you're not planning to re-use it (it doesn't get GCed more aggressively or anything).
Similarly, private methods being prefixed with an underscore is also just a convention, you can access them from anywhere.
However, double underscores are used for magic attributes and name mangling for class attributes, which are interpreted differently! (See: https://stackoverflow.com/a/1301369)
>>> 1+1
2
>>> _ * 3
6The `first, *middle, last` trick doesn't work if your list only has one element:
first, *middle, last = [1]
ValueError: not enough values to unpack (expected at least 2, got 1)
> Separater for Large Numbers
The mistake is suggesting the unpacking without caveats, because it'll fail in situations that the naive solution doesn't:
items = [1]
first = items[0]
middle = items[1:-1]
last = items[-1]
This is a giant pain. Easy to miss. Sometimes forces you to deal with Optional[Something] instead of just Something.
Compare with Julia where default arguments are evaluated ... very late:
julia> f(a, b, c, d = a * b * c) = d
f (generic function with 2 methods)
julia> f("hello", " ", "world")
"hello world"
Looking at your Julia example, this seems much more friendly and less surprise and error-prone.
first, _, last = [1, 2, 3, 4, 5]
first, *_, last = [1, 2, 3, 4, 5]
Other than that, nice list of python tricks, I love not-so-known features because it can make code shorter and prettier!
I would never try to exploit this behavior to achieve some kind of benefit (avoiding max recursion). Any tricks you try to do with this is almost definitely going to to cause bugs that are very difficult to track down. So don't be too clever here.
I chose to learn Python because it seemed to be the easiest to read, which to my mind meant working in a team would lead to easier discovery and understanding. Then I see articles like this, and wonder if I'll have a lot of footguns to watch out for where the code isn't as clear as it seems.
* For dicts, learn .setdefault() vs. .get() vs. defaultdict()
* .sort(key=sortingkey)
* itertools groupby, chain
* map, filter, reduce
Here's the obvious question: how many more unknown-but-useful features are hidden away in other similar articles.
- `repr` often outputs valid source code that evaluates to the object, including in the post's example: running `datetime.datetime(2023, 7, 20, 15, 30, 0, 123456)` would give you a `datetime.datetime` object equivalent to `today`.
- Using `_` for throwaway variables is merely a convention and not built into the language in any way (unlike in Haskell, say).
We have definitely found this to be true in hiring. Many people’s Python knowledge seems to just be surface deep.
It has some wild features and crazy syntax and if you know it, it's probably awesome, but I too like to keep it mostly simple and obvious.
But as you said: if you know it, it's probably awesome. In my opinion, it never gets boring to discover new things in Python, and it does make you a better Python developer. Knowing what and when to apply certain knowledge is where your experience comes in.
# z = [(x0,y0), (x1,y1) ...]
import matplotlib.pyplot as plt
plt.plot(*zip(*z))
x = [t[0] for t in z] # etc
Using a list comprehension, such as your original approach, is pretty easily understood by anyone writing python and is easy to follow, it is also quite terse.
Your recursive unpacking zip thing is much harder to understand and read. This reminds me of the type of stuff you find in the codebase years later when the person who wrote it is long gone and you find a comment next to it that says:
# No idea why this works, but don't touch it
One of the problems I have with python is that there are a million super creative ways to do stuff, especially using less known parts of the language. People love to get super creative with it, but usually the simplest solution is actually the best one, especially when working on a team.
In your example above, you aren't even saving any real space. Both approaches can be done inline, the list comprehension is maybe a few extra characters. You're not really saving anything, just making it harder to read and maintain by others.
When I moved from a company that wrote in Python to one that wrote in Golang, I found that the restrictions that Golang offers is a huge benefit in a team. Because you don't have access to all these crazy language components that python has, the code written in Go would be almost identical regardless of who wrote it. Of course everything in Golang is far far more verbose than Python, but I actually found it 100x more maintainable.
In the python codebase it was very easy to tell who wrote different parts of a codebase without looking at the git blame, because there was almost a "voice" with the style of writing python. But in Golang it was more restrictive which meant that the entire codebase was more cohesive and easily to jump around.
columns = zip(*rows)
def transpose(list_of_lists): return zip(*list_of_lists)
a=1+3j
b=a+4j
np.mgrid[-1:1:5j]This one can be explained as "equivalent to np.linspace(-1, 1, 5)", i.e 5 evenly spaced points between -1 and 1. Normally the step size is an integer but with a complex "step" it switches the meaning from step size to number of equidistant points.
import random
some_value = 9 # return a number between 0 and, including, 100
if below_ten := some_value < 10:
print(f"{below_ten}, some_value is smaller than 10")
Even in the next example of the walrus operator - it is extremely contrived:
if result := some_method(): # If result is not Falsy
print(result)
if some_method():
print(True)The first, *_, last trick for example would be particularly obnoxious to encounter. The first element is my_list[0], last is my_list[-1]. Dead simple, way easier to understand at a glance.
`import code; code.interact(local=locals())`
Drops you into the interpreter and is sufficient for a lot of debugging problems.