Python-patterns – A collection of design patterns/idioms in Python
github.com
github.com
The stuff that is actually idiomatic python is either too convoluted or rather obvious.
I know this comes across rather mean, but this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples). There's maybe 2 or 3 good ideas in there, but even those I might be hard pressed to find a real usecase for.
But that's likely a me problem, I assume that for people more familiar with monads and all that jazz, it's easier to understand.
# note: `None and f()` returns None without calling f
balance = user and user.get_balance()
credit = balance and balance.credit_amount()
discount_program = choose_discount(credit) if credit and credit > 0 else None
Optional[Foo] is same as Foo | None, meaning you can operate on foo with Foo's methods, except that if it's None, you get NoneType errors.
Maybe[Foo] is an actual container. You have to map over or unwrap it to operate on Foo.
The big difference is ergonomics/opinion. You can't actually map over Optional, so every time you wanna use it, you have to manually check if foo is not None. Whereas Maybe, you can operate generically over it. Some folks think "is not None" is better. Personally I hate NoneType errors in prod and find that much more painful than a bit of indirection.
Background: an object that either None or true in a boolean context (true unless overriden for custom objects). Given such object, we can consider it in a virtual Maybe container/box. When we want to use it, we have to unwrap it using `obj and obj.method()` syntax. Then `obj.method()` is ordinary "unwrapped" call.
Just to remind you. Here's how "ergonomic" Maybe variant from the article look like:
# Type hint here is optional, it only helps the reader here:
discount_program: Maybe['DiscountProgram'] = Maybe.from_optional(
user,
).bind_optional( # This won't be called if `user is None`
lambda real_user: real_user.get_balance(),
).bind_optional( # This won't be called if `real_user.get_balance()` is None
lambda balance: balance.credit_amount(),
).bind_optional( # And so on!
lambda credit: choose_discount(credit) if credit > 0 else None,
)
You can decide for yourself what is more readable: all these lambdas or the `None and f()` code.
But `obj and obj.method()` really is not the same thing as `obj.map(method)`. "virtual Maybe container/box" is a nice idea, and does actually type-check with mypy (mostly), but you cannot actually compose it with other functions. The problem is, each time you do `obj and obj.method()`, you end up union-ing type(obj) and type(obj.method).
https://mypy-play.net/?gist=5b03d81a64453997984e448df9b889e7
main.py:40: note: Revealed type is "Union[__main__.User, None]"
main.py:41: note: Revealed type is "Union[__main__.User, None, builtins.float]"
main.py:42: note: Revealed type is "Union[__main__.User, None, builtins.float]"
main.py:43: note: Revealed type is "Union[__main__.User, None, builtins.float, builtins.str]"
I'd consider Java-like being a regression, not progress.
In my experience, most of the time when people complain about medium-large Python projects being difficult to reason about, it's because they wrote Java code in Python.
You can easily guess I'm not a huge python fan (nor java, but it so happens I work daily in medium-sized projects in python), and I do have problems with the "idiomatic/pythonic way", but lets face it - python, while very expressive, is often so convoluted that static analysis of code isn't even possible. If your IDE is not able to quickly determine a given var type, you expect the average developer to be able to do it?
Python is - first and foremost - a prototyping language, and an excellent choice for glue code. For big applications, you will actually learn to enjoy the bitterness and complexity of Java over python, and small stuff like strict typing.
As someone with experience (just not in Python) I don't mind if the examples are basic, as long as they're idiomatic. I'd prefer a list of very common tasks/problems solved in a pythony way vs some examples of unique and clever uses of python.
100% yes.
Idiomatic would be (to me):
1. things which python does naturally - for instance list expressions rather than manual creation:
managers = [
person for person in get_all_staff()
if person.has_direct_reports()
]
managers = []
for person in get_all_staff():
if person.has_direct_reports():
managers.append(person)
2. Things that Python makes natural / or less so - by the way it works.
For instance, because of the important whitespace, and no end brackets or `end` keywords or whatever at the end of `if` or function definitions, large nesting inside functions becomes very klunky to follow the logic. So idiomatic would be to use small named functions, and split things into smaller pieces once they become too nested.
3. Common patterns that many people do, or common functions. See the itertools recipies, for instance: https://docs.python.org/3.5/library/itertools.html#itertools...
Of course, one has to mention this:
> >>> import this > The Zen of Python, by Tim Peters > > Beautiful is better than ugly. > Explicit is better than implicit. > Simple is better than complex. > Complex is better than complicated. > Flat is better than nested. > Sparse is better than dense. > Readability counts. > Special cases aren't special enough to break the rules. > Although practicality beats purity. > Errors should never pass silently. > Unless explicitly silenced. > In the face of ambiguity, refuse the temptation to guess. > There should be one-- and preferably only one --obvious way to do it. > Although that way may not be obvious at first unless you're Dutch. > Now is better than never. > Although never is often better than right now. > If the implementation is hard to explain, it's a bad idea. > If the implementation is easy to explain, it may be a good idea. > Namespaces are one honking great idea -- let's do more of those!
So things like readability beats conciseness, so a huuuuge complex list expression is less pythonic than a manual loop when it becomes too complex.
>For instance, because of the important whitespace, and no end brackets or `end` keywords or whatever at the end of `if` or function definitions, large nesting inside functions becomes very klunky to follow the logic. So idiomatic would be to use small named functions, and split things into smaller pieces once they become too nested.
That's not idiomatic, that is common sense, and common to basically every language. As a rule of thumb, if you have nestings grow deeper than 3-4 levels you're probably better breaking it. This is also a common cyclomatic complexity measure in most static analysis tools. Python makes following the logic clunky even in one liners, because it isn't obvious your code is not branchless. I also understand it is a matter of taste, but I do get quite annoyed with the whole "idiomatic python" thing, because often is just stuff that encourages what - in my humble opinion - are often less-than-ideal practices, that don't translate well to other languages.
> So things like readability beats conciseness, so a huuuuge complex list expression is less pythonic than a manual loop when it becomes too complex.
You don't need to have a huuge complex list. Imagine the example you gave, where later you need to add 2 more conditions to the if clause. What approach you think will age better between changes and refactoring? Might as well do it without list expressions from the get-go, and not having the novice developer who will be maintaining it have to read it 2 or 3 times to understand what is going on.
Strategy pattern is a great example. First class functions make the GoF pattern a moot point. But it's still useful to understand that this is a useful thing to want to do, whether or not one has direct language support.
Things like the GoF model for how to implement Strategy are kind of a big deal in languages that lack first class functions. But there is also universal value in the basic idea of structuring your code so that you can inject behavior from outside. It's so valuable that many languages weave it into the very fabric of their design. FP fans fixating on the complexity of things like the GoF Strategy pattern and concluding that design patterns are just a band-aid over missing language features strikes me as being like fish noticing the inconvenience of humans needing plumbing and drinking vessels to get their water, and congratulating themselves for not needing water.
No matter what pattern we look at, we find it's actually hiding some absurd language weakness. Factories? They're just normal methods that hide the new keyword which introduces hard coupling at the ABI level and can't construct subtypes. Singletons? They exist to work around the fact that classes don't act like objects: classes with static state are natural singletons but they can't implement interfaces or be passed around as objects.
In better languages such as Ruby we discover that new itself is actually a factory method and nobody even realizes it. In Ruby classes are objects and natural singletons and there are no limitations to work around.
You weren't kidding
* "Python Design Patterns": https://python-patterns.guide/
* "Clean Architectures in Python": https://www.thedigitalcatbooks.com/pycabook-introduction
* "Architecture Patterns with Python": https://www.cosmicpython.com/book/preface.html
First, some generic tips on how to write Python code that's pleasant to work with: https://youtu.be/OSGv2VnC0go
And then this one digs more specifically into how not to write Python like it's Java, starting at 12:40 (with talking about why leading up to that): https://youtu.be/wf-BqAjZb8M
https://github.com/faif/python-patterns/blob/master/patterns...
I was nodding along like yup, yup, yeah… … …What?
The thing is, this pattern is crucial in C++. It’s how MLIR works. All objects are actually values that have shared internal state. Meaning you can copy them around as much as you want, just like Python, and you’re not copying anything except an internal pointer. It’s like a smart pointer but without any pointer interface, and I’ve wished everything in C++ worked that way. (Memory is scoped to nearest enclosing context, and contexts manage the objects below it. Very simple.)
The other case that this is useful is when you have a bunch of views into some data. Consider the concept of a global variable. Globals are great. I love them. But they require discipline. If you want to spin up a bunch of threads that each have their own view of that global, you’re hosed.
Except you’re not. What you can do is have a thread local variable that initializes itself to the value of the global variable. That way new threads start with the current value of the global. Why? Because suddenly you can just pretend like you’re using globals everywhere! Whenever you want to dish out some work, set the global to foo, then spin up a thread. Set the global to bar, then spin up another thread.
Instead of passing that damn value through 100 smaller functions, all the code simply refers to “the current user” or “the current webpage” or “the current widget”, which is a global concept. If you ever need to modify it, you set the global and spin up another thread.
Internally it’s implemented by a pattern similar to borg. The instance returns its thread local state, unless it’s nil, in which case it initializes from the global value.
The nitpick is that this needs to be true for child threads that spawn threads, otherwise the whole thing doesn’t work well anymore. Racket nailed this concept with thread cells. If a thread cell is preserved, any child thread immediately inherits the value from the spawning thread. And it’s all transparent.
With this borg example, it’s just literally a global variable. Or maybe it’s not and someone will come say why that’s mistaken. But I don’t think so!
Personally I'm a fan of dependency injection whenever possible. But borg pattern has gotten me out of a pinch (exactly) once.
It's really not the most compelling reason. I recently found a cleaner way that actually achieves identity-equality, which I think is a much better way of going about things. Not actually using it anywhere at the moment, but I think it's neat.
class Singleton(type):
def __init__(cls, name, bases, dict):
super(Singleton, cls).__init__(name, bases, dict)
cls.instance = None
def __call__(cls, *args, **kw):
if cls.instance is None:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
return cls.instance
class MyClass(metaclass=Singleton):
...
class MySubclass(MyClass):
...
>>> MyClass() is MyClass()
True
>>> MySubclass() is MySubclass()
True
>>> MySubclass() is MyClass()
FalseAny others like that? An opportunity to ask someone about that is pretty rare.
1. creating complex objects and having them valid at creation
2. many optional arguments for an object's construction
3. similar types for object construction
For e.g. in Java you might see something like:
Point p = new Point(10, 0, 5); // (x=10, z=5)
Point q = new Point(10, 2); // (x=10, y=2)
1. y needs to be specified explicitly to 0, because you can't expose an overloaded constructor that takes x and z (that would clash with the constructor that takes x and y
2. It _may_ not be evident that new Point(10, 0, 5); passes x, y and z in that order
3. It ties the constructor to the implementation. You can never expose a Point(double radius, double theta) constructor because that would clash with the Point(double x, double y) constructor.
The builder would solve all of these problems:
Point p = Point.newBuilder()
.setX(10)
.setZ(5)
.build();
1. named arguments, and
2. default arguments
so you could have:
p = Point(x=10, z=5)
q = Point(10, 2)
t = Point(radius=5, theta=45)
I will say, the book is heavily focused on OOP, which has fallen out of favor a bit with newer languages (but obviously very alive), and it's almost like a time piece. Feels like OOP was seen as this answer to so many business problems and organization of code.
Just interesting to imagine OOP at a time when it was new and exciting, since I grew up at a time when Java was ubiquitous and everyone was at least a little OO.
So you could do:
import greeter
greet1 = encapsulate(greeter)
greet2 = encapsulate(greeter)
greet1.name = 'Joe'
greet2.name = 'Sue'
greet1.greet()
greet2.greet()
name = 'nobody'
def greet():
print ('Hello '+name)
Hello Joe
Hello Sue
Is using modules like that nicer? That's subjective, I guess. I don't think it's nicer. You could use dataclasses to make classes somewhat nicer (though that is subjective too).
It it leaner than classes? I don't think it's something to worry about. Maybe use __slots__ in your class if and where needed. Is an encapsulated module still leaner than a class?
In the end both modules and classes are namespaces. Modules are singletons, classes have as many instances as you need.
def greeter(name):
def greet():
print(f"Hello {name}")
return greet
greet1 = greeter('Joe')
greet2 = greeter('Sue')
greet1()
greet2()
"""python
from functools import partial
def greeter(name):
print(f"Hello {name}")
greet2 = partial(greeter, "Sue")
greet1()
greet2()
"""
~/Code/local/hn ································ at 09:14:14
python3 hn_example.py
Hello Joe
Hello Sue
~/Code/local/hn ································ at 09:14:16
cat hn_example.py greeter.py
───────┬───────────────────────────────────────────────────────────
│ File: hn_example.py
───────┼───────────────────────────────────────────────────────────
1 │ import sys
2 │ from importlib.util import find_spec, module_from_spec
3 │
4 │ def encapsulate(mod):
5 │ s = find_spec(mod)
6 │ m = module_from_spec(s)
7 │ s.loader.exec_module(m)
8 │ return m
9 │
10 │ greet1 = encapsulate("greeter")
11 │ greet2 = encapsulate("greeter")
12 │
13 │ greet1.name = 'Joe'
14 │ greet2.name = 'Sue'
15 │
16 │ greet1.greet()
17 │ greet2.greet()
───────┴───────────────────────────────────────────────────────────
───────┬───────────────────────────────────────────────────────────
│ File: greeter.py
───────┼───────────────────────────────────────────────────────────
1 │ name = 'nobody'
2 │
3 │ def greet():
4 │ print ('Hello '+name)
───────┴───────────────────────────────────────────────────────────
I think this isn't a good idea though, i'd lookup __slots__ if leanness is an issue.
Do you think it negatively affects performance?
Why don't you think it's a good idea?
I don't see how __slots__ would be relevant to this? Isn't __slots__ a thing to define how classes store their properties?
Yes and no. Pragmatically, no, because in python usually performance comes from (ignoring using native code) avoiding dynamic dispatches (circa 500x speedups are common) and like in almost all languages, being aware of your allocations (varies wildly but avoiding allocs in loops for example will give remarkable speedups).
In theory though, yes absolutely this is slower - what was a query of a dict (or even better, an array in the __slots__ case) - i.e. grabbing an attribute from a class - is now going outside the process and asking the OS about stuff on the filesystem. Masssively slower.
>> Why don't you think it's a good idea?
Because you’re paying an overhead to use something in an unexpected way when the language provides a faster, idiomatic solution to this problem:
>> a pattern to encapsulate data and functions
That’s the class and object system described in one sentence
>> I don't see how __slots__ would be relevant to this?
You’ve heard of turtles all the way down? In python it’s dicts all the way down. Behind the scenes you can think of a loaded module being a dict, just the same as an object has a dict underpinning it.
Slots comes in as an optimisation on this dicts all the way down. An optimisation that’s not open to you if you (ab?)use modules like this.
But i’d finish up by saying this - i’m a random person on the internet that doesn’t know your use case. I stand by the above as decent general case advice but you might actually have a valid case to deviate AND you’re not going to unlock new discoveries in better ways to do things if you just always follow the dogma so don’t be put off by my negativity toward the idea but DO measure the outcomes you care about when experimenting to see if the modules idea is working for you.
>>> from dis import dis
>>> dis(greet)
4 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Hello ')
4 LOAD_GLOBAL 1 (name)
6 BINARY_ADD
8 CALL_FUNCTION 1
10 POP_TOP
12 LOAD_CONST 0 (None)
14 RETURN_VALUE
>>> dis(greet1.greet)
4 0 LOAD_GLOBAL 0 (print)
2 LOAD_CONST 1 ('Hello ')
4 LOAD_GLOBAL 1 (name)
6 BINARY_ADD
8 CALL_FUNCTION 1
10 POP_TOP
12 LOAD_CONST 0 (None)
14 RETURN_VALUESo with this we can skip the class definition, having "self" as the first parameter of every function and prefixing all class variables with "self".
That leads to so much cleaner code!
def encapsulate(mod):
import types
out = types.SimpleNamespace()
def replace_global_scope(f):
# via https://stackoverflow.com/a/1144561, but tweaked for py3
return types.FunctionType(
f.__code__,
out.__dict__,
f.__name__,
f.__defaults__,
f.__closure__,
)
for name in dir(mod):
val = getattr(mod, name)
if callable(val):
val = replace_global_scope(val)
setattr(out, name, val)
return outI don't care about code completion at all. When I type "a", I don't want anything to happen on my screen except for an "a" to appear. I think that all the flickering stuff that the average programmer experiences on their screen burns more mental energy than it saves.
My "encapsulate" was 4 lines of code at most IIRC. As someone mentioned, autocomplete in PyCharm did not work.
FWIW I care about code completion, and using a library that broke my workflow is not fun.
Not with the standard library pickle, but "dill extends python’s pickle module for serializing and de-serializing python objects to the majority of the built-in python types".
I like coding in Python because it makes me feel free from Java stuff, but those patterns have getting me worried.
I have found that Python really can show you what other language is someone's primary language. You can write Python to look VERY Java-like, or very Ruby-like, or many others.
Idiomatic Python does not look like this generally. Another commenter hit it well that this is just badly-hidden Java code written with Python.
Python is what you make it to be; don't let this kind of stuff from random people on the internet scare you into thinking you have to use GoF and Java-like patterns and classes everywhere. You CAN, but only if those are actually _useful_ to you.
You know? When I feel burned out of Java, I usually write a quite simple script in Python and then, normally I get energy to complete my tasks.
I don't imagine myself struggling against the same design patterns in Java when coding in Python.
Thanks for your advice.