Python Design Patterns (2018) | Hacker News Reader

Python Design Patterns (2018)

python-patterns.guide

372 pointsluu90 comments

I feel a bit contrarian about design pattern lists. These always seem to contain a list of solutions and after some fancy gymnastics, a list of potential problems they might solve. I feel it might be more useful to create a list of common software problems and follow them with a pattern that helps solve the problem. I think this would help address the issue of these pattern guides being applied overzealously.

Cookbook?

Pattern clusters would be better. Like “… so you want to design a programming language, this is the Visitor pattern, the Dispatch…”

Reading top comments might make someone feel that this article is not worth reading, but I encourage people to actually read it. Personally I find it really interesting.

For eg. reading about prebound methods here: https://python-patterns.guide/python/prebound-methods/. Seasoned python programmers probably wouldn't find anything interesting there, but for me, it was quite useful including instances of where it is being done in python standard library and discussion around when and when shouldn't it be used.

I agree. The information is interesting to read and it is well written.

Bound methods in Python are one of the keys to enlightenment to understanding its object system -- that self isn't magic and you could implement it yourself (for example having instance methods take a self, and cls parameter) if you wanted. One fun exercise is implementing an object that can "adopt" loose functions.

   def some_random_function(self)
     print(self.internal_state)

   ...

   mymagicobject.func = some_random_func

   mymagicobject.func()  # -> prints the internal state of the object

> a list of common software problems and follow them with a pattern that helps solve the problem

Better yet, improve the programming language, so that any user may benefit, not just those who know about the obscure list.

https://blog.plover.com/prog/design-patterns.html

If you're interested in Python anti-patterns instead, we wrote an open-source book on them a while ago: https://docs.quantifiedcode.com/python-anti-patterns/

Hasn't received any updates in the last years but it still holds up quite well, disregarding the Python 2 specific anti-patterns of course which become less and less relevant, as well as the Django-specific ones, which probably are hopelessly outdated.

Not a bad list. Feels like most of them should just be linter rules. I also found a few them don't really apply if you're using static types (which you really really really should), e.g.

* Returning more than one variable type from function call: That's fine; the type checker will make sure you don't make a mistake.

* Asking for permission instead of forgiveness: Makes sense for the example given (`unlink`) but it is better to e.g. check types explicitly with `isinstance()` than to catch `TypeError`.

Yes absolutely, that book was written before type annotations made it into Python though.

Asking for permission instead of forgiveness can easily become an anti-pattern in itself, because it allows lazy code that turns

`if known type has expected property`

into

`try prodding this Any object, and catch where it blows up later`

This seems to be missing my biggest gripe: adding new attributes to classes outside of `__init__()` or `__slots__`. Sadly I have to deal with code that does this with some frequency. Sometimes the attribute is added and accessed outside of instance methods entirely, ad-hoc by some caller. The result is total spaghetti. It becomes impossible to really know what the "interface" of the class is when you have to consider this, plus multiple inheritance, plus metaclasses, and then on top of that someone decided to hijack `__getattr__`. If I have to comb through the entire codebase to understand one class, that's a huge antipattern.

To avoid this, a lot of the new Python code I write is full of dataclasses. I do feel like I'm leaving a little bit of power on the table, but at least I can understand the code at a glance.

> This seems to be missing my biggest gripe: adding new attributes to classes outside of `__init__()` or `__slots__`.

You don't strictly need to use dataclasses to accomplish this. Yes, you get stronger guarantees with dataclasses (such as runtime checks, and these come with a runtime cost), but you can declare your members the same and use a properly configured linter that will tell you if you're assigning members from places you shouldn't.

Personally, I tend use dataclasses more for plain old data types. As soon as one starts needing/wanting to control __init__, I find the utility of dataclasses drops precipitously. But, I do still annotate my (public) members with proper types, trying to avoid `Any` as much as possible.

On a quick scan over the ToC, one important anti-pattern seems to be missing - automated code generation.

Honest question: I hear these terms used quite a bit. What qualifies as patterns/anti-patterns?

(From my naive understanding, patterns are some language features which should be used over conventionally accepted ways. Please correct if wrong)

Patterns are not really language features so much as the lack of them - they're "standardised" ways of solving a problem that the language doesn't have a feature for, so that at least everyone is writing the same thing and a reader might be able to understand the code at a glance.

Anti-patterns are pretty much the same thing, just where there is some problem with the common solution that everyone is using - perhaps there actually is a language feature that addresses this use case after all, for example.

So if I write a recursive function to calculate factorial of a number (or some permutation calculator), will that be considered a pattern? Because Python3 wouldn't have the factorial built-in. Thank you for the explanation above.

Once isn't a pattern, factorial isn't normally seen as a language feature. Implementing recursion via the Y combinator is a pattern (for languages that don't have built-in recursion) because you do it again and again for many different functions.

__init__.py shouldn't contain any "business logic" other than exposing symbols.

Validation?

Unit tests.

You only need to validate things once typically. Why force the user to run validation code over and over on a code base unlikely to change?

Oh, fair enough. I had misread the GP comment as being about __init__ (constructor methods) rather than __init__.py, package re-export.

I'd make a small and mostly pedantic exception to "nothing but re-exports in __init__.py" for cases where you have a module with data (non-Python) files adjacent to the code, but which doesn't need more than one Python file to define its module logic. No point to have a tiny __init__.py that re-exports from an adjacent somemodule.py in that case.

But generally yes, you're right.

Not idiomatic Python. Don't model your best practices in Python based on what makes sense in e.g. C++ and Java.

100000%, Python is really excelling at functional programming these days. With dataclasses & typing we now effectively have GADTs. Every few months I'm noticing small functional improvements, and I'm loving it! Even though it says `class`, I use it for GADTs, Protocols, and eDSLs--and it feels very idiomatic. These are the 'patterns' you see in SQLAlchemy, jax, pytorch.

Interesting stuff! What are the sources to learn more?

I don’t like these content about programming. Why listing a bunch possible of answers to some solved problems instead of try to dive deep into the problem themselves.

This feels like encouraging people to buy a bunch of cameras instead of teaching people how to take good photos.

A good solution may end up looking like some patterns, but sourcing good solutions from a pool of patterns is impossible unless the problem you are trying to solve is the exact same as the one a pattern is solving, in the same environment and context.

I do find the content can be very useful for many, but can we not call them patterns?

I agree.

Heuristics?

They're mostly useful as training wheels, once you've solved a couple of similar problems you can probably do better using local knowledge / pattern matching against actual experience.

I remember the Bridge pattern helping me along the way, I was trying to design a GUI with multiple back ends in C++.

Similar resources:

* Clean Architectures in Python: https://www.thedigitalcatbooks.com/pycabook-introduction/

* Architecture Patterns with Python: https://www.cosmicpython.com/book/preface.html

* Collection of design patterns and idioms: https://github.com/faif/python-patterns)

FWIW, if you really want to understand Pattern Language you should go and read the source material: Christopher Alexander's book of the same name and it's companion volumes. The GoF did a fine job, but there's a richer and deeper constellation of ideas and philosophy and design strategies in the original material.

- - - -

In re: Singleton pattern in Python: one interesting technique is to use a module as a singleton. When client modules "import singlefoo" the first import caches the module object (in the sys.modules dict) and subsequent imports use it: presto! Singleton!

My first thought was that it's nice to see someone take such a humane and scientific approach to public spaces, and a lot of it makes sense. But to me that's the real essence of the book, not the fact that it contains patterns. It should be required reading for city planners, not software developers.

Yeah, this is usually the best way for creating singleton in Python. But this makes mocking the singleton module in unit tests harder, as this line is run during module import itself. If the singleton is making a network connection (say) and you want to mock it for the unit tests, things get clumsy and harder.

> there's a richer and deeper constellation of ideas and philosophy and design strategies in the original material

Explanation with a bit more details: https://perl.plover.com/yak/design/

Some of these patterns have proved to be really powerful in real-world projects.

One challenge is having them fresh on my mind and being able to spot when it makes sense to apply them.

Have been thinking about using flashcards and spaced repetition to achieve better retention and ability to apply in practice.

Anyone has experience with this?

My experience is that worked, traditional textbook-style exercises are the best way to learn and internalize this kind of knowledge. A combination of shorter exercises with longer open-ended tasks is good. Basically more or less what you'd get on a homework problem set in a university course.

Maybe the fact that this kind of work ends up being "spaced repetition" in some sense is part of why it's helpful.

Maybe work from the other direction? Work through each pattern and think about where they might work in your existing codebases and then try implementing them just for the sake of hands-on experience.

I made the experience that they are hard to learn and understand out of context, so practicing on flashcards does not make much sense for me. What I did is programming some mini projects in which one or two patterns are used: Pub-sub is nice in anything UI related, commands+state is great for a simple undo/redo-manager etc.

Agreed. I'd use flashcards just to retain a fresh memory of what I've already learned with proper context and practice.

As others wrote, in depth practice is a sensible choice. Once it's done, I believe spaced repetition is a nice insurance against memory fading and can be as simple as a few big cards listing the concepts learned even if that's really not what spaced repetition proponent would suggest.

Use good libraries and pay attention to what makes them appealing. Large programs should usually be structured as a collection of libraries. When you are designing an API for your own libraries, think back to how other libraries solved similar problems at the API level. Many (most?) worthwhile design patterns can then be derived from scratch as a way to achieve a particular API structure.

Insanely powerful and underutilized technique: write a list of techniques you can apply (in this case, patterns). Just write the names, you want this to be dense. When you have a choice to make, look at the list. It doesn’t take long to scan 12 names.

That makes sense. Just thought about having the list as a colletion of flashcards and using a software like Anki to automate the revisions.

I know and have used these patterns. Some I use frequently, like Observer. The less frequent ones just fade in memory, and I get myself constantly in need to read about many again to refresh and identify which to use in a particular case.

And I suspect I'm missing many potential uses, due to not having them very fresh in mind all the time.

That's where I presume SRS can help...

I’m very familiar with SRS and I think you don’t need it here.

I get what you’re going for. You learn the patterns, but then you don’t recognize the opportunity to use them. You think you have to keep the pattern in your recent memory so you’ll recognize the opportunity to use it.

Here’s the trick: you don’t have to recognize the opportunity to use a pattern. You just have to recognize the opportunity to use a pattern. Instead of thinking “ah yes, Builder pattern would go here” you can just think “maybe there’s a pattern that would be useful here. I’ll scan my list of patterns taped to my monitor. Builder would maybe work? I’ll double check what that means”.

This is 1. Way easier 2. Consistent across many patterns. Instead of learning to recognize pattern A, B, C and so on, you just get a vague feeling of “pattern”.

This (obviously) doesn’t just apply to patterns. Super useful for web frameworks. List the 10 places you could put behavior (model, field, view, etc) and choose consciously. Same for data structures in algorithms and even all sorts of non-programming stuff.

I know python (3) is an object oriented programming language by design but none the less I was wondering if some “functional” pythonista’s had recommendations on some good “less (oop) is more” patterns to share in here?

https://github.com/mpkocher/Functional-Programming-Technique...

I covered a few core concepts (e.g., functions as first-class citizens, closures, partial application, etc...) and added a few real world examples of using a functional centric design. The text/format has some rough edges, but overall I think the text is useful for internalizing how to leverage a functional-ish approach.

Other resources.

https://github.com/sfermigier/awesome-functional-python

It's hard to completely get away from objects in Python, but I don't think that needs to be an objective to get away from OOP patterns. My suggestions would be:

- Don't use classes for information. Use regular data structures (dict, list, tuple, set) and use functions to manipulate them. If you really want/need something like a struct, use a dataclass.

- The itertools module has a bunch of useful constructs for creating and combining iterators. Take advantage of Python's iterators and construct your own when necessary.

- List comprehensions and generator expressions are great for clear and concise data transformations

- You're going to have to deal with state somewhere in your program. That's fine, some state is necessary to do real work. Don't feel bad about creating classes for things like context managers (e.g. for a database connection) so you can use it in a with statement and it will handle the setup and teardown logic for you. That way your business logic stays concise and Pythonic since those classes can be in separate modules and imported as necessary. I've found this 'functions in the front, classes in the back' approach to writing Python useful as it lets me think about what my program does in a functional manner without denying myself the ergonomics of Python's class-based ecosystem.

Can you point to any public codebases which illustrate this approach in Python? I would love to see some of these principles in action.

> Don't use classes for information. Use regular data structures (dict, list, tuple, set) and use functions to manipulate them.

One problem is many of these datastructures are easy to mutate. When I need to ensure immutability to a dictionary, for instance - quite often - I can't avoid wrapping it in a class and setting up an interface to lock down mutation.

If Python had a way to declare immutable datastructure, similarly to Clojure, it would make life a lot easier...

Frozen dataclass instances (https://docs.python.org/3/library/dataclasses.html#frozen-in...) might be of help.

Wasn't aware of this feature in dataclasses, that can be helpful in a few cases, thanks for pointing out

> class types.MappingProxyType(mapping)

> Read-only proxy of a mapping. It provides a dynamic view on the mapping’s entries, which means that when the mapping changes, the view reflects these changes.

https://docs.python.org/3/library/types.html#types.MappingPr...

This can be helpful in some cases, but it's precisely what I sometimes need to avoid, which is protecting a dictionary and having changes affecting only its own scope.

If function A passes a dict to function B, I would like:

1) Function A to keep the dict intact while function B can manipulate its content;

2) Function A can change the dict after passing to B, and B still keep the original copy, unaware of A's later changes.

One way of doing this is by deep copying dictionaries around, but it can easily become a big performance issue.

It sounds like what you really want is an efficient way of copying dictionaries, perhaps copy on write, something like the cowdict [1] project.

[1]: https://pypi.org/project/cowdict/

> Don't use classes for information. Use regular data structures (dict, list, tuple, set) and use functions to manipulate them.

I've written a lot of functionalish Python, and I've found that this is a consistent way to breed code that's hard to understand. Instead, I would default to (usually frozen) dataclasses to represent data—they're functional records that also let your code reflect the concepts that make sense for whatever you're doing. A list of dictionaries of strings looks like any other list of dictionaries of strings; a sequence of User objects has clear semantics and, as a benefit, is harder to mutate in unexpected ways.

Wouldn't TypedDict come in handy here? This way we can kinda mimick a structural typing system. The only disadvantage I ran across here is that TypedDict can't handle recursive fields, unlike dataclasses. Otherwise, it seems to me like a more Pythonic choice, reminding of Clojure

Scrap all the factories. The class name is also a function and a constructor. Just pass it around as parameter, and bypass all the factory stuff.

Use named parameters and bypass the builder pattern.

Store operations in objects/lists. Bypass the strategy patterns with this knowledge. Did you know myobject.mymethod is a function, and you can do x = myobject.mymethod, and then x()? Store methods in lists, pass them as parameters, partially-apply parameters to them.

Generators are more than what the iterator pattern will never be.

Learn Go and spend some time in its world, then come back to Python IMHO. You'll start to appreciate the subset of Python that's just functions at the module and class level and ignore all the complexities of metaprogramming, class hierarchies, etc. You'll find a lot of folks using Go are coming from Python and burnt out on big, complex codebases there--the simplicity and "less is more" is a breath of fresh air to reapply to Python.

This is great. I always thought that GoF patterns is largely obsolete in 2022, it would be nice if the author agreed and has a list of GoF patterns that really dont make sense any more. Builder was one I thought should be included but OP seems to think its relevant (though different).

In terms of obsolete GoF patterns, Flyweight is my nomination. It's not that it's useless, is just that in an era of 64 MB of RAM, it makes sense that people might need to know about it, but in an era of 64 GB of RAM the need is diminished to the point that only the people who have those problems need to worry about it.

And if you do have those problems, it isn't even necessarily what you would reach for first. Though it's at least on the list.

I think I've used flyweight in exactly one production system, and it was a trading system. Lots of transactions (easily tens or hundred of millions in a given day in a single process), lots of duplicated instrument data. Perfect use case for flyweight. But, yeah. It's a fairly rare use case. Good to know about, but not necessarily one you need to keep fresh in mind, ready for use.

Luciano Ramalho in his book "Fluent Python" mentioned that some (most?) of the GoF patterns are unneeded in a dynamic language like Python. I see [0] he is favorable to this site's author, so it is probably worth a look.

[0] https://twitter.com/ramalhoorg/status/960746929025142784?s=2...

How did you happen to see that? That tweet is perfectly relevant, it's byond uncanny that you had it at the ready.

Coincidence. I was trying to find the reference to Ramalho's comment about GoF and found his comment in a random search.

Interesting, thanks for sharing. Is the opposite also true, that when a language is statically typed, design patterns are likely to be needed?

It depends on the language expresiveness, not on it being static/dynamic types. Dynamic languages tend to have less limitations to expresiveness (no typechecker in the middle) and then you have programming languages with static types and decent typecheckers that don't require design patterns because they are still very expressive.

I found the quote. He says Peter Norvig wrote that "16 out of 23 patterns have qualitatively simpler implementation in Lisp or Dylan than in C++ for at least some uses of each pattern" and that Python shares some of the dynamic features of these languages such as first-class functions. I wonder if making GoF patterns simpler or even "invisible" in C++ using functions or callable-objects is implicitly not allowed because of the static typing of the language or if it just wasn't considered by the authors.

Most of the design patterns exist because of limitations in programming languages, especially java/c++/c#. The strategy pattern disappears when you can store functions in data structures. The abstract factory pattern disappears when you can pass functions as parameters. The builder pattern disappears when your progamming language has named parameters. The observer pattern is callbacks with additional steps. The mediator pattern disappears with support on variants of functional reactive programming. The iterator pattern disappears by using your favourite version of generators (or just go hardcore like Haskell and do foldable/traversable).

IMO GoF has been obsolete for the last 20 years (50 years if you consider that languages like ML, Lisp and friends already existed then).

Storing functions in datastructures with the purpose of applying a different logic depending on context/configuration but with the same goal _is_ the strategy pattern. It doesn't matter if you call it like that or not.

Do we have a special name for storing numbers in data structures?

Do we have special names for storing strings in data structures?

Do we have special names for storing lists in data structures?

Do we have special names for storing source code as strings in data structures?

Do we have special names for storing abstract syntax trees as data structures?

Do we have special names for storing compiled code as data structures?

Do we have special names for storing pairs of compiled code and related memory in data structures?

Why do we need special names for storing functions in data structures?

When I filter the list of numbers it's not a strategy pattern. Neither is when I filter the string list, or the AST list, or the compiled code+closure list. But, somehow, if it's a function list then it's a strategy pattern.

Free your mind of unnecessary labels.

> Do we have a special name for storing numbers in data structures?

Yes, we do. For instance, if I store numbers that describe a limit, I'll call it a limit. And it will give a hint how it is used. Do I call all numbers in a data structure limit? Of course not - the same is true for functions in datastructures.

> Why do we need special names for storing functions in data structures?

Well, it is helpful. There are many reasons to store functions in datastructures. For example to do mappings, to cache/memoize them or even to create interpretable datastructures, using free monads.

But the strategy pattern describes the situation where there are multiple functions and they all serve the same purpose and have the same interface (and thus are interchangeble) and usually one is chosen and applied - but it is expected to be different ones, depending on the context.

Do we really need a name for that or use it in code? I don't know. But that's not the my point - which is that the concept doesn't go away just because it is easier to use it.

> When I filter the list of numbers it's not a strategy pattern. Neither is when I filter the string list, or the AST list, or the compiled code+closure list.

Sure, if you don't change the way you filter then it's not a strategy pattern.

> But, somehow, if it's a function list then it's a strategy pattern.

Now you have lost me. Maybe we disagree because we are talking about a different thing?. A list of functions does not make a strategy pattern. And very often, strategies are implemented using inheritance (or composition) without any lists being involved. At least that's how I remeber the pattern when I saw it (in the pure functional programming world we indeed rarely call it like that) and how it's written in the GoF book.

You have a bunch of functions in a list. You choose one of these functions via some criteria, and run it with some provided parameters.

Why it is so different to having a bunch of numbers in a list and choosing one of these numbers via some criteria, then doing one of the allowed operations for that number?

Bear in mind with the function I'm doing exactly the same: do one of the allowed operations for it (invoke the function).

Different functions can expect different arguments. For numbers, they are all the same. That's a difference!

Well, you can argue that there are also different numbers and you can do devision only by some of them. But in fact, if you were to separate them and wrap them to do things differently, then this would get closer to the strategy pattern.

> Most of the design patterns exist because of limitations in programming languages, especially java/c++/c#.

No. The implementation of some DPs is trivial in some languages, but the DPs don't disappear. The idea of DPs primarily as boilerplate implementation recipes rather than solution concepts with rationales comes from the limitations of Java, etc.,. not the DPs themselves.

Builder patterns work very well for generation of templates. I use them a fair amount when dealing with Amazon's CDK as it helps with fixing a lot of readability issues.

I see people claim that GoF patterns are obsolete, and that builder is one of the ones that are old hat, but for making configurations readable it can be the best solution.

What's the advantage of:

t = TemplateBuilder() .withFoo(foo) .withBar(bar) .excludeBaz() .build()

over:

t = Template(foo=foo, bar=bar, baz=none)

I agree there's not much point using this kind of simple builder if you have named function parameters, but the GoF builder pattern has a couple of additional features which can still be useful. In the GoF examples the construction process is more complicated (maybe the order in which construction steps take place is significant, and maybe parts of the construction might be conditional), and the same construction algorithm can be used to produce different types of object. If I remember correctly, the main example in the GoF book is a builder for creating documents by adding a title, subtitle, paragraphs, etc, and the document being constructed might be a UI representation, an RTF file, HTML, etc.

> maybe the order in which construction steps take place is significant

That is just horrible...

> maybe parts of the construction might be conditional

No builder pattern needed for that, just use optional arguments and/or an option/nullable type.

I think the builder pattern is more useful when you have sets of parameters that make sense together, but may be mutually exclusive with others. A more compelling example would be an IP address builder that supported different versions.

    ipv4 = IPAddress().v4("127.0.0.1").build()
    ipv6 = IPAddress().v6("::1").build()

Trying to perform something like this in Python with merely a constructor on an IPAddress type would lead to a not-so-obvious interface (such as use X set of named args to construct a v4 address or use Y set of named args to construct a v6 address, but using X & Y in the same constructor is a runtime error). This is my contrived example for where the builder pattern might make sense.

With your example, I don't see any compelling advantage to use the builder pattern. In fact, I'd argue as-is, the builder is unnecessary, is more verbose, adds additional complexity, and you should just use the constructor with named args instead.

The point behind the builder pattern is to perform nontrivial initialization that is hard/difficult/impossible to perform in a normal constructor a la C++, C#, Java, etc. If you're not performing nontrivial initialization, you've just introduced extra verbosity and complexity for no gain, so don't use the pattern then. This is, of course, just my opinion.

For example, if you see code that has a lot of overloaded constructors followed by an `init` function that must be called after construction, or code with a lot of overloaded `init`-type functions, only one of which shall be called; the builder pattern might make sense there.

    ipv4 = IPAddress.v4("127.0.0.1")
    ipv6 = IPAddress.v6("::1")

KISS.

You do not have to use the same class for V4 or V6.

I'm well aware. I chose the example to somewhat mirror what happens at the native posix level with addresses. It's the same structure, but can only be built in specific ways. Probably should have actual stated something to take effect.

I was talking about complex configurations with repetitive, not extremely basic named parameters. Think about a loadbalancer or an API Gateway that is going to loop through and generate several lambdas and their security groups.

I'll post an example in a bit of what I mean, but using a builder lets me use functions to break down the template into readable pieces.

And I'm not saying that it is impossible, I'm talking about readability.

I just finished the first chapter, and so far I think it's really worth reading.

After years of Python development, I encountered all the same problems and reached all the same conclusions. I wish I read this at the beginning of my career.

Most of the patterns I haven't seen in practice. Aiohttp, Django, FastAPI, PyTest. They have a blend between monkey patching and metaclasses. It's strange not seeing them more deeply on that list. It kind it touches what you will study in pre academia for Java or C++ workshops courses and not what you will see applied in the industry.

In my experience, developers that reach for design patterns recently read a design patterns book and are desperate to apply them to problems, instead of trying to figure out the best solution to the problem. Design patterns (imo) should be used to get a feel for the design space that you can choose from, but it’s far from a total enumeration.

Unrelated soundbite by Peter Norvig: "Design patterns are bug reports against your programming language.".

Nice! I just looked at it yesterday when I was collecting resources on architecture of WebApps in python. I found that Refactoring guru also has python examples for them pattern section.

Well, this post turned out to be a gem. Loved it.

This is my favorite ressource on the subject. I recommend it as often as possible.

They're coming for python now. Want to reverse a string? You simply need an abstract-factory-decorator-observer.

The 'enterprise python' battle begins. Good luck.

> Python Design Patterns

My brain puts it next to: globe of China, Jewish Olympic games (real thing).

Reply on news.ycombinator.com