Advanced Python Features

Hey, I really enjoyed reading your post. It's straight to the point, the list is very rich and well illustrated with examples. I have been coding in python for the last 20 years and I nowadays rarely like reading these posts as much as I did for yours. Kudos.

I’ve coded python for 15 years and I’ve not seen many of these, so I’d consider them reasonably advanced personally.

One I think you missed is getters and setters on attributes!

I appreciate you writing the post and I enjoyed reading it.

Congratulations for ending up on the front page! (I hope the server hosting your blog is okay!)

The tricks are more advanced than what I've seen in most people's code at some jobs before! The post is refreshing because it sits in the neglected middle ground between the tons of Python-for-beginners content out there and the arcane "I'm a core developer writing about very specific things" content.

Very good job. I bet most python developers will learn something here. Python has changed a lot; for the better, judging by your examples.

I think the background of the post is very well explained on the blog.

But I’m surprised to read you went from idea to posting each day. I had expected maybe a week of preparation before starting execution.

Regardless it’s a great post, regardless of wether one think there is too much typing or too many new features in modern Python or one is a lover of niche solutions to shorten code.

There are plenty of language features which let you re-use or build your own guardrails in python.

I do with the concept of truthiness and falsiness would be taken out and shot though. It's been responsible for far too many nasty bugs IME and it only cuts out a few extra characters. Not a great trade off.

Instead of comfort I feel constantly annoyed that the language gives me so little that I have to produce more lines and very little in the way of convenience. It's a language that benefits a lot from AI completion, at least. Yeah, old code still looks the same as new code but that is due to a lack of progress on the language though.

I write an unfortunate amount of Go code.. because Go supports 2(?) of the features in this post (structural typing and generics, sort of, if you're feeling generous).

For example, circular imports aren't supported in Go at all. I run into this from time to time even if using an interface sometimes consts or other types are defined in the package as well and the whole thing has to be refactored. No way around it, has to be done.

Circular imports aren't encouraged in Python but Python never leaves you without options. Code can import types only while type checking or move imports out of the module level are quick workarounds. (I hope for typescripts `import type` someday.) That's what gives me a "comfort" feeling in Python, knowing that the language isn't likely to force me to work around the language design.

pick up a book on how to program pythonically and it will feel better, I promise. A lot of people program in python like they do in c++ or rust or javascript and it doesn't feel "right" to them. If you follow the pythonistas take I think you'll appreciate it better.

Extra stuff like type checking and annotations are definitely not the reason why python became that popular and widely adapted and used, but it certainly doesn't hurt to add and use them.

To be clear, I'm not expecting people to start adding generics to their quick hacked together Python scripts (in fact please don't do that). Instead, if you're building a library or maintaining a larger Python codebase, a lot of these start becoming very useful. A lot of the typing features I mentioned are already used by Python under the hood, and that a lot of Python developers just take for granted.

Case in point, the python-opencv (https://github.com/opencv/opencv-python) library has basically no types and it's an absolute pain to work with.

BTW thats a really good SO thread, thanks for linking it!

I love Guido's balance. You don't need type checking and annotations if you don't want to use them. They are nice when you're playing in more complex systems, however, completely optional.

> There is a reason why python become that popular and widely adapted and used, and it is not the extra layers of type checking, annotations and the likes.

Python got popular and widely adopted for the same reason PHP did: 1) it was readily available everywhere and 2) it's an easy language for beginners to pick up and get real applications working quickly.

But it turns out that the language features desirable for a quick prototype done by your fresh-out-of-college founding engineer aren't the same as the language features desirable for your team of 100 engineers trying to work together on the same codebase.

Python is at an odd intersection where it's used by everything from large teams building robust backend applications (particularly in the data/ML space), to data scientists hacking on scripts interactively in Jupyter, to ops people writing deployment scripts. You don't need type checking and annotations if you're writing a few one-off scripts, but you'd be crazy to not take advantage of them for a larger application.

A very good piece, that I enjoyed. But, I do get the impression that many of these more esoteric features, when reading a code-base, actually end up obscuring the developer's intention, and impair understanding. Without very competent IDE tooling, trying to work out what's happening in a piece of code is extremely difficult. It's turtles all the way down, and there's a lot more turtles now. A number of the new features, appreciated by many for good reason, tend to work against the Zen of Python which at some level explained the meteoric rise of Python as a bona fide implementation language over the last 30 years - simple, obvious, effective and practical.

please help me understand.

Up to now python emphasizes to be and remain the dynamically typed monkey patch happy core language it is, with the completely volunteer option to provide type hints and use them to your advantage as you see fit.

So you can hack away and monkey patch to your heart's content and nothing is taken from you. no rust borrow checker. no need to use a type checker. ducks everywhere.

and I'm not aware of features that have to be used, aka incompatible changes to the core language.

So what is the critique, exactly?

I mean it's completely optional. Even if you are to use python as a glue language, with absolutely 0 typing annotations on your side, it's still very very useful to have them in the libraries that you inevitably end up using with python. It's not even like it requires an additional build step, so it's really just free.

I have used Python for a long time for glue scripts, build stuff, test systems etc. and I've always thought it was great. I didn't like using it for larger applications though - when I did, I usually got bitten at some point by some effect of the dynamism (regardless of testing). At scale things became more unworkable.

With the type system my opinion on that has changed. We have a pretty large tooling codebase with protocol stacks, GUI, test automation etc. and it's all very maintainable with the type checking.

> did you know __init__.py is optional nowadays?

It has an effect, and is usually worth including anyway. I used to omit it by default; now I include it by default. Also, you say "nowadays" but it's been almost 13 years now (https://peps.python.org/pep-0420/).

> since 3.13 there is a @deprecated decorator that does what you think it does

Nice find. Probably worth mentioning it comes from the `warnings` standard library.

> the time package has functions for monotonic clocks and others not just time()

There's quite a bit in there, but I question how many people need it.

Anyway, it's always surprising to me how when other people make these lists, such a large fraction is taken up by tricks with type annotations. I was skeptical of the functionality when the `typing` standard library was introduced; I've only grown more and more wary of it, even as people continue to insist to me that it's somehow necessary.

> did you know __init__.py is optional nowadays?

It's not optional. Omitting it gets you a namespace package, which is probably not what you want.

> TypeVar supports binding to enforce subtypes: "TypeVar['T', bound=X]",

Using the new generics syntax you mentioned above you can now do:

   def method[T: X](...)

+1 to itertools especially. Absurdly powerful, and the recipes at the bottom of the doc page are terrific.

The only problem I’ve found is that for interviews, people often aren’t familiar with it, which can lead to you solving whatever puzzle they had in far less time than they intended, and without manually building whatever logic it was they assumed you would need.

I disagree. Adding "fancy stuff" by definition does not remove what made it attractive to you in the first place. You even acknowledge this in your second sentence. Something you don't know of is not capable to influence your opinion in any way. And when you know of it -- and dislike it -- just keep doing it the way you did before knowing it.

Lua is like this only moreso, and there's little fear of "progress" as large parts of the community have stuck with Lua5.1.

Of course, Lua is batteries-not-included so there may be the problem of "progress" in external libraries; in practice things like Penlight barely change though.

Similarly with 9.2, assignment using a walrus operator will also fail if the value is 0 (or anything falsy: https://docs.python.org/3/library/stdtypes.html#truth-value-...)

Simple doesn’t make something basic or beginner level.

> 5. Protocols: As a Typescript guy, this seems very cosy & familiar. And not very Pythonic. I'm not sure how to feel.

Isn't it super pythonic? One of the first things you learn about Python is that "everything is duck typed", but then the type system is primarily nominally typed. It seems like Protocols should have been there from the start, like Typescript interfaces.

And you can use * for the reverse (every parameter from here on needs to be keyword-only).

https://docs.python.org/3.12/reference/compound_stmts.html#f...

Reference: https://peps.python.org/pep-0570/

I would say that the main benefit of for/else is readability and ease of writing since nothing needs to be front-loaded. (The downside is an arguably redundant addition to the basic syntax, of course.)

If you replace the assignment in the else clause with something side-effectful it does make sense. But even then it harms the readability of the code to such a ridiculous extent. I've never not regretted adding an else clause to a for loop. Any cognitive benefits from Python's approach to for loops closely mirroring natural language just go out the window.

I do like the else clause with for loops. However most people are not familiar with it, and also `else:` as a keyword is confusing. I always remember it as `no break:`.

What about this?

    for server in servers:
        if server.check_availability():
            primary_server = server  
            break
    else:
        logger.warning("Cannot find a valid server")  # <---
        primary_server = backup_server
    deploy_application(primary_server)

This kind of search can be done a variety of different ways, and is worth abstracting, e.g.:

  def first(candidates, predicate, default):
      try:
          return next(c for c in candidates if predicate(c))
      except StopIteration:
          return default

  deploy_application(first(servers, Server.check_availability, backup_server))

If the team is familiar with Functional Programming concepts, I'd actually suggest

  available_servers = (server for server in servers if server.check_availability())
  primary_server = next(available_servers, backup_server)
  deploy_application(primary_server)

which IMHO better conveys the intent to the reader (i.e. focus on "what to do" over "how to do it")

The shorter version that saves the else: line assigns primary_server twice: it is OK for an assignment but a potential problem if doing something less trivial, and very ugly style in any case.

> Does modern, production-level python code use types everywhere?

I don't know about code inside companies, but most new open source projects I encounter use typing. Many old ones have been converted too.

> Is duck typing frowned upon?

No. You can use Protocols to define what shape you expect your duck to be. There's some discussion about whether you should use abstract classes or protocols, though.

Structural typing is a static equivalent of dynamic duck typing. It is compatible with type checkers. Protocols mentioned in the article work without inheritance. A duck doesn't need to know that there is SupportsQuack protocol. If it quacks, it passes the type checks.

Same experience here, Python’s typing experience is awful compared to TypeScript, even post-3.12. Mypy’s type inference is so dumb you have to write arguments like `i: int = 0`; `TypedDict`s seems promisable at first and then end up as a nightmare where you have to `cast` everything. I miss TypeScript’s `unknown` as well.

Hey kiddo… did you ever try something nastier? I’ve got something that will blow your mind and you’ll keep coming back

You don’t know but you are addicted to types

Come to the light - Haskell!

One of the most frustrating bugs I had encountered was when I was using memory mapped CSC format sparse arrays.

I needed the array indices to be int64 and specified them as such during initialization.

Downstreams, however, it would look at the actual index values and dynamically cast them to int32 if it judged there would be no loss in precision. This would completely screw up the roundtrip through a module implemented in C.

Being an intermittent bug it was quite a hell.

and maybe https://more-itertools.readthedocs.io/en/stable/

never used it but it seems worth a try

The quality of documentation is a known, highlighted concern in the community. There is particular interest in adopting Diataxis principles to improve documentation, both for Python itself and for packaging (see e.g. the user guide at https://packaging.python.org/en/latest/ , or the PyPA main site https://www.pypa.io/en/latest/ ).

If you want to help, there's a section on the Python forum (https://discuss.python.org/c/documentation/26) and a Discord server, and issues with documentation can also be reported on the main Python GitHub issue tracker (https://github.com/python/cpython/labels/docs).

It's more complex that decade ago, but still a relatively simple language. I can understand the article without much effort, while I scratch my head really hard when read about advance feature of Typescript or Scala.

Just like Microsoft maintains the Typescript handbook, I think the Python Foundation needs a "Idiomatic Modern Python Handbook" which always shows the most modern and idiomatic way to use Python features.

Python has always been quite complex, people get deceived by how simple Hello World looks in Python.

Using it since version 1.6.

I would recommend reading the article example once more. Going from the example, without the overload, the function would return a union type which means any time you use the function, you have to put a type check to the result to know if the output is a list or not. With overload, as soon as an argument to the function is a certain type, the output is determined, so you won't need to type check.

In other languages it's an alternate way to do default or optional values in function arguments. Python instead usually uses kwargs for this.

That's what you get when you add a typing system and come full circle to doing OOP like Java.

It's an implementation of "ad-hoc polymorphism", where, for example, it may make sense to "add" (+) together numbers of various types: integers, floating points, rational numbers, etc.

Thus the (+) operator for addition is "overridden" or "polymorphic" in the types of numbers that can be added together.

The argument for having a polymorphic signature rather than just multiple separate "monomorphic" functions is similar to that for "generics," otherwise known as "parametric polymorphism": why not just have a function `forEachInt` for iterating over lists of ints, a separate function `forEachChar` for iterating over lists of characters, and so on?

Higher levels of abstraction and generality, less boilerplate and coupling to any particular choice of data structure or implementation.

You could of course go the route of golang which indeed just had you write "monomorphized" versions of everything. Several years later generics were added to the language.

Alternatively, you throw everything out and never have to worry about typing or polymorphism, at the cost of static safety.

The only newish thing on this list appears to be structural pattern matching. Other than that it's all typing stuff which is all optional.

Nothing stops you from simply not using features, and even having CI checks that forbid them. Not sure why you’d need a fork.

In that example I gave, I totally agree with you. One area where I find walrus operators kinda useful is with dealing with iterators. For example,

'''

iterable = iter(thing)

while val := next(iterable, None): print(val)

'''

is a lot cleaner in my opinion compared to

'''

iterable = iter(thing)

val = next(iterable, None) while val is not None: print(val) val = next(iterable, None)

'''

Reason why I did not use this example outright was because I wasn't sure if people were familiar with the iter api, so I just chose a simpler example for the blog.

I disagree.

First of all, it takes a minute to search "python :=", and the construct itself is pretty simple. It's been part of the language since 2018[0]. I don't think "not knowing the language" is a good reason to avoid it.

Second, the walrus operator limits the variable's scope to the conditional, which can reduce certain bugs. It also makes some scenarios (like if/elif chains) clearer.

I recommend checking out the PEP for some real-world examples.

[0] https://peps.python.org/pep-0572/

Edit: my point about walrus scoping is incorrect. The new variable is function-scoped.

Couldn't you extend this line of thinking to any language-specific syntax on any programming language?

Don't use `match`, macros, lifetimes, ... in rust, someone coming from another language without them might not get what it means. Instead write the equivalent C-looking code and don't take advantage of any rust specific things.

Don't use lisp, someone coming from another language might not be able to read it.

Etc..

At one point if you write code and want to be productive, you need to accept that maybe someone that is not familiar with the language you're using _might_ have to look up syntax to understand what's going on.

> which is cool if you like static typing.

I don't believe any reasonable person would call Python statically typed, it just now has a pathway though which one can send additional documentation, with all of its caveats

  python3.13 -c '
  foo: dict[str, list] = "lol"
  print(foo.keys())
  '

And, yes, I am aware of the chorus of "just sprinkler more linters"

It’s never a good idea. I use `__all__` to explicitely list my exports in libraries, so that when one write `from mylib import `, the IDE auto-completes only the public classes and functions.

I don’t really see "smart tricks" here, just features of the language that you may not be aware of. I use most of them in my daily work without thinking about it: `@overload` and other typing features are useful to document the code, prevent mistakes, and help IDEs generate more helpful suggestions; keyword-only arguments force users to pass options as keywords, so you avoid calls like `foo(x, z, z, true, true, false, true)` and ensure the code is more explicit; for-else can sometimes clarify some complicated code; the walrus operator, short-circuiting with `or`, and chaining operators are just normal Python.

I strongly agree. The walrus operator ... argh.

When I code, I try to make everything I write not clever. I am not saving a byte here and there because I am not typing in a program from a magazine in the 1980s (I was there, I am not going back). No code golf. I did my time on the Timex-Sinclair with its miserable two kilobytes of memory and reserved keywords as special-function strokes at a character each.

Each line should do one thing, in general, and it ought to be obvious. Cleverness is held in reserve for when it is truly needed, namely data structures and algorithms.

One of my accomplishments which seems to have mattered only to me is when my apartment-finding/renting system, written entirely in Perl, was transferred into the hands of students, new programming students. Perl is famously "write-once, read-never" and seems to have a culture favoring code golf and executable line noise. Still, the students got back to me and told me how easily-ported everything was, because I had done just one thing per line, avoided $_ and other such shortcuts, and other practices. They were very happy to take it over because I had avoided being cryptic and terse.

I don't fully disagree, but the yield context manager approach is very common when writing test fixtures, you do:

  # setup  
  yield resource. 
  # teardown

Not that it makes it any less magical, but at least it's a consistent python pattern

The simple contextlib.contextmanager example doesn't really sell the benefits.

The main one is that it makes error handling and clean-up simpler, because you can just wrap the yield with a normal try/catch/finally, whereas to do this with __enter__ and __exit__ you have to work out what to do with the exception information in __exit__, which is easy to get wrong:

https://docs.python.org/3/reference/datamodel.html#object.__...

Suppressing or passing the exception is also mysterious, whereas with contextlib you just raise it as normal.

Another is that it makes managing state more obvious. If data is passed into the context manager, and needs to be saved between __enter__ and __exit__, that ends up in instance variables, whereas with contextlib you just use function parameters and local variables.

Finally, it makes it much easier to use other context managers, which also makes it look more like normal code.

Here's a more real-world-like example in both styles:

https://gist.github.com/tomjnixon/e84c9254ab6d00542a22b7d799...

I think the first is much more obvious.

You can describe it in english as "open a file, then try to write a header, run the code inside the with statement, write a footer, and if anything fails truncate the file and pass the exception to the caller". This maps exactly to the lines in the contextlib version, whereas this logic is all spread out in the other one.

It's also more correct, as the file will be closed if any operation on it fails -- you'd need to add two more try/catch/finally blocks to the second example to make it as good.

> This is all fun and games unless you have to debug someone elses code and they use a new feature that you didnt know about.

What’s wrong about learning things by looking at code from more experienced people?

You underestimate the social complexity of a simplified language. Everyone would disagree about the design (what is worth including because it is "simple" or necessary?), everyone would avoid using an handicapped language where features they need are gratuitously missing, and nobody would care about it enough to work on implementation and maintenance.

As a UD, I didn't know what a context manager was before reading this article, so you are not really far from the truth, tbh.

Yes, please. Given I'm forced to use Python, I'd welcome any "compile time" tools I can get. These days, it's using pyright with strict mode, which is pretty good, but there's still a long way to go.

Yeah I only worked professionally with python for about 6 months and I knew about most of them.