What new Python features are the most useful for you? | Hacker News Reader

What new Python features are the most useful for you?

42 pointsartimis42 comments

In recent years Python ecosystem evolved significantly. From using requirements.txt to dependency management tools like Poetry, from simple typing to Protocols [1] and Variadic Generics [2]. We've got walrus operator, match case statement, and finally even faster Python. More and more low-level libraries are being written in Rust [3].

What new features, tools, libraries (already implemented or being worked on) are the most useful for your work with Python? What are you the most excited about?

[1] https://peps.python.org/pep-0544/ [2] https://peps.python.org/pep-0646/ [3] https://github.com/pydantic/pydantic-core

View on HN

Just getting back on the latest open source Python ecosystem after a long stint of mostly writing Python in a corporate mono-repo context with bespoke tooling.

So far I like:

  - black autoformats my code swiftly
  - ruff is fast and replaces multiple python linters
  - pip-tools lets me manage multiple dependency specifications (local, prod, test) more sanely than manually syncing `pip freeze` with multiple requirements files. 
  - f-strings are nice improvement over .format calls, % string interpolation, string.Templates, etc.
  - I appreciate pattern matching and case classes in Scala. I expect to like them  using dataclasses and pattern matching Python

I use Type hints and using mypy strictly.

Huge fan of dataclasses as well.

Most of the python I write is functions / functional + data classes with type hinted everything and it seems to work nicely and make writing tests for functions very easy.

Using black, isort, mypy, pylint, pytest, makes every project simpler.

- einops (probably by far the most useful Python dsl for me as of late, esp. if you deal a lot with tensors)

- numpy __array__ interface standardization (easy compatibility across a lot of numerical libraries, e.g. torch, awkward-array, zarr, jax.numpy, dask, xarray)

- the various jit compilers (e.g. numba, jax.jit, torch.compile)

- hw-accelerated functional programming style (e.g. functorch.vmap)

- pybind11 for easy af bindings to c++

- named dimensions for tensors

- pandas-compatible ecosystem (eg seaborn)

- pytrees

- pyenv

- mamba

What I really need, though, is a realtime successor to matplotlib with a much simpler interface/dsl. Haven't looked into plotly much but probably will try it soonish

Matplotlib is really killing me. I never have learned it properly and switch between methods to address the same things in Matplotlib. Every time I have to plot something I have to Google how to do it or I have a template ready already. In my opinion it’s the most annoying library I have to use.

In situations where matplotlib is the lingua franca, like notebooks, I've opted for seaborn. You'll still have to deal with matplotlib's complex nonsense, but substantially less of it.

Altair/Vega ( https://altair-viz.github.io/ ) is really nice to use, although headless rendering outside of a notebook environment is a bit of a pain

I‘m not working inside of notebooks too much to be honest. I was doing so a while back but working in a modular fashion with other scripts is a pain so I went back to vanilla Python files in VS Code a while back. I’ll still take a look, thanks for the heads up!

Maybe not practical for your workflow, but making interactive plots in Pluto.jl is so much nicer than hacking around in matplotlib

If it was easier to use virtual environments with PyCall I think I’d abandon matplotlib entirely.

I had checked out plotly some years ago:

plot.ly now has an API:

https://jugad2.blogspot.com/2013/07/plotly-now-has-api.html

plotnine is an API based on Grammar on Graphics. I use it as an replacement for matplotlib.

I find myself using the walrus operator a lot nowadays, and find I often miss it when writing JS/TS (previously I mostly found myself craving arrow functions and the far superior type system from TS when writing Python).

In the 15 years I have been developing in python (and C++ at the same time, daily), I can't remember a single bug I ever had on account of an incorrect type.

I do like the := to save a few lines here and there.

3.10 does seem to run faster but maybe that is my imagination.

I wish more people would focus on writing good python.

3.11 runs a bunch faster.

what do you attribute to never finding a bug due to incorrect type? and what is good python vs bad python?

> what do you attribute to never finding a bug due to incorrect type?

If the function/method expects an int, string or complex type and I accidentally call it with the wrong type, resulting in an error, such as "can't add 1 to a string", or "can't access access that method because a string was passed".

>and what is good python vs bad python?

This is kinda of subjective but, for a start, don't just start at the type and do stuff until the bottom like a shell script. This is by far how I see most python scripts. All the other normal things. Don't repeat yourself (in moderation, a few times is OK). Keep it simple.

I guess I meant what do you do to prevent that from happening.

That is my point. I don't have any tools to prevent it from happening. I just don't do it when I write the code. If I have and my code is tested it comes up very quickly.

Us older people tend to keep the whole workflow as simple as possible. That is why I love python. There is very little overhead to just using it.

If you tend to make this mistake a lot it might make sense to add some tooling. I don't need it. If I wanted strict typing I would probably just use C++.

I think you're either misrepresenting reality or stretching the truth.

For example, if you're redefining bug as "bug that made it past testing" that's one thing.

But I don't buy anyone writing Python daily for 15 years hasn't seen the term "TypeError" which is what your previous two comments were implying.

OK, it's 100% likely I saw it, just so rarely I can't remember.

Testing. Over that long time I have used many test frameworks. My normal workflow is to try and exercise all the code I write as I go.

On the other hand I am 100% pedantic about using 'schema' (https://pypi.org/project/schema/) to validate all the types, structure and domains when I read json.

Nearly the same here. The only type error I make a little bit more than I would like, is expecting an iterable and passing a string instead, because I forgot parentheses around arguments.

But I made way more type errors when programming in C/++.

Why do you prefer schema to (fast)jsonschema?

'schema' is more a schema enhanced by python than just a checker. It can convert types, set defaults so after 'Box' I can simple access config with a.b.c. It may be subtle but it is much more powerful than it looks.

There are many of these things. Maybe some are better but I like this one.

So you use some type of testing methodology for your code?

You do get type errors, but you catch them with tests? Type hinting catches them without having to write tests.

I live on the opposite planet. basically every bug that I have ever had in python over my 15 years of using it has had to do with a type issue. So the increased use of types and python and ability to express type concepts in python is the thing that I'm most excited about.

it is not necessarily about type mismatch bugs per se, but more like without type information in the code I will misunderstand what kind of structure is passed in to a function, how to decompose it, what members the data has, what functions does it have, how do you spell them correctly, what elements to expect in dictionaries and what kinds of values do they have, and how can I make massive changes to code without knowing all those answers and expect it all to still work.

type annotations are solving a lot of those issues for me nicely, and the number of things that it doesn't solve well is decreasing with each version and the new features they are adding

- Pydantic: basically it might as well be part of the standard library for me - I'm excited by polars as a potential pandas replacement - Speaking of pandas i really like pandera and how it integrates with both pandas and pydantic

Why use Pydantic instead of dataclasses?

Pydantic can do validation. There’s also the related Typer, which builds command line interfaces from the type annotations on your entry point

You might also want to look at https://www.attrs.org, which is basically what dataclasses were supposed to be.

The most useful feature to me would be for Python to have long-lived standard editions of the language the way other programming languages do.

For example, clang and gcc can still compile C89 programs.

Good to have Possessive quantifiers and Atomic groups added to the `re` module (wrote about it here https://learnbyexample.github.io/python-regex-possessive-qua...)

Hope they will add support for Unicode sets `\p{}`, subexpression calls, (*SKIP)(*F), etc. For now, the third-party `regex` module is handy for such features.

Most used: f-Strings Second: pattern matching and dataclasses

If I ever could wrap my brain around asyncio, it would be an appreciated third.

I never used the walrus operator, IIRC, and I find type hints not so useful. Somewhere in the future I will try to use them in APIs, they may have their benefit there.

Edit: I forgot enums. Not so new, but a must.

The `click` library would be a nice replacement to `argparse`

"Click" and "DocOpt" are the 'argparse' replacements. They're the most semantic and intuitive way of developing command-line apps.

I've found Typer [1] pretty intuitive to integrate with existing code, it uses type hints to generate the CLI arguments.

[1] https://typer.tiangolo.com/

I find `typer` way more ergonomic than either argparse or click (which it uses under the hood).

If you're developing something more complex, you might find the feature set too small, but you can get surprisingly far with what's there.

The benefits from its type safety and lack of boilerplate alone are enough to keep me using it in favour of anything else.

Thanks for sharing Typer, I was completely oblivious!

> Typer uses Click internally. That's the only dependency.

Pretty clever way to walk on giants shoulders!

Which is fine when building a complex CLI tool which requires other external libraries; but otherwise the compexity of argparse is simply not significant enough to justify the cost of adding external dependencies.

The improved destructuring syntax is amazing:

    first, *middle, last = range(10)

Set and dict comprehensions are also very useful but aren't that new anymore. Also, the Rich library is amazingly useful for all kinds of console applications.

f strings. I am averse to complexity and they actually do reduce complexity. Like I actively avoid object oriented stuff, asyncio,etc... simplicity, predictability and maintainability rule for me. Most other python newer features I have seen add more code or complexity. I suppose managing packages, build environments and constantly migrating code is something some people enjoy so no problem with it until it becomes my problem by way of dependency hell. My favorite python feature is not a python feature but it should be: pyenv -- because backwards compatibility is not a cool new feature for language devs.

The lack of boilerplate. Low ceremony. Just write the code, save the file and run it.

That in itself is a great feature.

But all the other files, and the dependencies, and where is it going to run, and does it have permissions to see all the data and account resources it needs, and the testing, and...

Type hints, Data Classes, and Enums

Reply on news.ycombinator.com