Upcoming Python features brought to you by Python Enhancement Proposals
martinheinz.dev
martinheinz.dev
Pythons massive advantage and the reason it’s so successful is it’s simplicity. Adding syntactic features only adds more surface area for people to learn when coming to the language and barriers to entry for people looking to contribute to an unfamiliar codebase.
There are a few over the last few years I truly love, such as “f strings”, the typing syntax though I’m less convinced by (and I really like TypeScript).
The recently added “walrus operator” is a good example of a syntactic addition that is both brilliant for experienced Python coders but also adds friction to new people learning the language. It’s presence in a codebase could confuse people.
Python is supposed to “favour explicit over implicit” and has always been more “wordy”, symbol based operators that aren’t inherently explicit in their meaning (like this proposed =>) are the type I’m particularly worried about.
Edit:
I just want to add, I don’t think these additions “damage” the language, it’s just the additional friction for people to pick it up or approach an unfamiliar codebase that worries me (maybe unduly). I love Python and these don’t put me off wanting to continue to use it as my primary language. In many ways I look forward to using them as an experienced Python developer.
- context manager
- decorators
- inline conditions
- else keywords in loops, which nobody knows even exist.
- raise from (empty, or with params).
- yield from (espacially with send())
- comprehension lists
- advanced unpacking. This is valid: (a, (b, *c)) = [1, {*range(4)}]
- advanced slicing. This is valid: foo[2:8:2] = bar[:4]
Most of which existed in Python 2 already !
But it has never been a problem before. Why ?
Because the community doesn't abuse features, it's part of the creed. But also because half of the python coders don't even know they exist (they are just trying to solve their problem, but dev is not their job), and are productive without it, so they just don't use it.
In fact, a lot of tutorials on the internet never even mention those.
Since the walrus came out, I used it once, and never saw it on a client code base.
But when the community do use the features, they usually make things better: decorators made flask/click what it is, slicing made numpy/pandas what it is, type hints made fastapi/pydantic what they are.
Those worries are mostly that: worries. They don't reflect what happens in practice.
The other stuff is mostly designed by framework authors to by used by framework authors. And having nice libraries and frameworks is what really makes this language so useful for everyone.
It's trendy to claim Python is meant to be some kind of "pure environment" where the source code of Flask or SQLAlchemy or NumPy should just like a tutorial, but Python has never been that in the 10+ years I've been getting paid to use it.
I thought that was the aspiration of Go. As far as I could tell, Hacker News commenters hated Go and now it's mostly ignored and forgotten here.
def foo():
return "hi"
Similarly for context managers and many other features.
We can go back much farther than that even: why have lists and tuples, when they're basically interchangeable in most cases? Why have list comprehensions when a loop does the job just fine and might arguably be clearer?
I've watched new and inexperienced programmers trying to learn full, modern Python up close and personal. (That is, they were trying to get into real code bases that were using the many features in Python, not that we were in a teaching space incrementally introducing things.) Anyone who has done that in the last ten years knows it's not simple anymore.
New in 3.3:
- yield from: I'm okay with this
- PEP 420 / namespace packages: I think this is/was the best compromise, but this remains a fundamentally ugly topic for something that is sort-of necessary to have
- raise from None: This is okay for library code
- __qualname__: It's a mess. It's not like I'm against having it, but this is a corner of the language that practically invites you to hang yourself one way or another.
- importlib: It's better than the previous iteration but this package was and still is a mess. Again, lots of this is due to legacy aka "the other packaging packages".
New in 3.4
- asyncio beta (no coroutine syntax yet): While coroutines are highly useful, asyncio and Python coroutines simply are not good. This is the beginning of the badness.
- pathlib: Ok!
- enum: Ok!
New in 3.5
- async def, await: This is the release that really set colored functions in stone for Python.
- "@" operator: I've never used this once I think, but that's because I don't write math code in Python. If I did, I think I'd like it, so I'm okay with this.
- type hints: Another oof moment
New in 3.6
- f-strings: Yay!
- dict now ordered: Yay!
- async generators and such: Again, I think colored functions are a big blunder, but if you have them, you need this.
- Variable annotations
New in 3.7
- PEP 563: Type annotations can be turned into quoted annotations without a syntactic change through __future__ import.
- contextvars: "thread-locals for coroutines"
- PEP 562: Added the Python 2 misfeature back where Python 2 interpreted __get__ and __set__ on modules
- Typing becomes part of the core language, I think this is when you could start writing things like "list[int]" (instead of "typing.List[int]").
New in 3.8
- := Yuck
- "Positional-only parameters" because the simplified CPython API for defining functions has this, so ... Python code needs that as well? Wot?
- More typing
New in 3.9
- More typing
New in 3.10
- And even more typing
- Pattern matching
I pulled this out of your post because it's something I think about a lot of the individual changes. Individually, they generally make sense.
It's just that the sum total of all the individual sensible things is a large mess, getting larger every release.
I first used Python in the 20th century, and have clocked many thousands of hours with it overall. Now I mostly program in Go. (Actually, even after ~7 years part-time professional with Go, I'm not sure which I've used more yet! I may still technically have more time with Python. If I do, though, I'm close to the crossover.) Occasionally, I miss iterators, and with Go getting generics it's probably my #1 ask in Go. Other than that, though... I just can't help but notice that I don't actually miss that many features of Python very often. I miss Haskell features, I sometimes miss Rust features, every once in a while I'll miss something more exotic like an Erlang feature, but I almost never think "gee, I really miss this Python feature". It isn't that I can't see where I'd use them if they exist, I just don't sit there pining for them all the time. I consider this a big data point. I should also point out I'm talking about subconscious inclinations, it's not like I'm setting out and daily disciplining myself to not miss Python features. It's just a thing that happens or doesn't. YMMV. I'm probably more often grateful for a missing Python feature than I'm missing it (e.g., it's actually kinda nice after a certain scale to know what "x.y" does, rather than have to go resolving down an arbitrarily-complicated chain of classes, inheritance, method overrides, etc.).
Only for personal projects, with no outside contributors, no external dependencies, etc.
If you want to read someone else's code, e.g. to learn more, or see how one of your dependencies works, etc. then you may need to face these features.
If you want to allow external contributions to your project, then you'll have to face these features (even if that just means an informal coding standard that says "don't use XYZ").
If you want to work more closely/collaboratively with others, e.g. in an organisation, then you'll have to face these features (even if it's just a discussion about whether to allow them).
If you want to write tooling for the language, e.g. parsers, doc generators, linters, formatters, diffing tool, etc. then you'll need to face them.
Finally, if you're writing library code that's intended for use by others, you need to be aware of such language features, in case they break any of your assumptions. For example, a library might rely on the fact that lambdas can't re-bind variables (they can mutate the content of an object, but they can't change the reference, since '=' is a statement rather than an expression); the introduction of ':=' expressions violates that assumption.
It is really not my experience that people read PEPs and try to shoehorn in language tricks they don't need into their random scripts at $DAYJOB.
I would actually be thrilled to see a walrus operator show up in a PR from one of my coworkers. But I'd have to shoot it down because we need to be compatible with 3.6.
It used to be that Python was mostly similar to pseudocode, easy to understand and reason about. Now all this "syntactic sugar" just makes it harder to read IMO.
Yet, if you use it more than few years, you see how it's just a disguise.
Underneath superficial simplicity there's ton of leaking abstractions, semantics which designers designed themself into the corner, and gigantic C API that blocks any performance improvements.
Can't we ever exercise some discipline and keep things simple in software? More features isn't strictly better, there's a hidden cost that adds up over time.
I'd welcome mere perf / memory improvements, security patches, and ancient feature removals for the next 5 years or so.
I don't understand what is special about functions? Why can't you set the default value for an argument called `foo` to be a function `bar` by writing `foo=bar`?
For example I can define:
def f(x, l=>len(x)):
print(l)
Also the “arrow” is clearly pointing the wrong way, it should be “<=“.
Why not just do `l = len(x) if l is None else l`?
With the new syntax can you define defaults computed from other defaults? What about side effects? Why is computation happening in function args at all?!
if foo is None:
foo = []
Short answer is these downsides were discussed but the proposal went ahead anyway.
Imo I'm with you. This type of syntax has a lot of implicit behavior where it's going to be difficult to find/fix bugs related to it.
Because dev tooling can “see” and present information from the argument list much more easily than analyzing downstream behavior.
> What about side effects?
Those happen whether or not the computation is visible in the signature (though its more obvious to the programmer of the consuming code if it is visible.)
However this comes up a lot for new engineers during programming interviews. It's very common to see recursive algos use empty lists as default/start-case args:
def traverse(graph, seen_so_far=[]): # Incorrect
....
seen_so_far.append(node)
def traverse(graph, seen_so_far=None):
if not seen_so_far:
seen_so_far = []
.....
def traverse(graph, seen_so_far=>[]):It's more concise when you are familiar with it, and it improves introspection and dev tooling when functions use it, but it is not fundamentally clearer.
I presume this got accepted because it fixes a well-known gotcha with default parameters in Python due to early evaluation, where, for instance, the dictionary instance in `def fun(args={}): …` would be shared between all invocations, leading to all sorts of fun bugs. This is especially pernicious as most Python programmers will know other languages as well, where this tends to be handled much more sensibly (e.g. in C++, D, …) and default arguments are evaluated at each call site.
The Zen of Python, by Tim Peters
[...]
There should be one-- and preferably only one --obvious way to do it.
[...]
I've always interpreted this as to establish Python as being antithetical to the Perl idea of having a hundred ways to do something and trying to do it in the most clever way, readability be damned.
It's weird, I am usually a very late adopter of technology. Being early here has allowed me to watch Python mutate into one of the things I fled from. When I heard someone say that the standard library is where code goes to die, I felt my heart break a little bit.
>Function parameters can have default values which are calculated during function definition and saved. This proposal introduces a new form of argument default, defined by an expression to be evaluated at function call time
def foo(bar, baz=[]):
baz.append(bar)
return baz
If this new syntax fixes that, I see the value.
It does. The old syntax creates and assigns the variable at compile-time. => does it at execution time.
So this code:
def foo(bar, baz=[]):
baz.append(bar)
return baz
foo(1)
foo(2)
print(foo(3))
But this code:
def foo(bar, baz=>[]):
baz.append(bar)
return baz
foo(1)
foo(2)
print(foo(3))
In fact I would be surprised if it did get accepted as many members of the Core Dev / Steering Council have similar reservations about extending Python's Syntax as others have expressed in comments here. As demonstrated by their recent rejection of PEP 677 (Callable Type Syntax).
Sorry but did I miss something?
When exactly did "this saves 2 lines in the code!" become reason enough to introduce new syntactic elements?
My first thought is decorators. They were added in 2.4, back in 2003: https://www.python.org/dev/peps/pep-0318/
All they are is syntactic sugar for:
def decorate(func):
doStuff()
func()
return
myFunc = decorate(myFunc)
There are plenty of syntactic elements that only exist to save a line or two of code. I much prefer the walrus operator over:
x = GetNextObject()
while x:
doStuff()
x = GetNextObject()They can be that.
They can also be syntactic sugar obscuring a construct that would be much more readable if it were written as some nested loops and if statements.
Problem with all things that fall under the category of syntactic sugar: The small gain in readbility in some cases is more than offset by the LOSS of readbility when they are overused.
Just as an Example: Do I think Decorators are neat? Sure. When they are used in moderation, they absolutely are. However, when codebases start to have decorators everywhere, use multiple Decorators on functions, with some Decs with Args thrown in for good measure, things start to get tricky. And latest when they are used to do something completely different from their original intent (act as syntactic sugar for wrapper functions), what began as a nice idea, can quickly escalate into a debugging nightmare.
IMO, Language design needs to be much more careful about what is added to a language than what is left out.
For sure. I once wrote a list comprehension that was nearly 200 characters wide. I thought "eh...this is extremely unreadable" and rewrote it into several lines. I figured a rewrite into a few lines of code was the better choice than adding a comment, especially if I decided I needed to change it at some point. There was a performance drop, but it was only about a second, and only when the program was starting up.
Every language feature beyond basic binary operators can be abused, but the answer isn't to remove (or refuse to implement) them, but to smack engineers and tell them a firm "NO!" when they abuse them.
I have yet to see decorators being abused (Not denying it's happened, but I just haven't seen it myself yet), but I've seen someone abuse operator overloading. Someone had decided to write their own TCP socket class, and decided that "sock += 'some data'" was a great way to send data over the socket. The scapy library kind of abuses overloading the / operator in order to build network packets with multiple layers, but I think it works in that case because anything else might actually be harder to read.
What I really liked Python for was as a _productive_, high velocity language - when I just want to bang an idea or a prototype out. Now there are all these design considerations about what version to target, what language features to use.. blah. I just wanted 2.8+ with sane unicode, and that's it. Grumble grumble.
Despite my dislike, it certainly does excel at being a glue language, hence why all of our data scientists use it with numpy and various ML libraries.
Now, Python, with Pandas, NumPy, Matplotlib, etc, provide the best “get shit done” language. Throw in Jupyter Lab and i find myself extremely productive. might try type hints next.