Python 3.9 | Hacker News Reader

Python 3.9

python.org

99 pointskumarvvr47 comments

Caution: if you rely on pybind11 or a project using pybind11 (many projects do, like NumPy/SciPy/Tensorflow/PyTorch..), hold off on upgrading to Python 3.9.0 for now.

A change in Python 3.9.0 introduces undefined behavior in combination with pybind11 (rarely occurring crashes, but could be arbitrarily bad). We will work around it in an upcoming version of pybind11, and Python will separately also fix this problem in 3.9.1 slated for release in December.

Details available here: https://pybind11.readthedocs.io/en/latest and https://github.com/python/cpython/pull/22670.

I was about to ask why would they do a release if they know it's broken and blowing up their flagship libraries? It's worrying when the interpreter starts to favor shipping quickly over being functional.

But nevermind, the 3.9 release was 15 days ago and the bug was identified 8 days ago. I guess there has to be a release before people will start using it and find bugs/regressions.

There's a saying to always wait for the .1 release, this is a good illustration of why.

What makes pybind a "flagship library"? If it was very widely used this bug probably would've been caught by people testing the alpha/beta/rc versions of 3.9.

People using numpy are not in the fringe. If this wasn’t triaged during prereleases then it’s either that it doesn’t always cause problems, or there is an issue in the diversity of testers.

> If it was very widely used this bug probably would've been caught

As the top-level comment says, it rarely causes crashes.

> What makes pybind a "flagship library"?

NumPy, SciPy, and friends are, individually and collectively flagship libraries, and depend on pibind11.

> If it was very widely used this bug probably would've been caught by people testing the alpha/beta/rc versions of 3.9.

They are very widely used, but “probably” means “do enough releases and you will find exceptions”. Welcome to an exception.

the parent was probably talking about NumPy/SciPy/Tensorflow/PyTorch, which I would also label as flagship libraries...

I personally don't like Python (I prefer Julia/Ruby/Rust), but all the new advances in machine learning are implemented in Tensorflow and PyTorch, so I came back to Python just because of these libraries and infrastructure on top of them.

I was just using the expit function from scipy yesterday by the way, because that's where it was implemented.

I don't know if I would call the SciPy ecosystem the flagship libraries of python. I know it is very popular,and probably in the top 5 use cases, and I use some of them everyday. but in my experience they've always been off in their own world in relation to the rest of the python community. Most of the people I've met that use this stuff are either scientists or finance people that are just using it as another tool along with matlab, R, and excel. Which is what they should be doing, but in their minds they're not writing python code they're using Pandas, Jupyter, and Matplotlib.

But yes, never ever jump on a new release of anything, at least in production. Always stand back and wait for someone more optimistic to find the bugs

> I don't know if I would call the SciPy ecosystem the flagship libraries of python.

I mean, sure NumPy is integral to python finance and scientific computing...but it's also right at the heart of the popular Python roguelike tutorial, too.

There's Python code that doesn't use NumPy, sure (I've got some in production right now), but I can't personally think of a Python library with nearly as deep dependencies across nearly as diverse a range of use cases.

hold off on upgrading to a new major version is always a best practice, unless you really need a specific feature in the new release

Or you feel the call to adventure!

The annual release cadence is a disaster for Python. I've already run into countless interpreter compatibility issues where new code has to run in a different environment. 3.6, 3.7, 3.8, and now 3.9 are all still officially supported, which just adds a huge mental burden when reading python code, because you have to remember when each feature was released.

I still have code that relies on 2.7.... yes, i'm ashamed of myself :(

How does it add burden when reading code? Writing code, sure, but only when you don't research the environment you're going to be deploying into. Unless you started working with Python like last year you probably remember when new useful features were released, and for stuff you haven't used before, the Python documentation always tells you when it was added.

Say your production environment runs on python 3.7. New developer on the team opens a pr and you see dataclasses, a class attribute type hint, maybe a walrus operator and some required keyword-only args. They say they've tested the code in their local virtual environment and it runs. Which of these features break the build and which are the new best practices?

Sure ci will catch most of these but maintaining 100% code coverage is a significant burden and very rare in practice.

If your new developer installs a higher version than you are running in production, the issue is with your on boarding process, not python release cadence.

The issue is that python now is effectively 4 languages at once, making all aspects of working with it more complex and prone to error, onboarding included

Not now, breaking changes even between minor versions, have always been a thing in Python.

That sounds like a work culture issue. This would apply to any compiler or language. Why isn't everyone on the same version?

This hardly seems like a Python specific issue.

Shouldn't your development environment mirror production, e.g. by pinning the python version to 3.7 using a combination of poetry and pyenv?

I thought this was standard practice?

Python should do Long-term-support releases, and annual stable releases. (an old suggestion too - https://www.python.org/dev/peps/pep-0407/)

Unlike 2.7, I do not see a consensus on a 'standard' version for python 3 between 3.4-3.7. (not any of the others, that's for sure)

I totally agree with this.

I cannot upvote you twice, but here's a comment to show an extra bit of support. Please do this, PSF!!!

I love to use Python as counter example of C++'s complexity.

Have been using Python on and off since 1.6 and not only is the documentation also quite impressive, when one sums up language reference + library, just like ISO documents, its changes between versions make a Pub Quiz a very easy task.

The language is probably the new Basic, but to achieve 10 dan in Python there is plenty of stuff to chew on.

A good CI/CD, with some precommits and a containerised stack should avoid you all those troubles. Also, nothing force you to update to the newest version, you can stay approximately 5y on the same release, but keeping the pace makes the upgrades less costly and less risky in the long run because you don't accumulate deprecated features, you're forced to cleanup the abandoned dependencies, ... It also makes the team happier and more attractive when recruiting.

Dupe. 3.9 came out on 10/5

Discussion here:

https://news.ycombinator.com/item?id=24691565

Yup. My bad. I saw it in a tweet and thought it released today. Sorry.

I’m curious if anyone knows why they didn’t use Antlr for the new parser. My (limited) experience using it has been nothing short of fantastic.

Apparently they didn’t even consider it:

“We did not seriously consider alternative ways to implement the new parser, but here's a brief discussion of LALR(1). [...] Other variants of LR were not considered, nor was LL (e.g. ANTLR).”

[https://www.python.org/dev/peps/pep-0617/]

Yes, that sentence is what prompted my question :D

Sounds like the thought process was along the lines of:

“LL(1) makes it too hard to add complex new features to the language, we must change to something else; PEG is something else; therefore we must change to PEG”.

Also, the PEG parser's inability to point out unintended ambiguities fits well into the general anti-formalism of Python.

It’s interesting that the old parser had the same problem. Although it did the parsing using an LL(1) algorithm, it never actually verified that the grammar was LL(1) before doing so.

[https://discuss.python.org/t/should-there-be-a-check-whether...]

I'm a fellow fan of antlr but it's not really an option for the base interpreter. They're (rightfully) very stingy about deps. cpython supports a wide range of platforms and has its own very permissive license - it's a pretty self-contained codebase and they prefer to keep it that way.

Why do they keep breaking things? Don't they understand code doesn't get maintained at this rate in the real world? You write the code, and it should just keep working, like infrastructure. It's only when you have water gushing out of the ground that you worry about a water main... not because some plumbing supply company decided that certain types of plumbing suddenly don't work.

Things should last for lifetimes before they break, not 12 months.

Infrastructure doesn't "run for a lifetime" without the proper upkeep and regular maintenance... just like code. There is no set and forget in technology, just because you are doing the infrastructure work doesn't mean someone in the company isn't.

Does anybody have interesting, practical, examples of the consequences of switching to a PEG parser?

I think this is a rather innovative choice among the mainstream languages?

Take a look at the PEP - https://www.python.org/dev/peps/pep-0617/

I'm aware of that, but I was curious about real-world effects (the PEP is mostly theoretical).

The real world effects, aside from a little slower parsing speed iirc, would only be noticed by the compiler / parser writers.

Mainly making it easier to do syntax extensions...

For Python devs there will be no real-world effects...

> Mainly making it easier to do syntax extensions...

Yes, AFAICT the main reason for the parser change is to enable the pattern matching feature [0] that’s likely coming in 3.10.

[0] https://www.python.org/dev/peps/pep-0634/

Here is a readable PEP * https://www.python.org/dev/peps/pep-0635/

PEP 622 has been superseded by 634, 635 and 636.

oh, thanks. Either way, 634 is not particularly enlightening. 635 it is..

"RUSTPROOF STRINGS"

It's clearly time to EOL Python 3 and start using Python 4.

Reply on news.ycombinator.com