Python 3.9 is around the corner
lwn.net
lwn.net
For people doing docker based stuff with the latest and greatest always it might be OK, however, for enterprise environments where OS versions do not move as fast and you have runtime dependencies everywhere it isn't good.
We have to build C++ libs against python versions, have thousands of users having their workstation in a sane version of python globally where all dependencies (1k+ in-house/external packages) work well and etc. etc. (think CAD kind of software / Qt UIs)
We were still in the process of migrating towards python3.7 and it was a huge effort for barely any real gain. Hopefully new libraries do not use 3.8/3.9 only features otherwise we're in for a bad time..
In any case, it's really hard to get a workstation with less than a terabyte of storage in it. I have 2.9 gigs under mine and that's for 53 environments. I'm not sure that's a huge issue.
Well.. we do have it available but only around ~10% of the workloads make use of python3 and their libraries. We have created the two different paths in our NFS and if you're in python3 then you can't access python2 and vice-versa.
We will be migrating everything to python3 eventually but as you noted it is a big effort with a lot of possibility for chaos.
Yeah, except people start putting together their own Python environments in their home directories because that's what third-party packages suggest and asking admins to install random packages globally for random one-off tasks is not really sustainable. I'm sure there are workarounds, but it would be nice if virtualenv had some automatic way of figuring this out.
You should choose one: Isolated environments or dependency hell. Each has its own pros and cons.
All the gory details about the release timeline are in PEP 602 - https://www.python.org/dev/peps/pep-0602/
That is IMO not ideal. If it was security fixes / new features / improvements with no new syntax then I would be OK.
I would say a lot of companies still have a very big legacy code base in 2.7... new services are created in docker using the latest python3 but desktop applications aren't as easy to upgrade constantly.
Some companies also use NFS to store python libraries (we release 100s of times per day) and make some wrangling of PYTHONPATH to allow applications to find dependencies. Having to support multiple (incompatible) versions of python at a workstation isn't a good problem to have.
I mean how would that with new features even work?
Walrus operator, fstring, etc. should happen maybe once every two years ideally? Did we really need the walrus operator? The cost of adding it is making any library using it in python 3.8+ incompatible with runtimes of python3.8<.
When I make public modules, I personally weigh whether using a new feature is worth losing users stuck on an older version. That sweet spot for me right now is targeting Python 3.6 and using backported modules where necessary (e.g. mypy-extensions). I'm a big type hint fan and type hinting became much more usable in >= 3.6.
[1] https://packaging.python.org/guides/distributing-packages-us...
Python used to offer it frequently. from __future__ import ...
That’s for the desktop applications, we have a bunch of services in 3.8 already.
Ideally, for production environments, you'll want to pin version numbers for everything you can, so new packages don't intrude. I also like to keep testing environments with minimal dependencies and no pinning (I built `pip-chill` for that). Wouldn't that work for you?
But not if there's a nice new library you really have a need for that uses new 3.8 features, and you can't use it because you're stuck with 3.6 for example...
Having said that, It's not something I feel. I know some things can get faster and some syntax can be improved, but, so far, I'm pretty OK with those envs running on 3.6.
As for me, locally I use 3.8. If I do any 3.8-ism, CI will catch it.
For our webservices it is much less of a problem as we can isolate the runtime from software installed in machines or third party software dependencies that we integrate with (think a blender plugin, blender comes with its own versions of dependencies)
What's the problem?
You don't need Pip if you're using rez. You also don't use a setup.py, you use a package.py instead.
By and large, libraries don't backport vulnerability fixes. You have to update from 1.0.1 to 2.3.3 because they only were notified of the vuln after 2.3.2 was released. In practice, almost no open source libraries would also release a 1.0.2. Maybe some of the largest most well-run libraries will, but enough don't that you will run into this problem.
Well, if package consumers such as yourself don't put in the legwork to make that happen then yeah, don't expect others to do the work you personally need for free.
I'm sure that if you posted a pull request for that hypothetical 1.0.2 release then the project would gladly receive it with open arms.
People should keep in mind that open source software isn't a one-way street.
Besides the pursuit for the latest and greatest, is there really anything forcing you to upgrade?
I mean, Python 3.7 will be around for at least 2 more years.
https://news.ycombinator.com/item?id=1966033
Also, the LWN FAQ says that it's OK as it serves as marketing and helps get new subscribers:
https://lwn.net/op/FAQ.lwn#slinks
P.S. I am a long-time paying LWN subscriber...
When you think about why that is, it makes sense, but I still find it annoying that while my type hints are not enforced, if I type-hint to an object that doesn't exist, I still get a run-time error.
I kinda wish that these hints would just be interpreted as comments during runtime.
The objects are not type-hinted; variables are. A variable in Python will always have a value (because it's just a label on a value, not a container), and that value will always be an object, and objects in Python always have a type.
If you mean that you are passing a variable that can have a value of None, this is what Optional[<type>] is for... Or maybe I'm missing something -- can you give an example?
import numpy as np
def f(x:numpy.array):
pass
I just feel like this should throw errors when running mypy and not when running my script.
[0] https://docs.python.org/3/whatsnew/3.7.html#pep-563-postpone...
You can do something like this:
from __future__ import annotations
Class Node:
parent: Optional[Node]
2 -> 3 was mostly about Unicode and print being a function. There was a ton of operational improvements, but none of them were breaking changes that warranted a change.
Type hints are cute, but as other remarked, they are glorified comments. Mypy worked without them in 2.x . Not saying it’s bad to have them, but that it is cosmetic.
From my point of view, the major improvements are on side tracks: - multicore support via sub interpreters (multiprocessing et al is clumsy and awful on Windows) - some modicum of static safety; like constants and declaring new variables to help with typos.
You might say that these don’t fit in with Python, but I disagree. It would genuinely be a better language, in particular for its current use cases.
But no, Unicode and fancier comments / annotations it is.
That’s underselling them. They are definitely more than glorified comments and have unlocked a bunch of cool things like data classes and typed dispatch, as well as making a generally big impact on the ecosystem. Mypy worked around not having them in 2.x with what amounted to a hack, which nobody seriously liked using for a number of reasons.
Leaving mypy aside, attrs works perfectly well without annotations, and frankly I find it more powerful than dataclasses anyway.
As I said before, maybe it is just that my use-cases are so different to others' (data science vs. web etc.). But Python is screaming for proper multicore support, better static safety, and presumably other things too. Instead we get niceties.
And yes, of course, it's free, and it's great, and I'm not a contributor (not really in a position to contribute). So I'm not complaining. But also I gained next to nothing from upgrading from 2.7 to 3, and would gain nothing from going up the 3.x ladder. So it's hard for me to whip up enthusiasm about another version of type annotation goodness.
Thus assertion is simply outright wrong. Type hints were standardized and, albeit optional, they provide a fixed target for implementations to provide the same service in a perfectly interoperable way.
You might not see their point because you chose to not follow best practices, but others among us are very glad Python finally added official support for those.
I do, and they are useful. If I had to type them in as comments, I would still use them, and probably not suffer too much.
They are a good thing, but I really think Python has bigger issues ahead. I'm beginning to wonder if Julia will overtake it in scientific programming, a notion I would have considered ridiculous a mere year ago. And that, I think, would be a shame - good job for Julia, but for me, the real magic sparkle of Python is that it is a first-class choice at so many things, _including_ data science. If the herd moves on, and Python becomes yet-another-2nd-choice for data science, that would be a real blow to its ecosystem.
Hope to see this resurface in 2 days!
What’s New In Python 3.9
For those that care, they can establish procedures and then practice them (and improve them) to update their dependencies. This helps immensely down the road when security patches are released or other dependencies have significant updates. It turns such updates into boring non-events.
Otherwise users only update their dependencies due to security threats or dead dependencies, and it always results in a stress-filled circus usually followed by outages & bugs.
Though this will tend to come at the cost of making upgrades and patches into significantly more work, as they will be larger and you less used to doing so. As the user said, "a stress-filled circus usually followed by outages & bugs".
What kind of stable environment to work in do you imagine?
From where I'm sitting, the best option on offer for coping with this inconsiderate world is to keep in practice applying your updates on a regular cadence. It may be a grind, but it's perhaps one that gets easier as you do it more often. It certainly makes for smaller, more regular updates that in my experience are often more manageable.
If I want to collaborate with someone we now need to agree on multiple versions of multiple libraries - if my code needs at least version 3.6 because I use f-strings but their code needs 3.5 because a library will otherwise break, we now have a stress point that we don't need.
Even worse, I might have to collaborate with someone who is not a programmer (very common in data science) and now I need to spend an hour guiding them through the update process plus time in the future to fix whatever the upgrade broke in the background. For a language that made "easy to use" a selling point, that is less than ideal. (Side note: it also makes Python the only language for which I regularly need to check the date in Stack Overflow answers, as those get outdated faster than, say, Bash scripts).
How does that get solved in practice? In my experience, by pinning everything to 3.5 or 3.6 and never updating.
The longer the interval between releases the more likely that kind of library incompatibility is, and the more likely an upgrade becomes too much trouble to be worthwhile. If you have to upgrade one library to be able to upgrade your Python version, you can probably manage to test that. If you have to upgrade four libraries you're more likely to give up and pin to the version you're currently on.
> now I need to spend an hour guiding them through the update process
The update process should consist of deleting site-packages and Python binaries, installing Python, and running a simple script consisting mainly of "pip install x y z" steps. Or nowadays replacing a whole Docker coffin with an updated one. Can you elaborate on the required "guiding"?
There is a lot to be said for having language tools that compile your code, compile or directly link in any external libraries from wherever you chose to put their files, and then simply give you back a single native executable file.