Python 3.12.0 is to remove long-deprecated items
discuss.python.org
discuss.python.org
For languages where I have 10+ years of work behind, it's the exact opposite, and where I see the c/c++ model of not breaking backward compatibility a much saner choice.
Python in particular is an extremely bittersweet pill to swallow. The amount of small and large breakage I had ever since I started to use it for production work (v2.6 and onward) has been relentless. Small and large breakages requiring constant retooling and reworking. Minor release? Yeah, still breaks just as much as a major one. pyenv doesn't help when you dependencies need to be updated, and the updates do not support the lower versions you wanted to use anyway. Containerizing everything is a not a solution for a project is expected to be supported for years so the real way forward is to fix it, again, and again, and again. My experience is that every 6 months there _will_ be work just to fix bitrotting. A one year old python project will hardly even run unless it's using the stdlib only.
By comparison, I never experienced such churn with perl.
Literally every time I try to use someone else's Python code from GitHub I run into this crap.
I wanted my project to be easy to install and run, so I chose Perl.
only half joking.
It's reasonable to expect this is not going to change.
In order to make the most of Lindy Effect, I avoid any dependencies with less than 20 years of backwards compatibility.
For some that means a subset of features. For some like Python it means any python scripts must be optional and have Perl duplicates.
Also, to pile on top of the stdlib problems - it's probably clear to most everyone at this point - but NVIDIA and Google (tensorflow) are really some of the biggest reasons python sucks in this way. They're the ones that cause most of the breakage. Starting with Nvidia making breaking changes, which then propagates to tensorflow et al, and then to enormous number of packages that depend on these fundamental packages.
So to summarize, it's mostly Nvidia's fault.
That’s the point of civilization though. You iterate and what works out, other people copy and iterate even more.
See, that's a beautiful example, just not in the direction that I think you intend it. I can, in fact, take an appliance that's a few decades old and plug it into a modern outlet and it'll work just fine. I do not, in fact, have to preserve an entire electrical grid just to run my application^w appliance, because the underlying infrastructure maintains compatibility.
Which brings you to the larger generalization that you haven’t grasped: the further down the stack you go, the more stable it is. Python is nearest the top layer. Hardware changes in decade cycles. Electricity likely in century cycles. Keep going. The laws of physics change never. Obv the time scales aren’t strictly logarithmic. Human ingenuity means we could find some exceptional electricity specification tomorrow and convert the whole world in one decade. But it is relatively rarer.
This was frustrating, because I don't expect point-releases to break the standard library, let alone require a code fork to maintain cross-distro compatibility. It was disappointing, because I had never encountered such breakage from python while Guido was still in charge.
Removing deprecated stuff is… the point of deprecating stuff?
The point of deprecating stuff is to redirect to newer/better/saner APIs, not necessarily removing the thing.
This is where I like Java's approach. Stuff is deprecated in the JDK but fairly rarely is it removed. When it is, it's because the feature is either unused or so detrimental to the ecosystem as to warrant removal (see: finalizers).
The biggest boulder I currently see rolling towards me is the MongoDB driver since I need to upgrade the databases and the drivers can't cope with the latest version.
I realise this doesn't help you with historical API changes.
If you look for help in the MongoDB community and tag me (Joe.Drumgoole@mongodb.com) I will make sure you get help migrating your app.
This is perfectly exemplified by Python 3's insistence that you call print with parenthesis:
>>> print "hello"
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
I empathize with the lang devs in that having two forms of print is nonoptimal, but the fact it tells you to do something different while fully understanding what you said (as it were) is what really irritates me. It comes off feeling needlessly pedantic for the language.
What if clang says "did you forget a ';'"? It would have been better to just compile the code as if there was a semicolon?
I rather have something like this:
> Use exit() or Ctrl-D (i.e. EOF) to exit
> >>>
Less pedantic, less passive agressive. Doesn't fake being nice to the user.
Back when Python 3 was first released, the error message was not so clear, either...
I have a pragmatic take on this if only because of Python's ease of throwing a script together and its non-technical userbase. Were it something that effects correctness I'd more fully support this break of backwards compat.
- It is a maintenance burden on the compiler/interpreter writers to allow both modes
- There may be unexpected behavior when you use code from two different sources; one that expects strict mode and the other expects non-strict mode
- There may be interpersonal conflict when developers working together prefer one mode over another
“Why don’t we just keep both?” probably has many answers. The overwhelming one for me is that if everyone’s doing the same thing two ways, then everyone has to learn all the esoterics about the print statement.
“Why don’t you just do it for me?” is a cardinal sin for a runtime. And already exists in migration tools.
see also: the recent addition of match semantics in python
Is `print >> obj` a print call with redirection to the file-like `obj`? Or is it a operator call between those objects? What does `f = print` do, assign the function object or the result of calling print with no arguments?
While the Python3 lexer is specifically hacked up to recognize 'print' as a distinct lexeme — and to thereby emit an additional parser meta-instruction lexeme that triggers a special error-handling path in the parser if you then go on to make a syntax error per the newer uniform syntax — it doesn't actually know what you were trying to print. (That'd require a successful parse!) If the Python3 parser tried to do the compatible thing, that'd require ditching the uniform syntax altogether, and going back to the ambiguous parser.
It's a bit like Error Correcting Codes — the uniform Python3 parser knows enough to know that you did something wrong, and is provided by the lexer enough context to guess what kind of failure it was; but it doesn't have enough information to "Do What I Mean", because that's a strictly-greater amount of information.
If you type `print "arg"`, the syntax error is a bit more informative, but I wouldn't say that the compiler "understands" what you mean. It's just making a guess based on the fact that the previous token was "print".
On the other hand, supporting a special call syntax just for print, which includes more than just parentheses, would be substantially more complex. And then Python code in the wild would be more inconsistent and there will be endless debates about `print <args>` versus `print(<args>)`.
Ah, but that's because the Perl community rejected the one major attempt at such a change (Perl 6) so hard that it became its own separate language.
I've dealt with a lot of sideways yak-shaving work in my career and still frequently today and Python has been the absolute least of it.
I literally have 20+ years old perl scripts, usually doing one single thing (many of them) still working on new machines without issues.
I've rewritten some stuff from python(2) to perl (instead of python3) because i was unsure when a new rewrite for whatever reasons...
Now, more python stuff needs fixing, while perl still works.
If you had a set of single simple python functions maybe they wouldn't have broke as much if at all?
Like (python 2):
print "hello world!"
It might be time to move on…
Our codebase is was (and is) old, but the forced change came recently, when ubuntu decided to remove python 2 support (ubuntu 20.04?). And when ubuntu decides that they want python >=3.12 as the only version installed, this means another change and more work to fix stuff that worked just fine.
I am still shocked though by the Python project deciding to break billions of lines of legacy code. What other platforms demonstrate that kind of contempt for their developers?
Removing the print statement seems like a particularly negative trade-off: breaking code in the name of syntactic purity, instead of simply deprecating the print statement and letting it live on compatibly with the print() function.
Since then a bunch of new syntax (not all of which is bad) has of course been added.
It's also weird that until now they seem to have largely ignored performance and focused on adding random new features.
Modules being removed [edit: in 3.13]: https://docs.python.org/dev/whatsnew/3.12.html#pending-remov...
Updating CGI scripts will be a bit fiddly: https://peps.python.org/pep-0594/#cgi
Python 3.12 removals are here:
#!/usr/bin/env python
from wsgiref.handlers import CGIHandler
def app(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/html')])
return [
b"<html><head><title>foo</title></head><body>bar</body></html>"
]
if __name__ == '__main__':
CGIHandler().run(app)The suggested replacements either are asyncio based (which means whole ass rewrites as asyncio is really fucking opinionated), or are excessively restrictive in some stupid way.
"Infosec" at work tend to raise a ruckus when their scans detect "older" versions of Python on machines too, so virtualenv'ing to pin versions or similar is often not the most practical in prod without getting an exception.
Even if they don’t, from a cursory view, it seems it’s a pure python library (https://github.com/python/cpython/blob/main/Lib/telnetlib.py).
If so, can’t you copy the current code into your project? You would take on the burden of supporting it, but looking at https://github.com/python/cpython/commits/main/Lib/telnetlib..., that doesn’t seem to be much of a burden.
Regarding security, some would advocate that telnet, or whatever else, is secure at least as much as the network underlying it. So anyone who puts their "legacy" telnet apps on a VPC is fine, and has decades more to enjoy software that has already been running for decades.
Trusting the network for security was common in the previous century but standards have improved since then. For example, sending your password in clear text is no longer considered acceptable by mainstream security standards since it avoids the risk of passive network monitoring or accidental exposure.
> (of words, equipment, etc.) No longer in use; gone into disuse; disused or neglected (often in favour of something newer)
https://en.wiktionary.org/wiki/obsolete
Bringing that full circle, telnet used to be common but it has security issues (lack of encryption or integrity unless you tunnel over TLS, lack of modern authentication options, etc.) and so anywhere it’s still used we should be looking in to replacing it.
It just doesn't make sense to encrypt everything.
And everyone who uses "telnet as a program for connecting to a server on any port and talking the protocol manually" forgot that telnet is a real protocol and this type of abuse only works if the telnet client is sufficiently dumb / doesn't negotiate parameters. There is a reason why netcat exists.
Just because some old and rudimentary telnetd or telnet client doesn't implement it doesn't mean it doesn't exist / isn't used.
Apparently, it's impossible to migrate to ssh and why the ** I need to care about security here?
Does not matter, you stated without nuance that telnet should no be used. So you told GP that they should trash their telnet-only devices.
But in many cases out in the real world we have to support frankly awful old equipment that only offers telnet (if you are lucky).
I don't think we will see the back of such things before 2030 either.
Ok. Telnet is a protocol, it is not synonymous with TCP.
As for unencrypted traffic generally, I see no reason for fetishizing encryption. "Compute" is not synonymous with teh cloudz. Cloud tech is not synonymous with teh cloudz. It's trivial to put e.g. Nginx in front of something.
If you need encryption between 127.0.0.1 and 127.0.0.42 you clearly don't trust the hardware you're running on; I'm sure someone's got some clever way of hiding the private keys, but "clever" doesn't mean provably secure. Between containers... meh. Show me you thought about it and have a threat model.
Between datacenters... not so much. Do I care whether the connection is encrypted, or whether all traffic goes through a tunnel? I'd like to see a threat model. Is sigint included?
Even across the globe I might make an exception for a well-reasoned argument. That would almost certainly have to be data which was an observable: the time, a flow rate. Encryption is not the same as tamper-proof. Even unencrypted data might need defense against that. (Maybe secrecy is easier than nonrepudiability. Maybe.)
Secrecy is not identical to privacy. Security is not identical to either of those, and hypervigilance isn't the same as not needing to be concerned with something (these days security can be either of those). Observability might be more important than either secrecy or privacy depending on the application, and who is doing the observing.
A lot of it depends on where you demarc your rings of hell and how you defend things.
I send data in the clear sometimes and I'm ok with it. I also use a telnet client for decidedly "not telnet protocol" purposes related to diagnostics as well as observability.
- Set global and per-project python versions
- Not written in Python.
- Shims your PATH
- Linux, Mac, Windows
Hell, the fork could have used Go or whatever too but it went with a bunch of .bat scripts.
Which is to say, no pyenv is not an option for everyone.
git clone https://github.com/pyenv/pyenv.git
cd pyenv/plugins/python-build/bin
./python-build --definitions
./python-build 3.10.8 /opt/python/3.10.8
PYTHON_CONFIGURE_OPTS="--enable-shared" ./python-build 3.10.8 /opt/python/3.10.8Once you're used to the workflow it's pretty smooth. (We switched from conda 2 years ago)
Conda seems to be the most prone to getting in weird states or just hanging while "Solving environment." I have been happier leaving it behind.
Really the only two I would even consider using now are pyenv and poetry.
The other thing about Conda is that it doesn't cover just Python and Python packages, but all kinds of software that may be directly or indirectly relevant. For example, on Windows, there are Conda packages for the Windows SDK, and for various C++ compilers.
Does that mean I can do things like install black and jupyter once and use that install across projects?
I don't only use Win but I do use windows, so having to use different tooling makes pyenv a hard to swallow pill
Yeah “pyenv install 3.10.8” is basically rocket surgery, near impossible to learn.
$ pyenv install 3.11-dev (or use pyenv install --list to show list all available versions to install)
$ pyenv shell 3.11-dev
$ pyenv local 3.11-dev
$ pyenv global 3.11-dev
There you go, now you know pyenv. Call it with no arguments to show the other possible commands.
(I know there's pytest and nosetests, but especially when teaching people, it's nice to use what's in the standard library and unittest sticks out for having things named differently)
Would be nice to clean there up in some point to increase consistency.
You’re just crippling them for no reason.
In theory, you add a few libs and you're ready to upgrade. In practice, adding libs at scale is hard, and quite a few dependencies of dependencies of ... are using reflection to mess with internals.
As a result, there are still a lot of enterprise applications on JDK8 and JDK11, even if the security impact of this is bad and the fix should be easy.
For a typical example: see https://github.com/x-stream/xstream/issues/101
I disagree. The entire point of setAccessible is to say I want to access nonpublic things. I shouldn't have to also say "Simon says, pretty please" for each such access on the command line to be able to do so.
But you can't expect from your API vendor that they won't ever again change the internals of the implementation just because you forced your way in and monkeyed with the internals. Writing your code this way is a surefire way to need rework at some unpredictable time in the future.
If you're lucky, it just crashes. A worse possibility is subtly corrupting and destabilizing your program. That was what happened in some of these cases: hashcodes were stored/cached in collections after an upgrade, and people using reflection to 'restore' a collection didn't update these caches correctly or dropped them in the wrong hash bucket. Then the hashmap had elements that were both there and not there, depending on how you queried it.
One of differences of a senior engineer is that you not only say it works today, you can guarantee how it stays working long term, even when the environment changes. Things like tests and comments are part of that. API contracts are a big part of that, both in being explicit about them as an API provider, and not touching non-guaranteed parts as an API consumer. Using setAccessible like this is a grave violation of an API contract, and it takes away your ability to upgrade to later version.
> You can use https://github.com/isidentical/teyit to automatically modernise your unit tests.
I tried out teyit when it was a "Show HN" 10 months ago (at https://news.ycombinator.com/item?id=29948813 ).
It not only pointed out places where I was using a deprecated alias, but also fixed a few places where I was using the API poorly (using "assertTrue(a binop b)" instead of "assertBinOp(a, b)".
I recommend it.
That was developed for 2to3.
Huh. I did not know this - in lib2to3, the "asserts" fixer handles this sort of conversion, so you have the ability already .. so long as you don't use new Python 3.10+ syntax that lib2to3 doesn't handle. See https://docs.python.org/3.7/library/2to3.html#2to3fixer-asse... .
I wonder if serhiy-storchaka (who committed that What's New entry) knows about lib2to3's "asserts" fixer. And if that might be a relevant addition or change to the documentation.
Someone who cares about this might want to point it out.
I know setuptools is advanced and recommended but distutils worked fine for me and my users for so many years. It is going to be overhead for me to comb through all my projects and replace distutils with setuptools, test them and testing out packaging and distribution is quite a lot of work.
I am growing unhappy with Python due to these breakages. Is there some other programming language whose maintainers don't break the "user space" like Python has been doing time and again?
> For projects still using distutils and cannot be updated to something else, the setuptools project can be installed: it still provides distutils.
From: https://docs.python.org/dev/whatsnew/3.12.html#removed
I think Go qualifies here. (but it's still much younger than Python).
https://github.com/libguestfs/libguestfs/commit/26940f64a740...
Damn! I guess it's time to cross out my snippet of running a debug smtp server in almost any Linux distro & Mac:
python -m smtpd -n -c DebuggingServer localhost:25
> Remove the distutils package. It was deprecated in Python 3.10 by PEP 632
So apparently we need to replace
from distutils.version import StrictVersion
with
from pkg_resources import parse_version
A bit labor for my code but OK.
Though this project isn't exactly in the best shape, having prior bugs: https://salsa.debian.org/debian/syncthing-gtk/-/issues/2
> Remove the filename attribute of gzip.GzipFile, deprecated since Python 2.6, use the name attribute instead.
2.6 wow, that's some compatibility right there. About time that got removed then! Either that or undeprecate it, if it's fine to use. Any decision at this point is a good one.
hidden gem
As it is now, there were breaking changes in wxPython (likely due to the culture of breaking working code extant in Python) which result in WikidPad being broken.
It seems to me that if the Python community continues in this direction, nothing will work more than 3 months after it's last github commit.
[edit/append] No, I'm not sure it's wxPython and not wxWindows that is the issue. I'll have to stuff my Linux boot SSD in, and then try to build WikidPad again to know for sure (it's been too long)
I didn't write WikidPad, it seems to have been last maintained about 2012, but I use it for my notes, etc. I really like it, but the breakage on the Linux side is a show stopper. It's really unfortunate that Linux doesn't have a stable API like Win32, and forces dependence on source code.
I'd consider installing an older version of Linux to force older python, wxPython, wxWindows, etc... and try to figure it from there... the last time I looked at it I got a wall of confusing errors, and couldn't patch it enough to get any functionality out of it.
On top of that, wxPython is a Python module which interfaces with an external library (wxWindows). Are you sure your problems are even with the Python module, and not the result of breaking changes in wxWindows itself?
“Deprecated since version 3.6: pyvenv was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is deprecated in Python 3.6.”
Version is a nightmare btw. I have Conda just installed but sometimes the python is not the same as python3, one to 3.9 and one to 3.10. And unless you know the difference of pip3, pip and python -m pip (and not pip3…).
Using venv then somehow the script Kivy switch it out and fall to python 3.9. Where is it? And then it fail as there is no python in my macOS but just python3. Struggle whole night on how to fix the bash alias and fail then use symbolic link.
Just a “user” guys. Do not confuse me please.
“ There should be one-- and preferably only one --obvious way to do it.” sigh.
pyvenv, which was removed in Python 3.8, was a wrapper script around venv. The same functionality today is invoked via `python3 -m venv`.
That’s a pain when you already are using libs that imitate the “new” behavior.
Either the new tools are less good, or they destroy development in the existing tools.
Twisted was, and is, a really cool framework. Unfortunately, it’s doomed.
There were a ton of really good datetime utilities before … datetime.
Oh well. Complain complain.
Seems like such a wasted opportunity, to add a sliding window for n=2, and then in the documentation add a recipe for a sliding window function for any n.
It's nice to not have to depend on anything external, but boltons is something I just treat as an expanded part of the standard library.
Anyone care to comment on what they are using for receiving emails in Python?
Update: smtplib is still on, so that makes sense.
In most cases, you use an existing MTA (like Postfix or whatnot) and set it up to deliver mail to a Python script. Or, even less directly, you use an IMAP library to access mail after it's delivered to a mailbox, or use a mail provider which can call a webhook over HTTP when an email is received.
For the rare situations where you do really want to write your own mail server, aiosmtpd exists, and the migration process doesn't look terribly complicated: https://aiosmtpd.readthedocs.io/en/latest/migrating.html
What is the relation between a new Python version and changes in modules?
Are some modules considered to be part of the language?
As far as CPython is concerned, most of those modules are implemented in C, and therefore for practical purposes part of the interpreter itself.
> How can a language be released?
If we really want to be pedantic, a new version of the language spec can be released.
Translated to C++ terms, this is like if the ISO committee approved a change to the stl in the new standard.
> In the unittest module, a number of long deprecated methods and classes were removed. (They had been deprecated since Python 3.1 or 3.2).
I doubt your projects are using 3.1 or 3.2 (or older, except perhaps 2.7, but then you wouldn't care that 3.12 was removing something deprecated vs 3.11 which has it)?
https://docs.python.org/dev/whatsnew/3.12.html#removed
An example of a more recent deprecation is the 'distutils' module which was deprecated in 3.10.
distutils was functionally deprecated long before that. Even the Python 2.7 documentation (released in 2010!) recommended that users avoid distutils and use setuptools instead: https://docs.python.org/2.7/library/distutils.html
Last I checked, there was no good way to package hand-written Python/C extensions than setuptools, but that was a couple of years ago.
In any case, there's a large installed-based of setuptools-based projects making it hard to get rid of.
Nevertheless, there is no doubt that it is the worst software project that I have ever seen, during several decades, from the point of view of keeping compatibility between versions.
For other programs, I may happen to need to have installed 2 versions, or maybe 3 versions, at most, in order to be able to use other programs that are compatible only with certain versions.
On the other hand, for Python, during many, many years, I have been forced to keep installed all the time around 7 or 8 versions, to keep happy many other programs that claim to be compatible only with certain Python versions (and not also with any newer versions; some programs are even compatible only with a list of non-consecutive versions).
Building from source any program that depends on Python may frequently require various temporary configuration modifications, to ensure that the program is built only for the Python versions with which it is compatible.
I love writing Python code. I love its standard library and many third party libraries. But I loathe Python’s dependency management, and Python version upgrades are sometimes a huge pain. There are tools to help with this (pyenv or I am fond of asdf for python versions; poetry or pipenv for isolating package dependencies), but they only partially solve the problems and introduce new ones.
> to express disapproval of
> to withdraw official support for or discourage the use of (something, such as a software product) in favor of a newer or better alternative
In other words, deprecated is a strong signal that one should strive to migrate from items marked as such.
Compared to that, Java still carries some deprecated items around. But with Java 9, deprecation for removal was introduced, which is a strong signal that stuff will be gone in two releases. Kubernetes has a strict deprecation policy as well, such that skipping more than two releases is asking for trouble.
But the Python community fear major version changes. So the minor versions become major versions.
Now there will be an endless churn to keep programs up to date for "security reasons" and that cost money.
We can even pull the plug on programs nowadays to the give the users the very best experience! Otherwise the lusers might have felt satisfied with what they have.
Ye I am whining but only half joking.
Breaking backwards compatibility ruins my mood. And Python are getting way to much slack for what they have been doing to the users.
People wanting to stick to 2.7 are ridiculed and soon the software security police will start arresting offenders.
Using 2to3, the most trouble I had were db migrations, which I just squashed. The rest were ferreting out str() problems that 2to3 didn't find (eg. redis package happily taking strings but returning bytes by default), Django regexp url changes, and trivial stuff like that.
It took a couple of days work.
I also wanted to upgrade the frontend part (webpack build of Vue). I stopped after a few hours od going nowhere and am still apprehensive of approaching that particular thing.
Besides, a few years ago they moved from Mercurial to Git(Hub).
Sigh.
Deprecations are the way to indicate that support for the module will be eventually dropped. There is no reason to leave those modules there if they cannot be relied on.
https://www.activestate.com/products/python/python-2-7/
As you can see, it is still offered, and the PSF doesn't have a problem with that.
The problem with those other groups is that they didn't want to maintain Python 2. They wanted to take it and develop it, evolving the language separately from Python 3 - while also calling the result "Python 2.8" (and presumably later Python 2.9 etc). It stands to reason that people who own the brand don't want it to be associated with a third-party fork that's making its own major design decisions, no?
It would be nice if this was linked from python.org as well.
Is the disaster of the python 2->3 rollout preventing them from doing so?
Either way, doesnt effect me, python 2.7 forever.