PEP 594 – Removing dead batteries from the standard library
peps.python.org
peps.python.org
But I hate to see these get removed:
- cgi
- pipes
- uu
- telnetlib
Pipes is a small pure-Python module, with nothing tricky about it. Why delete it? It's good at what it does. Subprocess equivalents are a lot more effort to write, even if you can eventually get to the same result.
Uu is old, yes. But it's not broken either. Folks do still uuencode things from time to time. Why delete it?
Telnetlib is personally useful to me, since telnet is one of the easiest ways to interact with a qemu VM's console. I hate that the answer is "Use something on Pypi instead", because that increases my dependency surface and adds no value for me. Why delete working, stable code?
But there's a really big jump from "This tool is self-contained" to "This tool needs X from pypi". It ripples into deployment, source-control, everything.
I try very hard to make tools that don't need anything except for Python's stdlib. That's based on a perception of trust: that the stdlib will be supported, and represents Python's only stable dependency.
Once the team starts to erode that trust, Python's compatibility story will start to break.
Python changes in breaking ways--just look at the python 2 to 3 change. You will have to learn to deal with it. There is no explicit guarantee that python will continue to work in exactly the way it has always worked for you forever.
(this is an honest question, because when I saw this list I actually have a python script running on an apache web server that is using the cgi module to access post variables, and I never saw any need to change that, as any other "more modern" solution would be much more bothersome to configure and maintain.)
So presumably you are happy to have a web accessible endpoint running on your server that will pop a new process and start executing a local script (which might be reading/writing files, other services, etc.) without checking any permissions, enforcing any cross site scripting or CSRF protections, etc. It might be fine for the most trivial appplication. For anything that actually changes state or has side effects, I would be very concerned and it would never pass a serious security review.
* basic auth only exists at the proxy layer to protect apps with no auth.
* commonly deployed auth systems are secure without TLS?
because both are false. Check out the Rails Devise support for HTTP basic auth which is perfectly secure. And check that basically zero auth systems in the wild use PAKE and so are entirely dependent on TLS to secure the password transmission in flight.
E.g.
If mercury is in retrograde and the stars are aligned and it's Tuesday and the hacker is in position with Wireshark running on windows XP with a tethered pine phone, and we accidentally open all our ports, then we're completely vulnerable --> Ignore that guy.
Secondly, chances are that if you're building a CGI site, you won't be exposing it to the outside world at all because it's internal/personal jank that's built to do a single job, and not to look nice. If it's meant to look nice and and handle the stress of being used by a public userbase, then it won't be CGI in the first place.
(And fwiw the usecase I have does not have any authentication and doesn't need any. It's taking a POST var and doing things with it, but there are no secrets involved.)
I can understand some concern about the impact of removal of support for deprecated things where it potentially impacts archival scenarios (e.g., uuencoded text), but dropping NNTP from the standard library is probably the right call for Python. It's not like the protocol is so complicated you need a library to do it for you, anyways. ;-) (To be fair, I often use NNTP as my example protocol for implementing stuff because of its simplicity.)
It's not obsolete, and it is heavily used, it's working software that works well. Ripping it out just means now there will be N copies floating around that will all need to get patched/fixed instead of just a single version included with Python.
Is my comment. It's the 6th comment down. Clearly I didn't make a strong enough case, but I certainly did advocate for it and mention that it is in use.
Telnet the protocol is secure if you either (a) use the Telnet ENCRYPTION option (such as with Kerberos ktelnet), or (b) run it over TLS (standard practice in IBM mainframe environments in which Telnet is still heavily used). Most Telnet clients and servers, this Python telnetlib included, never implemented either. But people present the protocol itself as being insecure, when it is just that the majority of implementations of it never bothered to implement the security extensions which have been defined for it. Telnet-over-TLS is just as secure as SSH. Telnet has the advantage of having various standardised extensions which don't exist in SSH, such as IBM 3270 and 5250 block mode terminal support, the SUPDUP protocol used by PDP-10s and Lisp Machines, serial port control (enabling you to configure baud rate/parity bits/etc if you Telnet to a serial port server), and Kermit. Now, you may not use any of those features – in which case SSH will suit you perfectly fine – but some people do. And, in principle someone could define equivalent extensions to SSH, but by and large nobody has done so, and even if somebody did, they wouldn't be standard so little or no software would support them.
Also. If maintenance of aging modules becomes tiresome, maybe it is a strong hint to keep backwards compatibility in Python proper.
Core language backwards compatibility is a non-issue. Approximately zero percent of maintenance costs go towards that.
I'm reminded of the Hitchhiker's Guide to the Galaxy:
"It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying ‘Beware of the Leopard."
Or.. or, just consider that the community looked at the relevant trade offs and decided what it decided.
If someone wants an old built-in module they can use an old python version.
https://discuss.python.org/t/pep-594-removing-dead-batteries...
someone did volunteer to maintain cgi and cgitb.
But nonetheless those modules remained listed in the PEP.
What the cgi module did contain was some occasionally-useful support for working with HTML forms. As far as I can tell they're throwing that away for no good reason.
what else is left? emitting mime type headers and such and two newlines?
It does make you wonder why it was removed though. I can't imagine it was difficult to maintain.
[a2b_uu]: https://docs.python.org/3/library/binascii.html#binascii.a2b...
[b2a_uu]: https://docs.python.org/3/library/binascii.html#binascii.b2a...
The kind of person behind a decision like this has never heard of QEMU and won’t care even if you tell them. “Why don’t you use Docker or k8s?”
https://github.com/python/cpython/blob/a5b7678a67ac99edd5082...
It has three functions: one to encode, one to decode, and one to show usage information and handle arguments. It's only had 3 commits since 2010. There just isn't much to maintain.
I would express all of the concerns being expressed here were debated ad nauseam over the three year political process of getting this proposal accepted. If you want to vent your fears and frustrations in a hopefully more productive way, you can get involved in Python's process.
All that is to say, I'm thankful that the Python community gets things done, and people are willing to go through three year processes for changes such as this.
I also want to highlight these packages will be part of the stdlib for python 3.12, which will EOL in October 2028. In other words, if this change will affect you then you have up to 6.5 years to figure out and implement a plan.
For reference here is the first discussions: https://discuss.python.org/t/pep-594-removing-dead-batteries... and https://mail.python.org/archives/list/python-dev@python.org/...
And nearly 3 years later here is the second discussion: https://discuss.python.org/t/pep-594-take-2-removing-dead-ba...
I thought that feedback from the first discussion was carefully considered and I'm pretty happy with take 2. I have used the some of the modules being removed but largely agree now they are either legacy or there are significantly better third party options.
> Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as a new deployment model
First, MSI has better OS support than AppX, second, many users are still using Windows 7 and won't be able to use AppX. MSI is not some 30-years old deprecated standard.
In 5 years, Windows 10 will be the bare minimum for a secure OS from MS.
As for MSI, I’m torn: it is the most widely-supported and reliable packaging approach for Windows and, at the same time, a 20-year-old undocumented dumpster fire of an archive format spliced together with a scripting language forced into the mold of an allegedly-SQL relational database for which it is the sole user, and I can’t really find it in myself to blame Microsoft for wanting to kill that.
(At the same time, they also cheerfully promote an admittedly documented 10-year-old metadata format [WinMD] grafted onto a bytecode+metadata format for a different platform [.NET] packed into a Win32 executable format [PE] that is built upon an early and obsolete SysV object file format [COFF] with a stub executable for DOS [MZ] prepended in contravention of the original spec—as a substitute for a 30-year-old undocumented metadata format [TLB] that is actually two different incompatible formats [SLGT or MSFT] but at least doesn’t have so many layers, so I can’t exactly promote them as a shining beacon of logic and sanity either.)
It's interesting because once Win 10 support drops there will no longer be any x86 Windows path for Python, but there's still a lot of x86 Windows hardware that still works.
I suppose that's more an argument to move to Linux/FreeBSD though on that hardware... I can understand dropping support for something but single breaking changes shouldn't merit it.
Also, I agree that 7 was a fine OS, but it is more than two years in the grave... you cannot expect continued support for it. It is sad, but a statement of fact.
Given that any form of internet connectivity exposes you to a highly adversarial environment, I would personally be very uncomfortable persisting with Win7 - all it takes is one drive-by browser vulnerability and the attacker has a whole host of working exploits that can be used to further compromise your workstation.
I will not bother you further with the security aspect, but circling back to Python - your operating system does not exist in a vacuum, compatibility with Win 7 is actively degrading as it remains the same while Python and everything else continues to change, so it makes sense for them to prevent it being installed. That way the support community's time is not wasted even reading enough of a support issue to see that's on Win 7 and closing the issue - one or two in isolation would be tolerable, but in bulk even assigning or closing tickets would take a decent chunk of resources.
If I were to replace my msilib-based installer, I would migrate to Wix# [1], a fantastic high-level wrapper for WiX that lets you express your installer in a few lines of C# (and then generates the thousands of lines of XML that WiX needs, but you don't need to touch it). I wouldn't have used msilib if I had known about Wix# at the time.
Why would something like that add maintenance burden to the Python devs? I'd expect such code to be very, very stable.
For example, maybe they've received reports the code has bugs, but nobody thinks fixing is a good use of maintainers' time - or with such an old format, maybe no-one even knows what the correct behaviour should be.
Maybe they have decided all file loading and network-interaction code should be fuzz-tested for security bugs. Or that every such module should have at least x% unit test coverage. Perhaps the the code uses deprecated methods and needs to be modernised. Or maybe they simply feel every file ought to have an owner, and nobody trusted has stepped up.
how so? I see lower maintenance burden and vulnerability surface area as big rewards
I completely agree with you, but I'm not sure which side of the dead batteries debate you're supporting with it! ;-)
% du -h --exclude=site-packages /usr/lib/python3.10 | tail -n1
41M /usr/lib/python3.10
% ls -l \
**/*aifc* \
**/*asynchat* \
**/*asyncore* \ \
**/*cgi* \
**/*cgitb* \
**/*chunk* \
**/*crypt* \
**/*imghdr* \
**/*nntplib* \
**/*ossaudiodev* \
**/*pipes* \
**/*smtpd* \
**/*sndhdr* \
**/*spwd* \
**/*sunau* \
**/*telnetlib* \
**/*uu* \
**/*xdrlib* |
awk '{ print $5 }' | paste -sd+ | bc
877283
Is that significant? You decide. It doesn't seem all that significant to me given that the stdlib is 41M, so for space-constrained use-cases you're still not going to be able to copy/install all of the stdlib in the first place as it's still not even close to fitting on "devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit)" that the PEP mentions.
I do think that removing these packages is a good idea, but "space savings" seems like a very weak argument.
A decent first-go model for the amount of bytes to add is half a block per file. You’re adding about 168kB. That model would predict that for 84 files. I guess there are quite a few more in those (at least) 18 directories.
This way if import cgi fails then you can at least pip install cgi to fix that
This code is all open source, literally anyone could step up and do it themselves if they had the motivation.
In Java for instance, StringBuffer and StringBuilder offer a very similar interface but one offers locking. Nowadays, most locking concerns are left to the caller, not the called. HashTable/HashMap are another example... The nice thing is, old libraries "just work". Java is trying to hide older parts of the API with Modules, and it remains to be seen whether or not this solution will take off and it will be adopted into other languages.
It probably would, but the standard library is never going to be reduced to that size (I hope!), so how do these removals benefit those devices?
This would, for one, add a bit of churn to the standard library leading to a more healthy ecosystem. And two, give everyone a target to complain about, I mean, people are going to complain about something, might as well focus their attention.
The SC have also said this is a one time event and they won't accept a similar PEP in the future, modules in the future most be handled on a case by case basis and the preferred outcome will not be to remove them.
This is why I keep every program I use on my laptop running in the background at all times. Imagine having to wait for a process to start before you can use it.
"Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements." This really doesn't jibe with the "Hey guys we have tools to convert 2 to 3 so easily" and "we aren't breaking stuff." A good working library will continue to be a good working library until you break it.
"Modules in the standard library are generally favored and seen as the de facto solution for a problem. A majority of users only pick third-party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of xml. The removal of an unmaintained stdlib module increases the chances of a community-contributed module to become widely used."
First, part of the appeal of Python is that de facto standard. Recall all of the XKCD comics about it. I don't have to make choices about which of seventeen different Perl libraries I can maybe find that exist on CPAN and try to decide between them. This is one of the reasons why I fled Perl.
Second, why do we want a community-contributed module to be widely used? That's stated as a goal but without explanation. If there is a widely used community-contributed module that is better, put it in the standard library.
I could almost like "A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage ..." except that there ought to be a mechanism for cutting parts out of the standard library anyway if they aren't being used by your particular space-starved platform. This doesn't actually solve the problem being posed. It isn't the sunau module that is making or breaking the memory budget -- find a way to dis-include unused stdlib modules for projects.
Listen to this: "The crypt module has fundamental flaws that are better solved outside the standard library." So either fix them or move whatever is being used into the standard library.
Now, we can quibble about individual modules (I am rather fond of cgi, it's just so damned handy if you want to make something small without hauling in entire frameworks), but that misses the point -- the standard library exists to free us from the burden of sifting through N implementations of whatever it is we want to solve, where the complete coverage of the issue varies from "the happy path" to "eighty percent of the problem but sadly you need to do work in the twenty percent not covered."
I think the reasoning is that offering a flawed version in the standard library can prevent an alternative from being written and gaining adoption.
The tl;dr is that if you move a package into the standard library then you are restricted by the python feature release schedule (was every 18 months, now annual) and backwards compatibility requirements (see https://peps.python.org/pep-0387/).
This isn't necessarily true, it depends on what the library is targeting. Sure, you could try to use a libcurl 7.9 from 2001 but there's going to be security bugs abound and it's going to be a pain to send verbs like PUT that some APIs expect. Standards, security concerns and paradigms change over time and can easily break good working libraries that are unmaintained.
However, things like a json parser and writer are relatively stable and don't need many changes over time because of the stability of the format.