Hacker News Reader

Comment by elurg | Hacker News Reader

Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development.

This also applies to many others programming languages which have their own packaging systems.

While python packaging is indeed messy the needs of traditional linux distributions are by far the least important. Python packaging needs to better serve the needs of python developers, not those of sysadmins.

Having a "linux distro" interpose itself between you and your libraries is a fundamentally broken system not worth fixing. All decisions related to dependency choices fundamentally belongs with upstream.

View on HN

Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time.

There are distributions that keep up-to-date, though, e.g., archlinux.

Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on a system can be kept up-to-date using just one software tool (a package manager). You are not going to convince me in a million years that this is a bad idea.

Now, for things that are needed in development you sometimes need different versions, in particular if you happen to have an OS that ships very old versions. For that purpose there should be some sort of tool and not a gazillion tools that are all incompatible and behave in slightly (or not so slightly) different ways. 'python -m venv' is different from 'virtualenv'? WTF?

Also, if your distribution is up-to-date (e.g., archlinux) and you are pinning to older version there is also a problem. As the article puts it 'pin their dependencies to 10 versions and 6 vulnerabilities ago'. And if you actually want to maintain your software in the future you, at some point, have to go to the new version anyway.

Sure, so my lib works on mac, windows, centos and debian.

Tell me how I'm supposed to define my lib python deps ?

How am I suppose to package the lib ? Distribute it ? Deal with os versions differences? Allow isolation for my user projects ?

I'm sorry, doesn't pip already works there? Otherwise, there's pyinstaller which is great but it requires an entrypoint so it won't do with a pure lib that doesn't expose any script at all.

Yes that's the point. We won't use distro packages because only regular python packages allow this.

Edit : arf, sorry, I answered the wrong comment, I actually agree with my parent.

All right, sorry I didn't get it (I'm non native)

I have apt/brew - I don’t want to install yet another package manager. Anytime I see just use pip I just walk away and find something else.

> Perhaps the distributions would be more inclined to include up-to-date version if the standard in the python community was not to break everything all the time.

Yep... Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works.

Python? Three different software versions need three different versions of the same library, and new library versions are not backwards compatible with their older versions... Even Python 2.x -> 3.x was a pain in the ass too, and even within minor versions, you sometimes get breaking changes.

> Perl is a godsend... take a code from 20 years ago, run it on a modern system, and everything works.

That's more or less like "take a VB6 binary from 20 years ago, run it on some modern Windows, and everything works" - that's just because the ecosystem is effectively dead, so supporting it on new releases just means carrying over some stuff that worked 20 years ago.

So, Perl is the new Latin?

It can be:

  use Lingua::Romana::Perligata;
 
  adnota Illud Cribrum Eratothenis
 
  maximum tum val inquementum tum biguttam tum stadium egresso scribe.
  da meo maximo vestibulo perlegementum.
 
  maximum comementum tum novumversum egresso scribe.
  meis listis conscribementa II tum maximum da.
  dum damentum nexto listis decapitamentum fac
      sic
          lista sic hoc tum nextum recidementum cis vannementa listis da.
          dictum sic deinde cis tum biguttam tum stadium tum cum nextum
          comementum tum novumversum scribe egresso.
      cis

Source: https://metacpan.org/dist/Lingua-Romana-Perligata/view/lib/L...

It's funny, because original version of this library was written in year 2000, so literally more than 20 years ago (my example of "old") :)

https://metacpan.org/dist/Lingua-Romana-Perligata/changes

Your comment is not updated for 20 minutes, so it's effectively dead. Please, write a new one.

In 20 years when we'll all be using brainfuck, my comment will indeed be effectively dead.

What's the sin in supporting stuff that works?

This is the sad reality of modern development... new project? Why take some stable tech, when you can take a 2month old framework and an alpha version of a library, and do two rewrites, before the project fails due to breaking changes and abandoned software. It seems as if people actively avoid anything stable.

they still publish new main version of perl every year, and still support code from perl4.

But it works... it's universally supported on pretty much every os, even by default on most *nix based ones, and new versions and new CPAN modules are still written.

In 10 years, with python 4, or even maybe 5, everything will still be broken, and you'll still be claiming perl is dead, and i'll still be using the same stuff I use now, that worked years ago, works now, and will work then.

And people will still be shipping code faster and more efficiently than they would do in Perl. Because in the end, the advantages of using Python, in terms of readability and productivity, easily offset a bit of packaging pain - whereas the disadvantages of using Perl don't offset whatever marginal gain you get by using old infrastructure.

Or instead use another modern language? I avoid any python that I can’t apt-get/brew and haven’t considered writing anything with it for a very long time. There is always go/ruby/c# which seem to generally avoid the problems discussed here

And people will still be shipping code faster and more efficiently than they would do in Perl.

I feel that quite a bit of software written these days is designed to have a short shelf life. I wonder how many of the "Show HN" posts will work, or even be useful, in five years? How much will be maintainable?

I've been thinking a lot about Matthew Crawford's writings on mechanical things[0] and how they might apply to my own craft as a software developer. The work, as described generally on HN, is still all about moving fast and breaking things. It doesn't matter if the Python code I crank out today works in five years (or is maintainable in five years) -- I just need to get my product to market. Some of us work in other domains where sustainability and repair-ability are important: if my software is going to be in the field, in users' hands, for ten years, I need to consider how reasonable it might be fix bugs. If I have to fight just to get the software to run, I've already lost that battle. In these cases, the dependability of something like Perl is really great. The complaint against Python here isn't about the language itself (I think many people would agree it is quite nice to use), but rather the larger ecosystem, which makes it very hard to maintain software over the long haul.

[0] See Shop Class as Soulcraft, The World Beyond Your Head, and Why We Drive.

HN has been around for at least 5 years, right? Probably some interesting stats on how many 'Show HN's are live, dead or decomposed. Live: updated in the last year Dead: not updated in the last year, but still available. Decomposed: not findable/available.

> I feel that quite a bit of software written these days is designed to have a short shelf life. I wonder how many of the "Show HN" posts will work, or even be useful, in five years? How much will be maintainable?

Yep, all the python2 code from 5 years ago doesn't work anymore... It's sad how getting python2 to work on newish distros is becoming a great pain in the ass.

But that code won't work in five years, because all the distros will remove python3 in favour of python4. (also, the code from 5 years ago written for python 2.x doesn't work now, becauses distros removed python2)

Do you really want to rewrite all your stuff every 5 years?

As an end user, I care more about code quality and stability than I do speed of shipping code. I don't want to use some buggy code that was pushed through production too quickly just because the devs are lazy!

Yep... if a thing works, don't fix it.... Modern devs "fix it" until it's broken. Just look at google and their communication platforms... every few years a new one that kills off the previous one, with practically no added value.

There is a lot of software, from 7zip, total commander, putty, vlc, windirstat, etc., that do one (or few things), people have been using them for decades, and pretty much all the features have been there for that long, without an artificial need to "ship something new, fast" every few weeks.

>that's just because the ecosystem is effectively dead

Perl may be "dead", but that's not why it's so backwards compatible.

It is because Windows has a stable API and still has all the old functions

Yes, Win32 API is ancient. No, it does not mean it's useless.

Perl 4, 5, or 6? What's CPAN's dependency resolution like?

That's a really weak gotcha: Perl 4 was deprecated 28 years ago and Perl 6 is a completely different language that's been called Raku since 2019. There are no dependencies to resolve between them. It's like worry about dependency resolution between Javascript and Java. They're not even the same platform.

True. But Raku (https://raku.org #rakulang) does have an Inline::Perl5 module, which allows you to use 99.9% of modules on CPAN (basically, only the ones using source-filters, and ones that are really, really deep into the Perl internals). So there *can* be dependencies between Perl and Raku.

>Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin.

I couldn't disagree more. Even if the same person is admining a system they develop on, they almost certainly aren't going to admin the systems their users deploy on. The admin role and developer role should be completely separate with different goals and requirements.

The system Python should only be used for system scripts. The end. Nowadays ideally it shouldn't even be visible to non-admin user accounts, and users certainly shouldn't be installing packages into the python being used by system scripts, not even using virtual envs.

The python your system uses and how it's configured should be decided by the distro. If they want to break it up into weird packages and funky paths, whatever. That's their problem because they and maybe (_maybe_) system admins should be the only people using it.

As a developer you should be making your own decision about which version of Python you are using, what modules get installed into it, what venvs you have and how you manage them. This should all be considered with one eye firmly fixed on the target deployment environment and how your Python will be configured there. The right answer, of course, being that you should be packaging the required python along with the application.

Interesting; how would you isolate the system python from the user in Debian/Ubuntu. Do you have a link where I could learn more about this?

Don't put the system python in $PATH. Any command that needs the system python can be launched through a shim/wrapper script that sets the PATH. (Or even better to execute python directly to avoid passing the system python in PATH.)

Wow. It never occurred to me to approach it that way, thanks!

You may not be able to completely, but basically if users can write python scripts and run them using system python that's a side effect. It's purpose is to run system scripts, and any other use must not interfere or compete with that in any way.

To be hones that wasn't always the case. In the early days Python was provided on these systems for users just as much as administrators and in the early days there weren't any system script as part of the OS that used it. However now there are, and the packaging and configuration of the system python has in some cases even been mangled somewhat to suit the needs of the distro. It's time to make a clean split between system python and user/developer python.

RHEL / CentOS 8 did this a few years ago. System scripts use a separate Python binary.

Ahh, I see! Thanks for the clarification!

I would start by putting it in a path that won't be included in the users default $PATH, perhaps /sbin or /usr/sbin

Ooh, this sounds quite nice. And it would also be compliant with the Filesystem Hierarchy Standard.

This is exactly the approach RHEL has taken with version 8. Typing ‘python’ results in ‘command not found’. All the system tools that use python are set to use a custom path dedicated for those tools.

Instead of assigning blame we should interpret the tension as a sign of an important need having no good solution. It's quite challenging to overlay packaging/versionning systems .. but theoretically it's an interesting and general question. Maybe people should model and solve the whole issue (if there's some solution at all; which I hope)

The fundamental problem is the difference between Curated and Uncurated packaging systems:

- Linux dictros have curated packaging, often outdated but stable.

- PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure.

It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the uncurated route. It only becomes an issue for a developer if their software is published by a Linux package management system, but 99.9% of developers will never have that.

> The fundamental problem is the difference between Curated and Uncurated packaging systems

Exactly this! It is expected from linux distros that they have curated packaging. I think that is good and I really expect it to stay that way.

Whether you want curated or uncurated packages depends on the use case.

As the user of a program, I definitely want curated packages.

As the developer of a program, I want to specify myself which version of a dependency I want to develop against and I don't want to be hindered by the linux distro in doing that. I do think that developers in that context are not always supported that well in linux distros. And on the other hand, I do think that tools supported by the programming language can assist in that scenario. (for example installing multiple versions of the compiler and runtime in the users home directory and being able to easily switch between those versions on a project basis...)

> Python packaging needs to better serve the needs of python developers, not those of sysadmins.

Please substitute s/sysadmins/users/, and realise your developers are users too.

I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations is a mess; upgrades can be impossible - I'm hard-stuck on Python3.6 on one project!

I find myself rewriting many smaller tools in Go or Rust, just because I can upgrade the toolchain at any time, and/or ship a static binary. But Rust has a very high barrier to entry, and Go tends to be simplistic.

I'd fully jump ship today, but Python has just too much momentum behind it.

I agree to some extent, but I believe they were mostly trying to convey that much of the work is done by volunteers who are largely only motivated/have the resources to test a small subset of potential deployment scenarios. Those contributions are still valuable and contributors themselves don't owe you anything. The best way to fix this issue would be to get involved (or switch languages as you suggested, that's fine too).

Language specific package managers are the antithesis of package management. There is no "management" in "install multiple versions of stuff in this 10 levels deep dependency tree". Upstreams can pick their dependencies but at the same time cannot control what packages depend on them and make it into your app and neither can you. That's a job for your distro.

What happens if your project needs an update, but it will hose the OS? Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT?

I used to think OS package managers should be the end-all be-all, but the use cases for OS package managers are very different from language runtimes. While different Linux distros were fighting between themselves, they completely ignored use cases for projects and language runtimes. Sadly, the result is a mess for everyone.

I think people need to figure out what approach works best for them. I'm of the opinion that any core technology to your business needs to be decoupled from the OS. It makes OS updates too messy and you're tethered to whatever the OS supports. My field uses a lot of Python and every company figured out quickly they need to run their own binaries in addition to packages.

At one company, they packaged up custom RPMs. It wouldn't be a problem to package up and distribute Python libraries. Others had their own package system (no OS or runtime fit their needs). It seems like most people use something like virtualenv.

Regrettably, this means there's no easy answer for people new to Python and the right choice will probably change as you grow. But I really think the answer is Python should come up with something that works for Python and let OSes do their thing.

> What happens if your project needs an update, but it will hose the OS?

Whay do you mean by "needs an update"?

> Do you expect someone working in Python in Windows who wants to distribute a datetime package should package for RPM and APT?

Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.

This was 10+ years ago, but I remember something like the installed Python package had a caching bug that was affecting production. The update wasn't compatible with some OS scripts so the machine would no longer boot using the updated package. Problems like that came up fairly often, but I remember that was the most egregious.

Another scenario is that the OS shipped with Python 2.5 (supported until May 2011), but we had third-party tools that required Python 2.7 (shipped July 2010). Switching OSes (where things like monitor or hardware drivers weren't yet supported) was a ridiculous pain to test and certify. Decoupling OS and Python+package versioning was a huge relief for everyone, but won't make sense for everyone.

> Absolutely not. I expect a serious user of that library on an apt based system to package it and submit the package to their distro.

I try my best to personally do this and push for a work culture that does this, but even if this was done I can't fathom waiting on an OS update for existing code to percolate down. The risk tolerance, scope of concern, and agility between an OS and whatever project pays the bills are very different.

One of the annoying things is python setup.py bdist_rpm seems essentially deprecated (though it still works somewhat).

There are different cadences and roles in play here.

Cadences:

Distribution - e.g. Ubuntu 21.10, 22.04; RHEL 8.4, 8.5, ..., 9

Library - e.g. simplejson 3.17.4, 3.17.5, ..., 3.18.0

Language - e.g. Python 3.8.0, 3.9.0, 3.10.0 (This makes Python particularly annoying because for a C project it shouldn't matter if you used gcc 8 or gcc 9 when producing binaries, but with Python it very much matters which version of Python you run with)

Application - e.g. Gimp 2.8.22, ..., 2.9.0

Roles:

As a sysadmin you want to use the distro package manager to install tools for administering your system so that you can connect to wifi, monitor resources, etc.

As an application developer you want to talk to the language specific package manager so that you can use the latest versions of libraries so that bugs are fixed, and you're not shackled to people using RHEL 6 or 7.

As a Debian/Fedora packager you want to talk to the distro because that is required if you want to submit an application upstream to support users who want everything in the distro package manager.

As an application packager you may want to talk to the distro because of the above, but you could also target flatpak (or snap) so that you can use all the latest libraries without worrying about packaging for slow moving distributions.

As an end user you want to use the distro package manager because that's your embedded mental model and workflow. But you should definitely consider using flatpak (or snap if that floats your boat) so that you can use the release stream wherein the libraries are unpinned from the underlying distribution's cadence.

As an end user you should never need to deal with the language specific package manager to install applications or libraries. If you need to install using the language package manager (npm, gems, rocks, cargo, pip) then the application has not really been 'distributed' or 'packaged' imo. If you're doing this, you're off-piste trying out new stuff.

There are more details and a great conversation that can flow from this, but language specific package managers are not the antithesis of package management and supplement it; just not for the 'end user' role. Well, yes for the 'end user' role, just not directly.

This is the most insightful comment.

All the other observations on this topic (and I've read very many over the years) are akin to the six blind men and the elephant.

Wouldn't be better if the python programmers did a bit of planning ? Or are the things so boring that every python minor version must be incompatible with other python minor versions.

Features get added. So you can't use a match statement in 3.9 but you can in 3.10. If you write a library you need to be conservative in which features you use or applications can't use you.

Why not just upgrade applications ASAP? Sometimes things are removed over three or so versions. So you can't move from e.g. 3.9 to 3.10 without making sure you're importing types from the correct location:

https://docs.python.org/release/3.9.0/whatsnew/3.9.html

> Aliases to Abstract Base Classes in the collections module, like collections.Mapping alias to collections.abc.Mapping, are kept for one last release for backward compatibility. They will be removed from Python 3.10.

These things are planned. It's just much much harder to handle this because you need the interpreter at runtime while in compiled languages you just use `-std=C99` and you can compile the code and then link.

> All decisions related to dependency choices fundamentally belongs with upstream.

No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.

I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries. At best they just limit to breaking updates to once every few years when they update them.

It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

> I really doubt that the python libraries packaged by Debian are any more secure or stable than the latest release of those libraries.

They are more stable because I can keep using the same version for two years, and I'm not being pushed to the latest version that has (intentional or unintentional) breaking changes every two months. Yes, there might be a bug or two in there that have since been fixed, but I very much prefer the failure I know over unexpected failures.

They are secure because Debian (and distros like it) backport security fixes to their packages. You can argue about whether they do a good enough job keeping up with vulnerabilities, but at least I know that once I install the update from Debian, my machine is secure, and I don't have to wait for the upstream authors of all software on my machine to release updates that upgrade their dependency.

> It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

Yes, but version locking isn't my problem. The crucial difference is that distros pick a version and support those for years, while upstreams usually force you to use the latest version all the time to get security support. With distros _I_ get to decide when I upgrade, and the reduced frequency is a nice bonus. Having a single entity for all software on the system is also valuable, as there's just one tool to learn and one place to check for updates.

The python libraries packaged by Debian are provably more secure or stable: they receive security backports while also not receiving feature updates.

The combination of this 2 aspects is what provides better stability and better security.

> It’s essentially like version locking packages except some random Debian maintainer decides when it’s time to update.

Not at all.

That is not "provably more secure or stable". I think it's pretty safe to assume that maintainers backporting security fixes is more secure than just not updating at all, but even that isn't proven. It being more secure than updating is much more questionable, and is probably going to vary greatly between packages.

I am not sure if Debian can really keep give packages security updates without feature updates. For example, Debian packaged Chromium seems really outdated and having many unpatched CVEs: https://security-tracker.debian.org/tracker/source-package/c...

Yes, exactly this. Only I view it from the other way round: to try to do development with distro-supplied language packages is a category error.

Distro-supplied interpreters and their associated libraries are there for the applications supplied and supported by the distribution. Unless you are developing something to be part of the distro, they are not for you.

Do not let the distro get between you and your libraries. Supply your own.

>Unless you are developing something to be part of the distro, they are not for you.

. . . forgetting the original purpose and mindset behind Linux in the first place. From a 1990's point of view; it was always intended to be a hobbyist OS; and in most cases, one had to compile all the binaries and kernel one's self.

There was no such thing, really, as a "user" or "admin" - everyone was considered, and expected to be a developer.

Yes, and distributions arose to solve the problem that plagued early Linux: how to have a set of applications and libraries with consistent and mutually compatible versions of everything. If you're compiling everything yourself, building it so it all works once is an achievement. Keeping on top of as many moving targets as there are binaries in the system so that everything keeps working over time is not practical once the number of moving parts gets high enough.

Distros solve a real problem, but the trade-off is that some parts of the system must exist to serve itself.

> Python libraries shipped by distributions are so old that this mechanism is mostly useless for python development.

Absolutely not. Most distributions ship versions that are released before the distro release freeze.

The whole point of using a distro is to have a reliable and trusted development and production platform.

> Python packaging needs to better serve the needs of python developers, not those of sysadmins.

Use Fedora, or Arch, or another developer-oriented distro.

Exactly agreed on that. I don't understand what value distribution is providing by repackaging python libs, they re always way too old to be usable, and they re global while I work on many projects, with their own incompatible requirements. Maybe I am dumb, but I exclusively use virtualenv and pip..

> I don't understand what value distribution is providing by repackaging python libs...

I want to easily and safely use some app my distribution ships. I want to receive security updates automatically for all such apps. I don't care what language it's written in or what its dependencies are.

These app packages provided by the distribution have dependencies that are also packaged by the distribution so that dependency resolution works.

Since the point of a distribution is that it can run apps, the value is that a distribution works at all.

Valid remark. I had "development side" point of view, not "usage"

I agree with this for pure Python libraries. However, as soon as you get into things that bind to C libraries (Numpy, GDAL, etc.) it quickly becomes much easier to use the package from your distro.

Reply on news.ycombinator.com