Python: Please stop screwing over Linux distros

There are distributions that keep up-to-date, though, e.g., archlinux.

Serving the needs of python developers vs. sysadmins is a false dichotomy. Python developers develop on a system that they need to admin. One great thing about linux is that everything on a system can be kept up-to-date using just one software tool (a package manager). You are not going to convince me in a million years that this is a bad idea.

Now, for things that are needed in development you sometimes need different versions, in particular if you happen to have an OS that ships very old versions. For that purpose there should be some sort of tool and not a gazillion tools that are all incompatible and behave in slightly (or not so slightly) different ways. 'python -m venv' is different from 'virtualenv'? WTF?

Also, if your distribution is up-to-date (e.g., archlinux) and you are pinning to older version there is also a problem. As the article puts it 'pin their dependencies to 10 versions and 6 vulnerabilities ago'. And if you actually want to maintain your software in the future you, at some point, have to go to the new version anyway.

- Linux dictros have curated packaging, often outdated but stable.

- PyPi, NPM, and others are Uncurated, anyone can publish, could be unstable, could be insecure.

It's down to the developer to decide which route they want to take, and at the moment most want to move quickly with the latest tools. To do that you have to go the uncurated route. It only becomes an issue for a developer if their software is published by a Linux package management system, but 99.9% of developers will never have that.

Please substitute s/sysadmins/users/, and realise your developers are users too.

I've been doing Python for almost 15 years; and I'm getting really fed up with some things. Packaging is a mess. Distribution is a mess (for servers/IoT - Docker saves the day; for desktop - I feel like giving up). Managing the installations is a mess; upgrades can be impossible - I'm hard-stuck on Python3.6 on one project!

I find myself rewriting many smaller tools in Go or Rust, just because I can upgrade the toolchain at any time, and/or ship a static binary. But Rust has a very high barrier to entry, and Go tends to be simplistic.

I'd fully jump ship today, but Python has just too much momentum behind it.

No. As a user I want dependency management (and all of software distribution, to be honest) to be handled by the party that's best able to keep things working while at the same time keeping them secure. Linux distributions have a much, much better track record at that than most upstreams.

Distro-supplied interpreters and their associated libraries are there for the applications supplied and supported by the distribution. Unless you are developing something to be part of the distro, they are not for you.

Do not let the distro get between you and your libraries. Supply your own.

Absolutely not. Most distributions ship versions that are released before the distro release freeze.

The whole point of using a distro is to have a reliable and trusted development and production platform.

Use Fedora, or Arch, or another developer-oriented distro.

If I can make an observation - it has nothing to do with the number of maintainers. The problem is deep, cultural and occupies a difficult space where it might be a bug or a feature.

The root cause here is that the Python project, and surrounding community, have little real respect for backwards compatibility. The complexity of Python setups is driven by the need to run multiple - potentially even mutually incompatible in the case of 2.x v. 3.x - versions of Python.

All languages have packaging problems, but Python is unique in my experience in the sheer number of Python installs that I need to manage simultaneously. I still have C code that works from around the time that I learned C. I'd need another Python environment installed to say the same thing about Python.

I think what it makes it more difficult in the case of Python is that it has decades of legacy to deal with. No consistent semantic versioning, packages that expect that they can modify their package path in-place (this is a nightmare for immutable systems, like NixOS or OSTree-based system like Silverblue), a wide variety of build systems that sometimes hook into make, etc. Solving this is a hard problem.

This is why an authority like PSF has to step in and say: this is how it is going to be done from here onwards.

[1] E.g. Rust's Cargo.

You might benefit from using Pipx in this case: https://pypa.github.io/pipx/

Pipx is good for the case of "I want to run a standalone Python application that is available through Pip, but not my system's package repo." This is a more common case than you might think.

It's a sensible alternative to `pip install --user`, and having self-contained deps for tool is a bit like `npm install --global` or even `volta install`.

This is a great recipe for disaster. Whatever you install in user mode will shadow anything installed system-wide, so when you try to run some system-wide project, it may now fail. I'm also not a fan of how it drops scripts into `./.local/bin`, since that's where I keep my own script, and is version controlled.

The installation will also be frozen and never get updated -- unless you remember to do it manually.

Finally, and worst of all, this leaves you in a dead end if your packages have conflicting dependencies, which is too often the case in Python-land.

Debian Developer here. Part of packaging work, for Python libraries or anything else, is to verify the reliability of the upstream developers, audit the code, set hardening flags, add sandboxing and so on.

I spotted and reported vulnerability myself and it's not uncommon.

Nuitka can also apparently compile an entire dependency tree into one binary.

1. Server applications that run in a dedicated environment.

2. Tools you write and run just on your machine (or some virtualenv, whatever).

3. Redistributable cli or desktop applications which end users will install and use.

For the first two types, you should never have any issues with Python and its dependency situation. You pin everything, and that's it.

For the third kind tho, it's complete pain. Different distros ship different Python versions, so you need to support all of them. You also have to consider that dependencies can't be an EXACT version, you have to support a range of them, and a variety of combinations.

And then, one dependency has a version that works in Python3.6, and another for 3.9. But they had an API change, so which one do you use? It'll break for half your users either way. Of maybe just put some `if version <= 3.6` all over the place, like we did during the py2->py3 transition?

https://github.com/openstack/pbr https://github.com/codrsquad/setupmeta

> If I would get to redesign package management (both for Linux distros and for languages), I would have one package manager that installs everything into a global package cache, and then pick the correct version of libraries at run time (for Python: at import time). Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable. Instead, make it easy to get bleeding edge versions into the repositories. They can be installed side by side and only picked up by the things that actually use them.

You may want to check out Guix and Nix - their approach is pretty close to what you're describing.

A common solution to this is if you still want to run traditional distros is to just run "bare infra" (whatever that means) on the host OS and everything else in containers or Nix.

I think this requirement made sense when disk space was scarce.

I think this requirement makes sense if you trust that your distro is always better at choosing the 'best' version of a dependency that some software should use than the software author.

Nowadays, I think neither is generally true. Disk is plentiful, distro packages are almost always far more out of date than the software's original source, and allowing authors to ship software with specific pinned dependency versions reduces bugs caused by dependency changes and makes providing support for software (and especially reproducing end-user's issues) significantly easier.

Isolating dependencies by application, with linking to avoid outright duplication of identical versions (a la pnpm's approach for JS: https://pnpm.io/) is the way to go I think. Honestly, it feels like the way it's already gone, and it's just that the distros are fighting it tooth & nail whilst that approach takes over regardless.

I believe Nix and Guix also work this way.

I do not have a good idea, but other ecosystems evolved much more sane in the realm of packaging. While not ideal, Go has done a fairly good job - and the "module" operations are instant - which they should be.

You can both allow different versions of the same packages to coexist while also managing updates and installation/removal of software. It doesn't have to be this way. Software should be able to ship with its dependencies included and work and not rely on the whims of the OS getting it right.

I’m not exactly sure how it works but I think I’ve heard that newer releases of Enterprise Linux (EL8+) support multiple channels of the same package or something similar.

Essentially what Bundler has done for Ruby since pretty much forever.

I would love to see this pattern elsewhere.

???

$ apt-file search /usr/bin/venv | wc -l 0

I do agree with you however that python3.9 should be moved into libpython3.9-stdlib, and hence be available to all installations.

However, I thought the point was helping distro package management, which, to my knowledge, is not really built to support multiple installed versions of a package at the time: `dnf upgrade` for example, will upgrade all single instances of each packages to their newer release.

> I would have one package manager that installs everything into a global package cache, […]

There is exactly one package manager. If you're on Debian or Ubuntu, it's dpkg. If you're on RedHat, it's rpm. If you're on Arch, it's pacman. Yes, some of the BSDs have two (base packages + ports tree), but they're the odd ones out.

pip, cargo, go*, etc. are not the same thing. I know they're called that but they don't perform the same function: none of them can create a working system installation. Let's call them module managers to have a distinct label.

> and then pick the correct version of libraries at run time (for Python: at import time).

That's easy for Python, and incredibly hard for a whole lot of other things. A module manager can do that. A package manager needs to work for a variety of code and ecosystems. Could it try to do it where possible? Maybe. But then the behavior is not uniform and made harder for users to understand. Could it still be worth it? Sure. But not obviously so.

I would also say that this is just giving up on trying to keep a reasonable ecosystem. It's not impossible to reduce the dependency hell that some things have devolved into. It just needs interest in doing so, and discipline while effecting it. I'd really prefer not giving up on this.

> Get rid of the requirement that there is only one stable (minor) version of a package in the distribution at one time. This has become unworkable.

This is to some degree why distros are breaking apart Python. Some bits are easy to install in parallel, some aren't. There can only be one "python". Worse, there can only be one "libpython3.9.so.1.0".

> Distros, please stop screwing over Python packaging. It is incredible that Debian/Ubuntu pick Python apart and put modules into different packages. They even create a fake venv command that tells you to please install python-venv.

They're trying to achieve the very goals you're describing. Trying to give you a working python without having to download and "install" some weird thing somewhere else. And at the same time trying to keep the module managers working when they're replacing some module but not all of them.

On a subjective level, it's obvious you have a strong distaste for this ("they even create a fake") — but could you please make objective arguments how and why this breaks things? If you're getting an incomplete Python installation, that seems like a packaging bug the distro needs to fix. Is that it? Or are there other issues?

> If I would get to redesign package management (both for Linux distros and for languages),

Well, and now we're here: https://xkcd.com/927/

And, I'm sorry to say this, but your post does not convey to me the existence of any essential C codebase packaging knowledge on your end. I don't know about other ecosystems, but I have done packaging work on C codebases (with Python bindings no less), and you don't seem to be aware of very basic issues like header/library mismatches and runtime portability.

If you are interested in this topic, please familiarize yourself with the world of existing package managers, the problems they run into, and how they solve them. There's a lot to learn there, and they're quite distinct from each other on some fronts too. Some problems are still unsolved even.

Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. There is one right way to do things and the way is supported by comfortable tooling, that works so well that you literally don't even consider thinking about anything else.

The way python does dependencies is totally unpythonic. The fact that it is 2021 and this isn't fixed or at least the number one priority of things that need fixing sheds a dark shadow on the whole language – a language that I like to use.

Poetry is good. But it isn't as good as cargo, because it also has to deal with all the legacy cruft. To run code developed with poetry on a non-poetry system you have to figure out all the ways of dealing with envs, paths and such.

Issues like these get me a little fired up, because the collective brainpower wasted on something that should have been elegantly solved in one place is gigantic.

Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.

Do you have an example of a situation when the two tools had issues? I've been using both for years, and don't remember having any problems.

- python is more active than most alternatives, you have new packages created every day.

- python is massively used outside of the web, unlike JS, ruby or PHP, that are 99% web. You get Python in SIG softwares, data analytics, automation, pen testing, sysadmin, biology, etc. It's a huge graph.

- python is used by the distro themselves to code features of the OS. E.G: you remove Python, there is no yum.

- python has a rich compiled extensions ecosystem, produced from c, c++, fortran, and assembly. It's very complicated to ship them.

- it's much more common to have several Python installed than for other dynamic languages. So isolation matters even more.

So the difference is the sheer size of the problem.

Edit: Ah, apparently the steps to reproduce are: do "apt install python-pip3"; do "apt install python3.8"; when pip3 complains that it's outdated, update it with the command it itself suggests.

What about working on a project with a team?

What about deploying the code on another machine?

Trying to get the script to run on other OS's than just Debian (or Linux).

- brew install: fails probably

- pip or pip3 or whatever: fails probably, if it succeeds, breaks something else

- look for program-specific install programs/instructions (like the aws cli for example): maybe works, probably bombs

Python's reputation (and sales pitch) among developers is a language for people that don't want to program or learn software development. It's basically the new BASIC.

Their packaging and installers only reinforce this reputation.

Not that it is sweet roses in javaland or many other language ecosystems. Dependency graphs are complicated, because they are graphs when people want them to be simple trees.

As for virtualenvs, why does a user of software that happens to be python need to know that?

Native dependencies are not really handled except in an ad-hoc way, either.

If you want to import Python packages into a real packaging system, you are confronted with the tools whether you want to be or not.

This is an very quick route to a maintenance nightmare imo.

If you have totally unpinned dependencies, and you come back to a project after a year untouched, or 5 years, and it no longer works - which dependency update broke it?

I don't agree that using an outdated package is necessarily a problem at all. Some versions are done! You don't need the latest version of every possible package. You don't necessarily need to update _ever_ (which is why this differs from CI). These updates are often entirely unnecessary churn.

There absolutely are vulnerabilities in some old versions, and those updates are necessary (but tooling & notifications to easily handle this have dramatically improved in recent years, especially on GitHub). There will also be vulnerabilities in new packages though, which may be unknown, and will often not exist in older much simpler versions.

Using a well-tested version of a dependency that does exactly what you need is not less secure than chasing the latest version at all times without a specific reason.

I've found manually updating packages on the rare occasions where relevant vulnerabilities arise, and using existing working versions without changes the rest of the time has been perfectly effective over many years now, and avoid the shifting sands of external dependencies wherever possible means that a project that worked 5 years ago still works _exactly_ the same today.

I rather see more software go in this direction, valuing reproducibility & known correctness (i.e. with isolated pinned dependencies, in some form) over 'always be latest' dependency updates and the complex & hard to reproduce bugs that those shifting dependency interactions can create.

Nix is a bit of a cult. Its theoretical aims are laudable, but in order to get there it forces you to do a lot of work and reason strictly in its own way. Whether all this work is worth the rewards, I think is open for debate.

It's easy to learn coming from Python, even more so if you can start with Scala 3 right away. You cannot really get more high-level than that. There are good libraries to work with databases, Quill comes to mind if you want something user-friendly. And you can always fall back to the many Java libraries in the big data ecosystem (even though you should probably avoid it if you can).

For personal projects on my desktop, I have given up and just use pip install, and then don't try to share those scripts.

“Just learn $language_package_manager_of_the_day, and hope it doesn't break anything when your system changes” is the equivalent of the 90s “./configure ; make ; make install”, and we should have moved past that for end-users by now. Most users are not developers, and installation procedures need to cater to both.

As a matter of principle, I prefer to avoid software from non-curated repositories like pip and the like. Installing the debian-provided scipy and numpy is more than enough for me.

I don't understand what's so special about python that needs its own "package manager" when the distro-provided one is already good. If I need something, I install it using "apt install", regardless of what language it is written in.

For example, if gnuradio requires numpy then a distro that won't distribute numpy can't distribute gnuradio

The fact that it is possible to get all of these confused is in itself a problem I'd say :(

As far as I can tell, they're a strict requirement, if you ever work on more than one project and want to isolate the dependencies (because you need different versions of the same package, because you want to ensure you've tracked all deps, whatever).

Is it that you don't isolate dependencies between your python projects at all, or is there some other solution you prefer?

In summary: bin/run-app -> bin/venv-python -> .venv/bin/python -- I'm not sure how to make it simpler than this. The cost is perhaps in disk space, but I don't care about that.

Or don't do the above and deploy via docker and use pip to install to the system python in the image.

One thing that people do like about JavaScript is that you can import multiple versions of the same package. This solves that depA requires 1.0 and depB requires 2.0.

However this makes auditing nearly impossible. Small projects can end up with well over a thousand dependencies that is just unmanageable.

IMO, the Java or APT packaging ecosystems solve the issue in a far superior way.

> A library can be used together with other libraries and so cannot pin its own dependency versions (if all libraries did so there would conflicts everywhere) but instead can only specify constraints on its dependency versions (eg >=1.0.0).

This is a Python defect. IIRC, for example, Node libraries can and do pin their dependencies without conflict, because Node has no problem using multiple versions of the same library in a process.

Python can't do this. It could (there have been proofs of concept that make this work), but it doesn't.

In many cases, applications depend on other applications.

This can easily been seen in most Linux package management systems and also for many applications that act as more user-friendly front-ends or automation for command-line utilities.

Examples:

* Debugger front-ends

* Package management GUIs

* Multi-media processing pipelines

* Many others