Cooperative Package Management for Python | Hacker News Reader

Cooperative Package Management for Python

lwn.net

100 pointsTomte65 comments

(Disclosure: I’m one of the PEP authors.) I want to clarify a point since many comments seem to go “I do container/Nix/virtualenv so this is not a problem”. That’s absolutely correct, nothing will ever go wrong if you do any of those things. This is a solved problem.

But there are a significant number of people that do not do any of these and insist on installing things into the system-provided Python and expect smooth sailing. This is the “problem” for pip and distro maintainers, and what this PEP aims to solve, to guide people toward one of those solutions.

Thank you for your work on this!

Some context, I'm a data scientist who grew up with R, and run Linux.

When attempting to learn Python, this kept causing me massive problems as I'd have a weird mixture of system installed and pip installed packages.

I think it probably stopped me from really getting into Python for 1-2 years.

So, I think this work is really, really important and I'm really glad someone is working on it.

I probably haven't thought this fully through but wouldn't it be simpler to just have a system venv - root protected - perhaps distributed using the system's package manager? Then if you mess up site-packages at least you wouldn't break the system tools.

It's already the case, that's why you see some mistaken tutorials telling you to "sudo pip install".

A venv is just a dir and some path, and your os already have some dedicated to their own python install, even if it's not called a venv, but something like dist-packages + manual PATH fudging.

This proposal makes the distinction clearer by putting safeguard to NOT to mess up with the system stdlib.

You should be using "--user" or a local venv, and "-m" to call commands.

See my other comments for more tooling.

I'm missing something I guess - I don't understand your comment in the context of this claim near the start of the article:

"The root cause of the problem is that distribution package managers and Python package managers ("pip" is shorthand to refer to those throughout the rest of the article) often share the same "site‑packages" directory for storing installed packages."

Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no?

Of course by poking around the filesystem tree and messing with managed files as root, you could mess up the system venv but that's possible with any installed package.

>Having a system venv - managed by the system's package manager - would mean this "root cause" would go away, no?

Oh, but you need to explain to pip that it's managed by the system's package manager. Which is what this does with the "EXTERNALLY-MANAGED" file.

And you need to do that because the system's package manager and pip have separate sources - with `pip` you get the packages from pypi, with the system's package manager from its repo.

To supplement, the “root” issue of the whole situation is some users are actively installing things into that system env. So the PEP is providing a mechanism for pip to detect it’s being used to install things into a system env and have a chance to refuse.

As I said, a venv is nothing more than a directory + PATH setup. OS already have a dedicated one (in fact, sometimes several). The problem is not that the separation does not exist, the problem is that pip is using the OS real estate.

Creating a new-new separate directory using virtualenv is not going to do anything, just add one more dir to the mix.

We (openSUSE) are doing this in controlled situations. We have a build system (OBS) that create VMs for us every time that we want to create / update a new package.

Those VMs are isolated from the network, but we can have partial control over it via the .spec file. This allows the call to some external "helpers" scripts that are described as a build only dependency, and that create a Python venv, and help with the re-allocation (maybe in /usr/local, or /opt, for example)

The cool thing is that the venvs are self-contained by a lot, including the Python (we can include it inside the venv, so is de-attached from the system Python version, but this is totally optional) and the system libraries (.so) that are both dependent of the Python version and the system libraries.

A non-trivial example is this: https://build.opensuse.org/project/show/systemsmanagement:sa...

If you check the subprojects you will see the venv for Fedora, openSUSE, Ubuntu and others.

If the build system is minimally powerful, you can do really crazy stuff. For example, there is one member of the team that is thinking on wrapping the final venv into SquashFS, to decrease the size and simplify the installation.

I'm thrilled to see people adressing this (decades old?) footgun. I hope there will be more collaboration in the future. The split universes system and language specific package managers operate in are a cause for much more issues.

This sounds like an improvement. In my work related laptops, I still favor conda because it seems to do really robust dependency checking.

I do many short experiments that are usually not work related (in the sense that it is not paid for work). I strongly favor spinning up fresh VPNs for these experiments and when I need Python (and often Hy/hylang, a Lisp syntax for Python), I usually just use the system python3 installation and use “—user” for pip installs to my home directory. There are probably many reasons not to do this, but I justify this bit of sloppiness because I tend to discard VPNs fairly quickly.

Do you purchase a license for Anaconda? Last time I checked, it seemed like conda required a license for professional use. I may have misread it though.

No, I have never purchased a license. If a paid for license was required for my personal laptops (which I use professionally) then I would probably just use virtualenv.

You only need a commercial license under certain circumstances, and it only pertains to accessing the default package repository at repo.anaconda.com. You can still use conda-forge etc. Conda itself (the packaging tool) remains open source.

Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.

>Using virtual environments with pip, instead of installing packages system-wide, has been the recommended practice to avoid conflicts with OS-installed packages for quite some time. But it is not generally mandatory, so users sometimes still run into problems.

I use virtualenv any time I use Python. That said, it would make way more sense if OS-installed packages used virtual environments, so that a user can pip install whatever they want on the default python install and not have to worry that it could bork dnf or any other system utility.

There shouldn't be a "python install" at all, let alone a "default". Whether an application uses this-or-that Python interpreter, or Ruby/Bash/Scheme/whatever should make about as much difference as whether a C program was compiled with GCC or Clang: it's an implementation detail, which doesn't need to be co-ordinated with everything else on the OS (although the package maintainers may want to coordinate, e.g. to allow re-use of files via soft-/hard-links, bind-mounts, etc.).

When I "install foo-1.0 into my python environment", that's an annoying implementation detail (which is also imperative and non-atomic, i.e. there is a time where the python environment exists without foo-1.0). What I really want is "run a 'python' command where 'import foo' will have foo-1.0". That shouldn't break other applications which just-so-happen to use foo-0.5, or whatever.

This is similar to the static vs dynamic linking debate. It’s easiest to deploy things if they are all self contained, but Linux distros have a strong tradition of dynamic linking and really don’t want to have (say) two separate paycopg2 installations if they can be shared. And once there’s a place to share stuff, users are going to inject new stuffs into it and give the package manager all sorts of trouble.

What’s still annoying to this day is Python imports. I have been doing Python for a while now and I still get tripped up when some bit of code in a folder isn’t importable. I’ve found adding the package dir that I’m working on to PYTHON_PATH works but then my editor doesn’t resolve packages for autocomplete.

If you have foo.py and bar.py in a directory and you want to import foo.py from bar.py you only need to add an empty __init__.py

You would then `import .foo`

https://www.python.org/dev/peps/pep-0420/ Python 3.3+ no longer requires the `__init__.py` file to make a package.

> You would then `import .foo`

The `import .foo` is a syntax error, and `from . import foo` does not work for scripts [1].

The basic `import foo` works if the directory is on your PYTHONPATH. That will be the case if you run `bar.py` as a script.

It gets more difficult if you want to separate scripts and importable modules into separate sibling directories (bin vs lib). You can't use relative imports to get around that, you need to use a virtualenv (or manual PYTHONPATH wrangling).

[1] https://stackoverflow.com/a/14132912

Use virtualenvs, that's the problem they solve.

Python now ships with the venv cli tool.

If you point your editor to the python interpreter in the venv, it will detect everything automatically.

As sibling says, you really should use virtualenvs.

But if you're not and you're using vscode, this is the relevant thingy to do to get vscode to play nice: https://code.visualstudio.com/docs/python/environments#_use-...

It's a good safeguard, and it's going in the direction of the other initiatives to make python package management default behavior saner.

PEP 852 is the another one to follow up: https://www.python.org/dev/peps/pep-0582/

It basically uses the concept of node_modules, making python interpreters load any local directory names `__pypackages__` . There are 2 differences though:

• Unlike JS, python can only have one version of each lib for a given setup.

• Since having several versions of python often matters, you may have several __pypackages__/X.Y sub dirs to cater to each of them.

It also forces you to use "-m" to call commands written in Python, which is the best practice anyway. I hope it will push jupyter to fix "-m" on windows for them because that's a blocker for beginners.

If you are not already using "-m", start now. It solves a lot of different problems with running python cli programs. It's an old flag, but too few people know about it.

E.G: instead of running "black" or "pylint", do "python -m black" or "python -m pylint". Or course you may want to chose a specific version of python, so "python3.8 -m black" for unix, or "py -3.8 -m black" on windows.

To test out the __pypackages__ concept, give a try to the pdm project: https://github.com/pdm-project/pdm

At last, some other tools that I wish people knew more about that solves packaging issues:

• pyflow (https://github.com/David-OConnor/pyflow): it's a package manager like poetry, but it also installs whatever python you want like pyenv. Except it provides the binary, no need to compile anything. It's a young project with plenty of bugs, but I wish it succeeds because it's really a great concept. Give it a try, it needs users and we would all benefit from it becoming popular as it's a very sane way of setuping a python dev env.

• shiv (https://shiv.readthedocs.io/): it leverages the concept of zipapp, see PEP 441 from 2013, meaning the ability that python has to execute code inside a zip file. It's a successor to pex. Basically it lets you bundle your code + all deps inside a zip, like a Java .war file. You can then run the resulting zip, a .pyz file, like if it were a regular .py file. It will unzip on the first execution automatically and run transparently. It makes deployment almost as easy as with golang.

• nuitka (https://nuitka.net/): takes your code and all dependencies, turns them into C, and compiles it. Although it does require a bit of setup, since it needs headers and a compiler, it results reliably in a standalone compiled executable that will run on the same architecture with no need for anything else. Also it will speed up your Python program, up to 4 times. In my experience, it's also easier and more robust than pyinstaller, cx_freeze and so on.

• pyodide (https://pyodide.org/en/stable/): python compiled to WASM to run in the Web browser. Useless for web programming given the huge size of the runtime, but great for teaching, as it allows students to basically access a zero install full featured python dev env by clicking a link. Try it out, it's awesome, you can even create and query a sqlite db, thanks to the virtual FS: https://notebook.basthon.fr/

nuitka is at https://nuitka.net/

thanks, I'll fix my bad copy paste.

On my computer, I only use Python software that is in the Debian repos. So all dependencies are handled by Debian.

Everything else I run in a Docker container.

This way I never need a venv.

I hope these changes will not make my workflow harder.

Running 3rd party software in containers not only makes dependency handling easier. It also is a security thing. Can you really trust the whole dependency tree below any python code that you want to run? I prefer to keep it separate from my main OS.

No, it fact it will make your workflow easier: no matter what, no script using pip will break your install.

I don't know if I actually have something you would call an "install".

I run 3rd party software in fresh Debian containers. So there is nothing in there I would be afraid to "break".

The container has a system python stdlib, and it's used by apt. If it breaks, everything breaks. I've destroyed it twice, with style I must say.

If any script, or yourself, run "pip install" using admin rights (the default on a lot of container techs), you have a chance to break it.

"Chance to break" what?

It is not as if I manually work in a terminal inside a container that is somehow permanent and valuable.

I just once create a dockerfile with the commands to set up the application I want to use. That's it. From then on, I use that container whenever I want to use that application.

If pip installs something that has a dependancy with the same name than any of the packages you install, it can cause a conflict, because by default, they will override each other.

A simple workflow that doesn't use Docker:

* Exactly as parent comment says, feel free to use the system Python but install packages with the system package manager (sudo apt install python3-numpy etc.). NEVER run sudo pip anything.

* Use virtual environments (either venv or virtualenv) and do not use sudo for this. Keep these within your home directory (either within or next to relevant project directory if that's your preference, or within something like e.g. ~/envs/).

So using Docker is making your workflow harder IMO because it's harder than the above and doesn't offer any benefits.

Not sure what you mean by "no script using pip will break your install". Why would you run pip from a script? Once you've activated your virtual environment, you'd just run pip install -r requirements.txt. And what is "your install" that you'd break? You can't break your system Python with pip because pip doesn't have sudo access to break it. And it can't contaminate any virtual environments except the one that's active because it doesn't even know about them.

While the best practice is to use a venv even in docker images, docker users often believe that because they are into an isolated container, they can pip install their heart away directly in it. Also, docker image default users are often root.

They may also want to install tooling that they want to use outside of a venv, such as a command line tools, and do so outside of venv, not even using pipx.

Or they may just don't know about virtualenv, or follow a bad tutorial/documentation.

Or just do one of those curl stuff | bash that pip install something.

Or think they are in a venv, but they are not (because they don't use -m) and it doesn't work so they try to solve their problem with a bazooka.

This is also true outside of docker.

Bottom line, plenty of people do the wrong thing, and since we can't prevent all of them from doing so, or educate the entire world, the next best thing is to mitigate the impact.

All, in all, I still think out best bet is something like pyflow (see my comment about it). It solves so many problems at once, including this once. I hope it becomes the de factor standard.

I think you misunderstood the original comment. They seem to be using Docker in the first place because they think it will help simplify managing Python dependencies. The answer is surely that their life would be simpler if they didn't use Docker at all, and instead use venv/virtalenv on directories within their home directory. Your comment about how to use pip if you're going to use Docker doesn't apply - the best answer is not to use Docker at all (if you weren't going to anyway).

There's not point talking about pyflow or some other tool to simplify your Python dependencies until you've said me _why_ I need another tool. What's wrong with the plan I listed above (with no Docker)? If I want a new set of dependencies, I just make a new virtal env, activate it and pip install away. If I want to get rid of it, I `rm -r ~/envs/foo`. What could be simpler than that? There's just no problem to solve in the first place.

I actually use a similar setup as yours, but that's no the point. All my comments are in the context of the posted article, which explains why we need such a move.

People don't always use this setup, whatever is the reason. You won't change that.

The next best thing is to make sure if people don't, it doesn't break the world.

> People don't always use this setup, whatever is the reason. You won't change that.

You might change that if they're explicitly asking on HN what the best setup is! For those people you can just recommend to use the same simple setup as us.

Running 3rd party software in containers not only makes dependency handling easier.

True, but it does making using the software together with any other software harder. Any variation on

  foo data.csv | bar

becomes a lot more of a pain when foo and bar live in two different Docker containers.

Eww. I get the rationale, but Python's packaging/import logic is already ridiculously convoluted. The underlying problem is using global mutable state (e.g. /usr/lib/.../site-packages). Giving each application its own package directory is better; there are Python-specific solutions to that (like virtualenv mentioned in the article), but I prefer Nix since it's language-agnostic (it can handle non-Python dependencies too).

I'm also not a fan of all the Rube Goldberg machines being cobbled together to appease Docker's fundamentally broken way of working:

> Distros that produce official images for single-application containers (e.g., Docker container images) should remove the EXTERNALLY‑MANAGED file, preferably in a way that makes it not come back if a user of that image installs package updates inside their image (think RUN apt‑get dist‑upgrade). On dpkg-based systems, using dpkg‑divert ‑‑local to persistently rename the file would work. On other systems, there may need to be some configuration flag available to a post-install script to re-remove the EXTERNALLY‑MANAGED file.

Here a "single-application container" is assumed to contain an entire OS rather than, you know, a single application. That was supposed to make dependency management easier, since we can tailor the OS to just that one application; but based on this article it sounds like even that was a lie. Should we revisit the assumption that Docker makes dependency management easier? No, the OS maintainers are now expected to change their distros, to add another layer to the Rube Goldberg machine.

I also find it pretty alarming that 'apt-get dist-upgrade' is given as an example of something we might want to do to an "official image". What's the point sanctioning a snapshot as "official" if we're just going to immediately, and non-determinisitcally, overwrite arbitrary parts of it based on external-server-state du jour?

While I agree about Nix being a cleaner solution, it will never be adopted, because... it's a cleaner solution.

Nix is pretty much incompatible with everything, because our entire legacy of software stack has state, and we need to accommodate for it.

So we keep adding those hacks because we favor compat above purity.

Even nix has to, after all, there plenty of bash calls in NixOS config scripts as soon as you do something non trivial.

Sure, but I'm not saying everyone should adopt Nix; I'm saying the solution to problems caused by global mutable state is to avoid adding more global mutable state.

If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers, and (b) why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.

> If 'apt-get dist-upgrade' is breaking the packages we pip-installed into /usr/local, I'd ask (a) why we're using two package managers,

debian packagers won't use pip for obvious reasons. dev need pip because it's portable, and doesn't need packagers validation.

It would have been better if something like nix would have been adopted 30 years ago by both community, but it hasn't been so we have several packages managers. It's even worst now with poetry, conda, snap, flatpack and docker.

> why 'put these files in these locations' is being solved implicitly by running imperative, non-deterministic commands to mutate things in-place, rather than e.g. extracting a .tar.gz of known-good dependencies.

pip now uses wheels, and basically does that. Whl files are zip, and pip just unpack it at a know location. The problem is, this location is currently shared with the OS if you don't use --user or a venv. This article address that problems. My other comment also talk about some other solution.

It's very hacky, because, well legacy and all that.

You missed what he was saying.

Nix solves this problem in multiple different ways. One of which is packaging, but also the entire /nix/store is mounted read-only.

A sudo pip install would have no ability to break the system or remove something that was installed by nix.

I get that, but since our entire legacy stack is stateful, there is no way around that unless you nixify the entire stuff.

Immutable works only if the entire chain is.

> our entire legacy stack is stateful, there is no way around that unless you nixify the entire stuff

AFAIK Docker images aren't stateful: they're immutable snapshots, content-addressed by their cryptographic "digest". That's an elegant idea, which works perfectly well for legacy systems without "nixifying" anything (although personally I prefer to generate container images using Nix rather than Docker, but that's besides the point ;) )

Images are typically used as a "base", with further "layers" on top; those layers are also immutable: they use the same content-addressable "digest" mechanism, then get union-mounted together. This lets us use a known-good base, and add our application-specific parts on top, like a /var/www/html directory or whatever.

The problem with creating a layer via 'apt-get dist-upgrade' is that we no longer have any idea what will be in the image (it depends on when we build it, as repos get updated); we have to store/transmit both the old and new versions (the base and our layer); any security audits for the base image are invalidated; etc.

This isn't a "legacy" problem either: it's simply a rejection of the 'base image + layers' concept. It's fine to reject that concept, but there are cleaner ways to do it like debootstrap (which is the legacy solution, around far longer than Dockerfiles running 'apt-get dist-upgrade').

Also, the Dockerfiles controlling this stuff are usually kept in git repos and hence are also immutable. However, there's another anti-pattern of pushing their results to some ":latest" tag, which defeats much of the point of versioning.

These sorts of bodges are everywhere.

Consider the problem actually being discussed in this article: that files managed by 'pip install' may conflict with those managed by 'apt-get'. APT has been able to detect such conflicts for decades (even longer than Nix!). The article even mentions 'dpkg-divert' (part of APT's conflict-resolution system), but only as a patch for Dockerfiles to avoid the new pip-install conflict detection! A slightly different patch (e.g. running 'dpkg-deb -b' on the pip-installed libraries) would not only avoid problems from the new pip-install behaviour, but would also allow APT to tell us if we're breaking things.

I don't see these issues as a 'legacy problem' either; it's more like a WorseIsBetter approach which (a) undermines improvements when they're introduced, (b) propagates throughout the industry as "best practice", then (c) has to be supported forever, even by those who don't care to use it.

Each things you talk about are plateform dependent, but the solution for pip is platform independent and those way easier to implement. A perfect solution you can't implement is of no use unfortunately.

It's not just about finding a technical solution. A lot of packaging stuff is dealing with organizational barriers.

> Giving each application its own package directory is better

One simple way to do that is

cd projectfoo mkdir libs pip install -t libs PYTHONPATH=libs python foo.py

That way you can use system packages like a database connection library together with the local application dependency farm.

I don't understand why dependency management is hard in python.

In powershell, there's a PSModulePath which works like Path: search folders for dependencies. This can be modified within the session.

You import dependencies by specification, which might just be the name or may contain version constraints and/or a module guid.

Granted, powershell only got a package manager in the last decade. Not long enough to thrash out a contentious pep.

Python is an odd duck in terms of dependencies because it gets very integrated into the system; think of it almost like "portable scripted C." There's the pure python packages, which because of how python's search and import process works, means you need a separate library root for each set of incompatible dependencies, and these must be orthogonal.

Then, you have compiled extensions, which have the above issues, plus now you gotta get some binary distribution machinery involved.

Then, you have FFI/so/dll/dylib type extensions, which have the above complexities, plus the linker/LD_LIBRARY_PATH involved.

Then, you have the interaction of site-packages, since python is used by my OSes for scripting.

Why do all languages have to reinvent packaging and typically do it poorly. If instead pip and co could be configured to output an rpm or deb file which would then be installed everything would work much easier. Instead we have to deal with the mess that's virtualenvs, which causes python devs to often be way too specific in version specs, which only causes the mess that virtualenvs are trying to solve.

Seriously the only sane way language package managers should work is by hooking into the system package management, maybe with an option to install in the user directory.

FWIW for portage (Gentoo) there is g-sorcery[0], which can create ebuilds for Emacs (m/elpa) and python packages automatically. Similarly there is also cargo-ebuild[1] which can create ebuilds for rust programs/libraries, including a list of all dependencies with hashes.

I've successfully used cargo-ebuild in the past to create ebuilds automatically, it's a breeze. I'd be surprised if similar tools didn't exist for deb/rpm based distros.

I'm always surprised how many problems people have with python on other distributions. On Gentoo I can install multiple versions at the same time, install libraries for any of those or just a specific python version of my choosing (including pypy), set a default python implementation (pypy) and of course I can pick the version of the libraries I need, too. It's not as flexible as nix, but quite close. Not a python dev though, so YMMV.

[0]:https://github.com/jauhien/g-sorcery

[1]: https://github.com/cardoe/cargo-ebuild

Because many system package managers doesn't support multiple version of a same package easily. Also, there are system (used to) without a package manager.

> causes python devs to often be way too specific in version specs

I don't think anything could fix that, because it is totally up to library developers to decide whether they want to make breaking change or not. So being able to use different exact version of dependencies for different projects are important.

BTW, I find it annoying that many distro package manager doesn't make it easy for user to install package for themselves without root. Even Nix requires you to create /nix IIRC.

This is exactly the thing I'm talking about. Because python allows everyone to install everything into some virtual environment, everyone just grabs the latest version available and puts it into their requirements.txt without checking if that version is even required and never updating that version again.

They then say they need virtualenvs to solve that problem.

The funny thing is the same people who do this then turn around and complain about some instability because some c library changed their API after 10 years.

Also wanted to add that I completely agree with you on package managers not making it easy for user installs.

The requirements of a package manager for application development are different than the requirements of a package manager for a system. The system packages need to play nice and be compatible with all other software intended to run more or less permanently on the same device. A package manager for application development is installing dependencies for building and testing an application meant to be run on other devices.

What are you supposed to do if you're developing an application and you want to use the latest version of requests for Python but some system utility your OS relies upon is using the distro-packaged version of requests, which is older? Your distro package manager doesn't give you the means to install dependencies on an application-by-application basis. Either everything uses it or nothing uses it. So you get virtualenvs. Back in the old C world, you would have just built and installed some .so files into a project-specific deps directory and set LD_LIBRARY_PATH when running your application for testing. virtualenv is doing exactly the same thing but automating it. And, of course, there is now conan for native libraries if you're still doing C development.

Heck, rpm won't even let you install into your own directories if you're lucky enough to get a relocatable package because you have to be root to acquire the global rpm db lock. What distro package manager lets you install temporary dependencies on an application-by-application basis? Possibly Guix?

In this vision of the world, how does one install Python packages on Windows or MacOS? Or even on Linux distros that don't use RPM or Apt?

Let's suppose that some language decides to use the system package manager to install packages. They add support for RPM. Then for Apt. Then for Portage. Then for Pacman. Then for Zypper. If they're not too exhausted at this point and have any spare time between trying to keep things working on every flavor of Linux around, they decide they want people to be able to use their language on Macs. So they add support for Homebrew. Except that some Mac users don't like Homebrew and want to use MacPorts. Some others use Fink. So the intrepid language devs add support for those two as well. Now our poor maintainers are supporting eight system package managers that all have their own quirks and oddities that makes keeping all this working a massive nightmare. And all of this work is not done just once, it has to be repeated for every single package that someone creates for the language.

Oh wait, but users still can't even install language packages on Windows! You know — the most popular OS of all. Oops.

Perhaps this sheds light on why no programming language does it this way. Languages have to allow package installation on not only your personal favorite distro of Linux, but also on vastly more popular OSes — Windows and MacOS — that don't even have standard package installers that they can use. The only scalable way to do this is to allow installation of the language and its internal package manager on each platform and then use a consistent internal system that is abstracted from the platform for installing language packages. Which, not coincidentally, is exactly what every programming language does. Python is a little odd in that it doesn't have a single blessed package manager, and that causes a lot of problems, but using the system package manager is not the solution.

So the argument is because Windows and osx don't have sane (or any) package management everyone else has to suffer? The work having gone into reimplementing python package managers alone would have been enough to implement an interface for every one of the major Linux package managers many times over (there are also several translation tools available as well).

I do concede that Linux package managers should make it easy to install as users as well.

You say that as if fairness has anything to do with it. It doesn’t. Programming languages need to let people install packages on all the major platforms. End of story. Any approach that doesn’t make that practical and scalable is dead in the water regardless of fairness. Even just on Linuxes, there’s enough package manager variation to kill that idea for installing language-level packages with all the different system-level package managers. If you want to lobby all the Linux distros and Apple and Microsoft to adopt a common package installation standard, that would be great but until that happens, that’s just not a feasible way for languages to install packages.

I should clarify that I'm not saying remove the ability to install packages into a venv or even straight into the system.

However this article is about making pip to better interact with the system package manager on Linux. To me this seems like a bandaid solution instead of generating a rpm/deb file and installing this. You essentially get protection against overwriting other package files, but also protection against the package manager overwriting pip installed files for free. Instead we put a hack in place to add another db (a file) to check for both the system package manager and pip.

Clearly I'm missing something but why is nobody proposing this solution at least for this context?

I think it could because there are much more packaging format other than rpm/deb. What about Arch Linux, Gentoo or even BSDs? It is easier to maintain packages in the same way across platform.

As I said before you can still provide the other method for installing packages, but at least you cover >90% of all Linux installs (and I'm an arch user).

The thing is simply if I need to install system packages I want them to show up in the system tools. The problem with pip and co doing their own thing is it never shows up in the system tools. That's the same on windows, I don't see it those in the software manager. As a user/admin that means I now need to keep track of one more tool to see what is installed on my system.

Another obligatory plug for Nix, which officially supports Linux and macOS, and seems to work on Windows (Cygwin, WSL), BSD, GNU/HURD, etc. too. It can run system-wide, or limited to an unprivileged user's home dir.

Nix markets itself as a "purely functional package manager", but after using it for years I dislike this characterisation. I think of Nix as a build tool, like Make, but scalable & composable enough to "build" an entire OS (or fleet of cloud services, if you use NixOps).

In this context, we can build Python interpreters with our desired packages 'baked in' to their site-packages dir:

    rec {
      nixpkgs = import <nixpkgs> {};

      my-scientific-python = nixpkgs.python3.withPackages (pyPkgs: [
        pyPkgs.matplotlib pyPkgs.numpy
      ]);

      my-game-dev-python = nixpkgs.python3.withPackages (pyPkgs: [ pyPkgs.pygame ]);
    }

These are similar to venvs, but they're immutable and created atomically.

We can build our own bin/ directories with arbitrary contents, e.g.

    nixpkgs.buildEnv {
      name  = "my-binaries";
      paths = [
        nixpkgs.gcc
        nixpkgs.valgrind
        nixpkgs.shellcheck
        my-scientific-python
      ];
    }

This results in a folder containing a bin/gcc, bin/valgrind, bin/shellcheck, bin/python3, etc. where that python3 binary has our desired packages available.

If we want to make an "application", e.g. to "install" on our system, we don't need to expose these underlying details; e.g. there's no need for any 'global consistency'. For example, we can write scripts which "bake in" references to their dependencies, e.g.:

    writeScript "hello" ''
      #!${my-game-dev-python}/bin/python3
      import pygame  # This is guaranteed to be available, thanks to our shebang

      # We can splice in references to any file/directory we like

      import json
      with open("${./my-config.json}", "r") as f:
        config = json.loads(f.read())

      import subprocess
      subprocess.run(["${nixpkgs.firefox}/bin/firefox", "http://news.ycombinator.com"])
    ''

When we build scripts like this, those ${} splices will be replaced by the paths to real files/directories built by other Nix builders. If we add the above script somewhere on our $PATH, e.g. called "hello", then we can run it via the command "hello", and it will start up a particular Python interpreter with Pygame installed, then launch Firefox on Hacker News; yet Python and Firefox themselves will not be "installed" (i.e. they won't be on our PATH). Other scripts and tools are free to use their own Python interpreters if they like.

We can do the same with any language. The 'writeScript' builder used above is pretty trivial (writes the given string to a file with the given name, and sets its executable bit); we can use more complicated builders, to do any compiling, testing, etc. we like. The 'runCommand' builder is a general-purpose way to run Bash code, e.g.

    nixpkgs.runCommand "my-pip-env"
      {
        buildInputs = [ gcc gnutar gzip python3Packages.pip ];
      }
      ''
        mkdir "$out"
        pip install -t "$out" my-favourite-package

        wget "https://example.com/source.tar.gz"
        tar xf source.tar.gz
        cp source/my-extracted-file.txt "$out/sacred.txt"

        mkdir -p "$out/foo/bin"
        gcc -o "$out/foo/bin/runFoo" source/foo.c

        # etc.
      ''

Nix actually has built-in support for downloading tarballs, checking out git repos, etc. so the above is just illustrative (we should also split up these separate-looking steps into separate builders, for better caching)

I agree Nix sounds very powerful and interesting. I've been wanting to try it out several times, on the other hand I don't really have a use case for me personally.

One downside of Nix compared to, e.g. Make, is that it's quite heavy-weight; e.g. we can't just run it as a standalone binary.

Despite this, it can still be used in small, gradual ways. Some nice starting points:

nix-shell as a #! interpreter, e.g.

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p gnuplot -p "python3.withPackages (p: [ p.numpy p.scipy ])"
    import numpy as np
    ...

This is a standalone .py script (not a .nix file). When executed it will run nix-shell, which will set up a temporary environment whose PATH contains gnuplot and a python3 executable with numpy and scipy in its site-packages (the '-p' arguments). It will then run the script using the python3 command (the '-i' argument), as if the #! line had been '#!/path/to/that/particular/python3' all along. These dependencies will come from Nixpkgs by default, which is one of the largest and most up-to-date package repos in the world (see top-right of https://repology.org/repositories/graphs )

Another nice "baby step" is again to use nix-shell, but this time we write a .nix file defining all of the development tools we want to use, e.g.

    with import <nixpkgs> {};
    mkShell {
      packages = [ gcc gnumake valgrind shellcheck emacs ];
    }

If we call this file 'shell.nix', then running 'nix-shell' within that directory will start a shell with all those packages available in its PATH. This is useful when we don't want to install a bunch of tools system-wide (or, as we get more specific and explicit, it lets us use different versions of tools for each project).

It's too late now, Python has been destroyed by its own packaging. Burn it to the ground, then burn the ashes, then bury them. Then start a clean rewrite of the language. We will agree never to speak of this episode again.

I don't think it's that catastrophic. I think they should have forced a hard cut with Python 3, but adoption there was hard enough already so maybe they made the right choice.

Another part of the issue is that it makes system packages worse. The system Python works fine if, and only if, you use some kind of external package manager. Then ensuring that the libraries you have work with all the applications you use is up to the package manager, which works okay.

If we burn that down and force virtualenvs, system packages become less optimal. Packages from package managers will either have to bundle their own libraries (meaning duplication and larger package sizes), or we have to have a "system virtualenv" where we manage those packages. A "system virtualenv" is basically what we have now, except there would be more steps to actually be able to use it.

What we really need is for OS package maintainers to take a hard line that people should not be installing packages into the system Python without using apt/pacman/yum/dnf/whatever. Attempting to run "sudo pip install $thing" should print an error like:

"Installing packages into the system Python without using yum is unsupported. You should use a virtualenv or alternative tooling to prevent potentially system-breaking incompatibilities. See https://example.documentation.site.com for more information.

Add the --i-dont-care-if-im-unsupported flag if you're sure you know what you're doing and want to proceed anyways."

And I really think it should come with a scary and long flag like that. It empowers a lot of engineers to make a case that they need to get away from that pattern quickly. Pip is packaged (at least I know it is for Ubuntu), so the package manager can definitely add code for that.

I don't think we need to stop people from shooting themselves in the foot, but there are not nearly enough warnings that you shouldn't be doing this. I don't think there are any currently, even though I think it's basically unanimous at this point that it's technically better to use virtualenvs. I don't even think it's something that linters check for.

Reply on news.ycombinator.com