Hacker News Reader

I have yet to come across a scenario where installing a python library as global ended up being an issue in the python ecosystem. You either need it, or you don't. Short of some contrived scenario where a piece of code is not able to utilize a certain package's latest code and you have to juggle two different versions of that package. If I had something like that, I'd fix the code, not juggle old package versions or install all dependencies in my project's sub-folder.

> "For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv."

It's not that complicated:

    sudo apt-get install python-pip python-dev build-essential

    sudo pip install --upgrade pip

Then you can install any python package to your heart's content with:

    pip install PACKAGE

You don't need virtualenv, nor do you need ensurepip.

>"In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup."

I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM. I have absolutely no hassle installing anything in the python/pip ecosystem. Except maybe MySQL and OpenCV, I'll grant you that. But those are separate discussions as they require external dependencies outside of python's packaging.

I've run into several issues with global installs because the Ubuntu maintainers have decided to make changes to libraries I want to use.

Another example happens when compiling scientific libraries that require C extensions. I've seen disagreements about which versions of libraries to link against. For example, I've run into issues when installing scikit-learn via pip because it linked against the Numpy version in /usr/lib rather than the correct Numpy in my virtualenv.

Ubuntu maintainer here. Can you be more specific, please? I'd like to understand if there was a general reason or if we're doing something that we should not be doing.

Appologies for my crappy memory, but the only case where I can remember the package was when I ran into issues with the PIL library being OS-managed and having basic functionality not working, and the answer I found that worked was installing it in a virtualenv. I also went through some chat logs of me complaining and I saw I had similar issues with urllib3.

The main issue is that they're modified at all, so they don't match up with the docs or what people say online, so until you realise that you're not running the normal code and need to run inside a virtualenv it's very confusing.

The only changes made to PIL in Ubuntu in any current release (Precise, Trusty, Xenial) are security fixes: https://launchpad.net/ubuntu/+source/pillow.

> The main issue is that they're modified at all

Not modifying packages at all would also be a major issue. We'd end up shipping dysfunctional packages, or packages with major bugs, or not fixing bugs during the lifetime of a release.

Distributions do a ton of integration work that I think goes unnoticed by users. We routinely find issues and send them upstream - every development cycle. Users don't notice because they'll also find the bugfixes in upstream releases because we sent them there. But upstream releases don't all happen in lock-step, so we end up having to carry some patches to make everything work with everything else.

In the general case, we can't win. We'll always upset someone, which is why I asked for specifics. In general, we will:

(0) Avoid changing anything in a stable release, so users don't have things changed on them - except for (1).

(1) Fix bugs (including security bugs) as they are reported. Because of (0), this generally means that we cherry-pick fixes, which does cause some divergence from upstream. But if we don't do this, then other users complain either about the bugs or, if we pull everything into a package in an existing stable release, about things being changed under them.

(2) During development of a release we fix for integration issues, since every package in the distribution is expected to work with every other package - and (though with some exceptions) we generally only ship one version of each thing in a release. This means that we sometimes need to patch things (we prefer cherry-picks from upstream, but sometimes we don't get a response from upstream in time for our release schedule) in order to make everything work.

But in the specific case of PIL, we appear to not have had to do any of these things, except for some security fixes.

For urllib3, it looks like we had to make some modifications to fix security issues with regards to TLS and certificate verification in Trusty, and one bug in Xenial related to broken IPv6 square bracket handling, and that's it.

I don't really see how we can do anything different here. Should we have ignored the security issues and shipped vulnerable packages? Or should we have upset many more users of the stable release by bumping behaviour under the feet of existing deployments?

I'm not a Linux distro expert, I'm not even a python expert, all I noticed is that my code didn't work with the OS-managed PIL/urllib3 and installing the versions from pip in virtualenv fixed things. I don't know the reasons, just that was my experience. Maybe it was just the OS versions being old, I don't know.

The maintainer approach works well for everything you maintain, it just sucks for everything else, including installing out of repo software or writing your own software. It's made particularly bad by Python's packaging system which defaults to global installs.

FWIW, I lay this one more at the door of Python than I do Ubuntu, you guys are just dealing with their packaging system and I think that "Python packaging is a good now" is just ignoring the shitty parts.

Frankly it's easier for me to just always run my code inside a virtualenv than it is for you guys to make some drastic changes to make my life marginally easier, when I don't even really know what the problem was. It's just that this is a non-obvious thing to do for someone who hasn't been in the python ecosystem for long.

It sounds like all you needed was a newer version. Most distributions deliberately try to ship a version that was current at the time of release and don't update it later except for bugfixes. This is for the user stability reason I gave above. This is what a distribution release is, and why many Ubuntu users prefer the LTS release over every six-monthly release.

If this is the case here, then this isn't a Python-specific problem at all. Developers have a choice: use the chosen distribution release version, or if that's not new enough, then find another way. For Python, one (commonly accepted and good) answer happens to be virtualenv. Many choose, because of their particular circumstances, to go all virtualenv and ignore distribution packages altogether.

Can I guess that you used an Ubuntu LTS, so you effectively opted in to packages that are older but don't change often?

Some solve this by never using a distribution version at all, which is fine. But understand that this is the nature of what a distribution is, and applies to the entire ecosystem, not just the Python one.

It's possible I had an old version, but I was trying to do super basic things with PIL on 14.04 LTS; literally just cropping a JPEG and things were failing.

Do I have this problem in Java/Node/Go/Rust? No, I don't.

The reason I don't have this problem is that dependencies are not installed globally, they are defined per-project and often statically linked in, there is no such thing as a conflict between two of those projects.

> It's possible I had an old version, but I was trying to do super basic things with PIL on 14.04 LTS; literally just cropping a JPEG and things were failing.

It sounds to me that you hit some kind of straightforward bug then, rather than some systemic issue.

Here's a common scenario: you maintain more than one project, and you can't just simultaneously upgrade e.g. Django for all of them (or literally any library where an upgrade may be non-trivial).

Yes, eventually you'll (hopefully) upgrade all your projects to the newest and shiniest versions of all your dependencies, but if you need to maintain some semblance of stability, that's not always immediately possible/practical.

I'd rather use docker over venv in that case. My host python is for horsing around, when something is to be deployed I document and isolate the dependencies in a dockerfile. venv would only solve half that problem and only for python, not including outside delendencies.

Docker doesn't work if you don't have root.

You need to have access to the docker socket, if you're using sockets to connect. That's not necessarily only root.

https://news.ycombinator.com/item?id=12291092

True. I also work in a HPC context where that's a problem. That's actually a place where I had to hack together something with venv in order to use an output analysis with a python that's compiled differently from the main Fortran executable. But that's quite an edge case.

This is actually wrong. You can be added to the `docker` group. On Docker for OS X you also don't need root.

In general if a tool requires root to run, of course a system administrator could make some changes to the system to allow a user to run it (adding the user to a privileged group, making the binary setuid, etc).

But that's not the point. The point is, if I'm on a system where I don't have the ability to make system-wide changes (run docker as root, add myself to a special group, etc), then I can't run docker on that system.

If the solution to packaging is "I'll use docker", i.e. I will need essentially an entire OS as a dependency of my application, this shows that Python packaging is broken. More sane packaging system allow to define dependencies per project, and only install packages globally when required

Not everyone is running debian flavored os, in fact not everyone is running linux.

for windows, in powershell

     (Invoke-WebRequest -UseBasicParsing "https://bootstrap.pypa.io/get-pip.py").Content | python -

but generally in that case, one doesn't need to install ensure_pip, and can use that. And one doesn't need to install pip at all, since it is an optional come-with

but in most cases, the `curl url | python -` or equivalent solves the problem

> I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM

Please avoid making such assumptions about other people. It's just a little step from there to ad hominems.

To make my point more clear:

I love Python and 'm used to Python/PIP, but when I had to work with NodeJS and saw NPM, I immediately thought: These people got it right. Not the language, mind you, but the package management!

As long as you're using Python 3, pip and pyvenv are bundled with Python. Pip is also included with Python 2.7.9 and later for Python 2 users.

Having pip installed by default is really nice, so that it consolidates people to use pip unless they have specific use cases e.g. Anaconda users. However, pyvenv/virtualenv isn't that great for the reasons the poster mentions, specifically that you have to set up something instead of just installing in a local folder and referencing that. I do think that pip is more mature and in some ways is better than npm, but in the respect of where it installs modules, it's definitely behind (I do realize that this is also a Python issue wrt where it references modules, but installing modules global by default is a pip issue. At least they're working on doing more local installations by default[0]). I shouldn't have to worry about where my modules are installed in general, and if I'm working on multiple projects with different package versions, I shouldn't have to think about virtualenvs because if I mess up and forget to activate it, it may or may not give me strange errors during various parts of development.

0: https://github.com/pypa/pip/issues/1668

I've also found venv to be a pain if you're trying to set up your editor to execute scripts in the virtual environment. Npm project? Build and run. Python project? Figure out how to make VS Code activate the environment first.

That is one of the things I really appreciate about Node/npm. Drive space isn't a concern particularly, since packages aren't gigabytes in size, so having them installed locally is fine. The biggest gripe is you have to include `node_modules/` in your `.gitignore`.

NPM is also a pain in the neck - I've had to wipe out the node_modules folder many times because it got in a bad state when adding dependencies.

Other than that, I agree.

In my experience, the only time this happens is with `npm uninstall`. If you only `npm install` stuff forever, node_modules seems pretty stable. Disk usage, however, is another matter.

I'd say Conda [1] is as straight forward to use as NPM, and handles virtual environments (better than virtualenv) as well.

I don't really see the issue with creating a virtual environment in the case of specific dependencies, and otherwise dealing directly with the global environment (which really isn't that different from e.g. NPM and JS, when you think about it).

[1]: https://github.com/conda/conda

Composer in PHP-world works even better than NPM, while being similiar in some ways.

conda is way better than plain pip (although it is still compatible with it), and possibly better than NPM. It's just not well known enough yet.

I've always argued that npm is nodejs's killer feature. Semver is suggested and used almost everywhere. Setting up the development environment for almost any project is just `npm install` from within its directory. Dependency hell isn't a show-stopping issue: it's perfectly fine for libraries A and B to have sub-dependencies and to depend on different versions of C. No longer do I have to wrestle with things when I try to install a project that relies on a different version of what I already had installed on my system. (Sure, there's some gains to be had to updating packages to consistently use the latest version of C, but when we're talking about libraries on the order of kilobytes, that really shouldn't be a blocker issue stopping something from working.)

It almost seems, based on TFA's risibly misguided "dumb idea" link to npmjs.com, that the author thinks python's story is better than node/npm's. It's called "separation of concerns", kids, and it's a good thing, that one part of a script doesn't have to worry about what a different part of the script is doing. The fact python never got it to work correctly is python's problem; node handled it from the beginning.

DUB for dlang is heavily inspired by NPM.