Hacker News Reader

Comment by darkwater | Hacker News Reader

Just an en-passe mention (under "Documentation") to the new Bookworm way of handling Python modules which already pissed me off^W^W^W surprised me in Ubuntu 23.04. Basically you cannot install modules with `pip3 install module` (or `python3 -m pip install module`) without using a virtualenv or using the .deb packaged version in the repos. I've got used to it but it was a headache for a few days after the first upgrade I did.

View on HN

> you cannot install modules with `pip3 install module` (or `python3 -m pip install module`) without using a virtualenv or using the .deb packaged version in the repos.

Wasn't that always a really bad idea to do, though, since it was so easy to break your system Python that way?

I'm not saying it was the best way but if you just ran some small 3rd party script or wrote some small Python script with non packaged deps, well, it was easy, fast and usually safe. But forcing this new behavior is surely a good decision.

I'm fairly ignorant of the python ecosystem beyond extremely casual use for a few scripts, how does it break your system to do this?

There's a few ways Python packages can be installed.

They can be installed at the system level, making the package available to any Python program on the system. This can get problematic if not all of your Python usage can function with the same versions of all system-installed packages.

They can be installed at the user level, sitting in your homedir instead of somewhere under /usr. This gives some isolation from the system defaults, but may just move the problem down a level - you still have the potential for version conflicts.

They can also be installed at the virtualenv level, and now the package install is specific to Python programs launched from that virtualenv. Upgrading a package can only break stuff in that virtualenv, it can't spread beyond it.

If you mostly use stuff from the standard library and stable packages that don't change their APIs much, you can go a long way without venvs before it actually bites you.

The thing that _really_ irritates me about debian (almost the only thing, actually) is how flipping hard it is to use it as a development environment/system.

It's _great_ as a system, but for the love of $DIETY, why isn't there some sort of way to say: `sudo apt install $ALL_THE_COMPILERS@[latest||version]` ...semi-similarly to the "exceptions" made for frequently updated packages like `firefox`, `chromium`, etc...

I know of `asdf`, but it seems like a big miss that it's sometimes so hard to use debian "properly" (ie: via `apt ...`) to actually develop.

Or similarly, wouldn't it be great if similar to `LSP-language-server-protocol`, there was something like `PSP-package-server-protocol` where `npm`, `go.mod`, `pip`, etc... were all like: "yeah, the debian system packages for python, perl, etc. are great for debian-on-debian", but we should also support pulling an arbitrary project at an arbitrary version and make a best-effort attempt at being able to isolate, share, manage, upgrade packages between them.

You're right, it requires a lot of research and reading to it the "right" way. Also, there are multiple correct option, ie. you mention `asdf`, however I'd recommend `volta.sh` instead.

From my experience, I'd recommend sticking with debian packages for everything if possible (they may be outdated, but might still work).

Altenatively, you can also cheat a bit, use docker to separate your dev environment from your system.

Isn’t Volta only for JS?

I always recommend rtx.

(belated), but thanks for the `rtx` recommendation, I checked it out and it seems like "the right thing". Also keep an eye on the `pixi` tool which got some visibility here recently: https://github.com/prefix-dev/pixi

> Or similarly, wouldn't it be great if similar to `LSP-language-server-protocol`, there was something like `PSP-package-server-protocol

If system package managers could support the same use cases as packages managers for your favorite languages then the latter wouldn't need to exist.

Distribution maintainers have a very different idea of how dependencies should be managed than most developers using those package managers do.

/usr/bin/debian-configure-thingy.py => import foo_foo_foo

sudo pip3 install bar_bar_bar

...depends foo_foo_foo@v4.5.6

...but debian-configure-thingy.py expects foo_foo_foo@v1.2.3

(or import path conflicts or whatever)

Yeah it's insanely easy to break stuff.

You can by deleting the EXTERNALLY-MANAGED file [1]

I know some Python maintainers think users are too dumb to work with system Python, but there are also some packages that are really hard to deal with on a system level if you try installing them in a venv. Plus venv's are dumb (I use Docker if I need more than one environment for specific apps or dev work), since it's another layer of abstraction I'd need to learn just to use Python.

I'm not forced to use virtual environments for PHP, Ruby, Java, etc... why for Python?

[1] https://www.jeffgeerling.com/blog/2023/how-solve-error-exter...

I think that what bothers you with virtual environments is simply the name. A venv is literally just a directory with files in it. Most of the languages you mentioned actually use a very similar approach to installing dependencies, they just don't call it "virtual", they call it GOPATH, node_modules or something similar.

As a developer working mostly with Python I am so glad to have way to install a dependency in a directory that won't break my OS and without having to use docker. I still use docker for lots of things, but not as a way to resolve issues that can be resolved by creating a new directory.

>I know some Python maintainers think users are too dumb to work with system Python,

No, this isn't python maintainers thinking users are too dumb.

This is distro maintainers fed up with false bug reports and broken systems caused by python's inability of versioning the packages fed to programs without venv.

Using --break-system-packages is a far clearer way to do the same thing.

That's a very good change. You should never have a reason to vomit your app's deps into system, far too easy to cause problems on upgrade

I think `pip install --break-system-packages` will still work if you insist on mixing pip and deb.

This is fundamentally a python problem.

python with pip has no ability to version modules correctly in the main package store and feed them to applications by version.

If one system installed package needs numpy 1.26.0 but another system installed package that'll brick your system needs 1.13.5. The result is, "WTF THESE BUG REPORTS IN OUR DEBIAN ISSUE TRACKER".

So virtualenv is now enforced, for everyone because that's python's only mechanism to feed versioned packages that avoid breaking packaged system software

But hey, at least they added the arg `--break-system-packages`

to pip.

LOL

You can also edit/create a pip.conf and add

[global]

break-system-packages = true

This actually improves the user experience for people who acknowledge what they are doing.

Or for people who just don't care, ie. if you're just trying something in a docker image.

Can you use ensurepip to install pip, and then use pip from there? I recently had to deal with missing pip on the last ubuntu LTS version, and that's what I had to do.

Makes some amount of sense to stop novices messing up their python install right off the bat, much like I have on most of my machines.

The way I run my NAS I have a different user for different purposes so just install modules user level with --break-system-packages. Is there a better way?

Do you know if `pip3 install --user module` still works?

Suse Tumbleweed made the same change and it doesn't work there (I use it through WSL if it changes anything)

Reply on news.ycombinator.com