Hacker News Reader

Comment by lazka | Hacker News Reader

The next Debian/Ubuntu releases will no longer allow `pip install` outside of a venv: https://discuss.python.org/t/pep-668-marking-python-base-env...

You can still force it via `pip install --break-system-packages ...` if needed.

View on HN

This makes me so happy. Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Different pipelines would be running in different requirements files. I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) and reprovisioned the whole build cluster resulting in pipelines breaking consistently where they should have been breaking: trying to do stupid stuff.

Personally I only ever use the system python packages on Linux if I can get away with it. Saves a whole world of problems.

> Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere.

Not everyone might like containers, but using them for CI seems like a good way to avoid situations like this, at least when viable (e.g. web development). You get to choose what container images you need for your build, do whatever is necessary inside of them and they're essentially thrown away after once you're done with what you need to do, cache aside. They also don't have any significant impact or dependencies on the system configuration either, as long as you have some sort of a supported container runtime.

Second containers (however controversial they may otherwise be). The official python Alpine container is < 20mb and the slim Debian variants are < 45mb. For many of my Python projects I end up needing to install various system dependency libs like CUDA, libav, libsndfile, etc. I tend to be a "fan" of containers generally and containers seem like the best way to handle situations like mine.

> I revoked sudo privs immediately for Jenkins (I didn't add them in the first place)

If you allowed sudo in your jenkins jobs you're morally barred from blaming python for screwing up the system.

The problem here started right at the hiring manager.

Something something butterfly in China.

The way it's meant to be.

On Linux, you either use the system packages via "apt install", or you use venvs.

EDIT: For context, I've meant "managed" distros like Debian and Ubuntu.

Nowhere in the official Python documentation (where 99% of new python users are going to go) does it warn or even talk about Linux and Debian specific issues like only using apt packaged versions of dependencies. It wasn't even until recent years that pip gave a hint or warning something might break in those setups. The situation with Python on Debian has been pretty bad IMHO with a cloistered group of people saying the status quo is just fine because it works for them exclusively.

"The situation with Python on Debian has been pretty bad IMHO"

The situation with anything has been pretty bad in Debian lately.

I'm all for the minimalistic approach in regards to Python. It's Ok to provide only the packages needed by applications and the core system. For everything else, there's pip.

EDIT: I've meant to say, there's pip inside a venv.

I agree, the problem is if you just 'pip install foo' on Debian, like 99% of python docs and readmes say, it instantly fails. Worse yet if you think you should just elevate to root and force pip install to work you can potentially break the Python install. It's a nasty footgun.

This is fantastic to hear. Hopefully this will be the beginning of a wave of other OSes doing the same.

For anyone on other systems who wants this kind of protection right now, pip has had this available for a few years at least:

    pip config set global.require-virtualenv True

I absolutely recommend doing it. Immediately.

Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break.

I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.

Good. Now we just need to get pip itself updated so it refuses to run outside of a venv, and refuses to run unless invoked with "python -m pip" and we'll finally have something at least half decent.

And don't even get me started about how much better npm is at publishing packages, versus pip's refusal to add the same user friendliness.

PEP 704 is a recent proposal to require a virtual environment by default for any package installer - https://peps.python.org/pep-0704/. Again, you can opt-out if you want.

Hopefully that’s not going to be the case inside a container!

Why? Can overwrite it and even if couldn't making a new venv is just a `python -m venv venv` away.

Venv in a container is unnecessary ritual. It's a container, it has its own entire root filesystem...

You could have a container whose entry point is a shell script that calls multiple Python programs that need different environments, or a multiprocess container that runs multiple Python programs, although I guess you could still address either by breaking down your containers differently.

Reply on news.ycombinator.com