Hacker News Reader

Comment by drekipus | Hacker News Reader

I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than

    python3 -m venv .venv --prompt="foobar"
    . .venv/bin/activate
    pip install -r requirements.txt

Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but I've never come across it as a python dev, and the moment I try to user/understand these tools I feel like they're against the "keep it simple, stupid" vibe that python gives me.

There's auto-venv for the truly lazy (like me) but other than that I don't rely on any of these Env/requirement wrappers or tools in any of my projects (virtualenv-wrapper for work but that was in their setup guide).

Is it a legacy thing?

View on HN

First of all some years ago pip wasn't really solving dependencies.

Then even with the new dependency resolving improvements it's way worse than poetry's in my experience.

Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file.

And finally the user interface is much more modern. `poetry shell` is way simpler than `source venv/bin/activate`. You don't even need to run `poetry shell` because there's `poetry run`. And to install a new project is just `poetry install` instead of `python3 -m venv venv; source venv/bin/activate; pip install -r requirements.txt`.

Not to mention that it's way easier for me to understand where the depenency conflicts from console output compared to even the newest pip versions.

> Also the requirements.txt file tends to get cluttered with non top level dependencies that make upgrading your dependencies an herculean task. Instead on poetry you define your top level dependencies in your pyproject.toml and all the actual pinned dependencies will be compiled into a poetry.lock file.

This has now been solved [1] in pip, so no third-party tool is needed. Put your pinned dependencies in constraints.txt and your top-level dependencies plus the line "-c constraints.txt" in requirements.txt.

[1] https://pip.pypa.io/en/stable/user_guide/#constraints-files

Constraints are not the same thing. Poetry locks the entire dependency closure. You can install with “no-deps” option, but then you have to specify specific versions of every dependency in requirements.txt

What do you mean? If you do "pip freeze >constraints.txt", that locks the versions of all installed packages, no matter where they came from.

As an example, let's create a venv and install some older versions of Django and its dependencies (current versions are 0.4.2, 3.5.2 and 4.0.6)

  $ python3 -m venv env1
  $ ./env1/bin/pip install sqlparse==0.4.0 asgiref==3.5.0 django=4.0.0

...and also Flask just to complicate the constraints file for the example:

  $ ./env1/bin/pip install flask

Lock all dependency versions in constraints.txt:

  $ ./env1/bin/pip freeze >constraints.txt

Create a requirements file that specifies just "django" and references the constraints file:

  $ echo '-c constraints.txt' >requirements.txt
  $ echo 'django' >>requirements.txt

  $ cat requirements.txt 
  -c constraints.txt
  django

  $ cat constraints.txt 
  asgiref==3.5.0
  click==8.1.3
  Django==4.0
  Flask==2.1.3
  itsdangerous==2.1.2
  Jinja2==3.1.2
  MarkupSafe==2.1.1
  sqlparse==0.4.0
  Werkzeug==2.1.2

Now we can create another venv with the exact same versions of Django and all its dependencies (but not Flask or its dependencies) using just pip and the requirements file:

  $ python3 -m venv env2
  $ ./env2/bin/pip install -r requirements.txt 
  $ ./env2/bin/pip freeze
  asgiref==3.5.0
  Django==4.0
  sqlparse==0.4.0

That’sa lot more than just

    poetry init
    poetry add x y z

Then later

    poetry install

And maybe

    poetry remove y

> And finally the user interface is much more modern.

I hear you on the pinning, but "UX" concerns like this drive me batty; I just use shell functions for this:

    venv() {
        . ${VIRTUALENV_FOLDER}/${1}/bin/activate
    }

    gitsu() {
        branch=$(git status | head -n 1 | cut -d ' ' -f 3)
        git push --set-upstream origin ${branch}
    }

    git_add_conflicts() {
        git add $(git status | grep 'both modified:' | cut -d ':' -f2)
    }

It's so much easier to add a little function in your shell than to write a new tool.

But I would prefer something that just works without me having to do that. Chances are it'll also save me work in areas I'm not aware of yet.

Oh me too, I love when a tool solves problems I didn't even know I had, until I have them, I look for a solution, and poof Vim (or whatever) has already solved it in a coherent way.

If every tool were like that that would be one kind of world, but we live in one that's more interesting. There's some Mark Twain quite like "the world owes you nothing; it was here first", or maybe the Buddha saying "it's harder to soften the earth than it is to wear sandals". I like to think that sandal-making is a core engineering skill, whether you're a carpenter or a computer engineer.

Are you guys actually using either poetry or venv on production?

Separately, I'm curious to hear examples of where dependency-clutter actually caused a problem.

We use poetry just for development, but run Docker containers in prod. When the image gets built we just create a requirements.txt (poetry export --format requirements.txt --output requirements.txt), copy that into the image, and pip install. Because this is built using the poetry lock file, it'll always be exactly the same unless we specifically update something with poetry.

I used to work at a place that was just using requirements.txt files that only included our direct dependencies. There was a project that needed updating after not being touched for a couple of years. The requirements.txt didn't change, but when we built the project again, some of the transitive dependencies used a newer version, and a bug was introduced from one of those updates. A bunch of time was wasted tracking down the issue, pinning the old version of the transitive dependency, and figuring out the damage caused by the bug.

As a result, the requirements.txt was changed to also include transitive dependencies. We had vulnerability scanning on our code, and it found a severe issue with one of the transitive dependencies, but there wasn't a version of that library with the issue fixed yet. Time was spent looking into this to see how we could be impacted. As it turns out, it was a transitive dependency for a library that we no longer used and removed from the project months ago. When you create your requirements.txt by running pip freeze > requirements.txt, you don't have an easy way of knowing which library requires which transitive dependency.

There's ways you can fix this using multiple requirements.txt files, but at that point it's a lot easier to use poetry, especially if you want to keep your development dependencies separate.

Venv, yes. My Dockerfile's build stage creates a venv and installs all our dependencies (some of which need compilers because they are source wheels that need to link with C++ libraries). Then I copy the venv over to the release stage and install only the release dependencies.

Maybe that could work with a `pip --user` install, but at least the venv guarantees that all Python packages and data will be in one self-contained folder.

The main problems with just "pip install -r requirements.txt" are that you might get a different result the next time you run it if a (transitive) dependency has released a new version, and that upgrading dependencies is an error-prone manual task. Both pipenv and poetry try to solve these problems.

Upgrading dependencies is also a way for me to resolve issues, so I don't fight it.

This is solved in pip nowadays with constraint files [1], so you can put your top-level dependencies in requirements.txt and all the transient ones in constraints.txt:

  pip install -r requirements.txt -c constraints.txt

[1] https://pip.pypa.io/en/stable/user_guide/#constraints-files

That's a feature, you should be using the latest minor/patch release when developing. If you really need to lock a dependency to a specific version it is possible, but you shouldn't be doing it. If a dependency is constantly breaking on minor/patch releases you shouldn't be using it.

I think it's a lot of javascript developers who end up having this problem, and all I can say is that the python package ecosystem is not the same. Please don't lock your dependencies to specific minor/patch versions, and please don't use so many dependencies that it becomes tedious to deal with changes. Especially when you're writing a library. If you're locking to anything other than a major release or a minimum minor/patch revision when producing a library things have gone very wrong.

I really couldn’t disagree with this more. Reproducibility is far more important than integrating unknown bug fixes, and upgrading dependencies should be a manual process during which you review the changes and verify that they don’t break your system.

My time and the time of my peers is too valuable to be debugging minor incompatibilities between a library 5 layers down the transitive dependency stack because versions aren’t pinned. It’s too important that if I go back to a 3 year old checkout of the code that it still runs and isn’t a wild goose chase of running down bugs and library incompatibilities.

You can always upgrade overzealously when dependencies are pinned, falling back to the workflow you’ve described, but the inverse isn’t true. If your tools don’t allow you to pin, you’re signing yourself up for churn at unknown and unpredictable times.

> My time and the time of my peers is too valuable to be debugging minor incompatibilities between a library 5 layers down the transitive dependency stack because versions aren’t pinned. It’s too important that if I go back to a 3 year old checkout of the code that it still runs and isn’t a wild goose chase of running down bugs and library incompatibilities.

That happens very rarely with our dozens of transitive dependencies. What are these packages that cause trouble often?

Nothing causes trouble often, because the class of problems is eliminated by pinning.

Still, it doesn’t have to happen often to be a serious hindrance to either developer productivity or your business. Imagine trying to roll out a fix for a production outage only to find a random transitive dependency has started breaking your build. Now your outage is extended to the duration of debugging and fixing an unrelated and avoidable problem.

>That happens very rarely with our dozens of transitive dependencies

Because you don't do ml. My requirements.txt is useless in 6 months because the transitive dependencies all have incompatible versions of common libraries by now

All we do is ML and image processing. We use huge libraries like PyTorch, ones with insane packaging like OpenCV, and things are so stable I haven’t felt the need to introduce version pinning yet.

Maybe you’re using recent less-stable projects? I found that the main ML packages rarely break things without months of deprecation warnings.

> That's a feature, you should be using the latest minor/patch release when developing.

You’re not wrong, but developers still need precise control about which version is pinned so they have a chance to figure out whether a bug is in their code or a regression in a dependency.

> If a dependency is constantly breaking on minor/patch releases you shouldn't be using it.

A more common example would be a dependency of decent quality, but there will still be bugs because there’s no such thing as bug-free software. And when the inevitable issue comes up, it’s super useful to have a switch that lets you pin down the regression and helps you figure out whether it’s on you or on the upstream project to fix the issue.

> Please don't lock your dependencies to specific minor/patch versions

Even if you allow a version range of "*" in your dependency file, it may still be a good practice for your app to have the resolved version number pinned in a dependency lock file, and even commit that lock file to version control.

> Especially when you're writing a library. If you're locking to anything other than a major release or a minimum minor/patch revision when producing a library things have gone very wrong.

You’re referring to the dependency file, not the dependency lock file, right? I couldn’t agree more. At the same time, it may still make sense for some libraries to have a lock file during the development process, and even commit that lock file to version control. You just can’t include that lock file in your release.

I don't agree. Repeatability is more important. Every developer working on a project should have the same version of each dependency. The testing server should have the same version. Every deployment should have the same version.

I don't want my tools "helpfully" upgrading me to a different version, I don't care if it's "minor". It's different and that's a potential source of heisenbugs. Any change to dependencies should be an explicit action and it should be a commit in SCM.

By all means show an ugly warning message to nag people to upgrade but computers live to serve us, not the other way around. The moment you start devolving these decisions to a computer you've lost control of your core competency, which is ultimately what code is running on the machine.

This simply doesn’t work on a large, shared codebase. Locking dependency versions is a must. You can’t just roll the dice that no breaking changes have been introduced by dependencies.

pip freeze > stable-reqs.txt ??

https://github.com/pypa/pip/issues/4732

This thread is just about pip freeze not including hashes, and while it's true that would provide a lot of reassurance, you can still pin transitive version dependencies w/o them.

This kinda works, but it isn't a cross-platform solution.

You don't need virtual environments when using npm or composer or cargo.

They also create lock files with hashes by default.

pip still doesn't use lock files. It's subpar compared to other ecosystems.

I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly.

If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program.

Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.

> You shouldn't need to use lock files if you're using semver properly.

I think this issue is unrelated to semver.

I've had so many issues over the years with Python packages, even very polished libraries like Flask and Celery.

For example, you could install something like Flask 2.0 but end up with a major difference in versions of its sub-dependencies depending on when you installed it.

That's because Flask has its own dependencies defined like this:

        "Werkzeug >= 2.2.0a1",
        "Jinja2 >= 3.0",
        "itsdangerous >= 2.0",
        "click >= 8.0",

The above means installing Flask 2.0 could one day in the future install Jinja 4 or Click 10 unless you lock your entire dependency tree.

I've also had all sorts of things break because Flask installed Jinja 3.1 which wasn't a problem 6 months ago when Jinja 3.0 was the latest release. I've also had cases where installing a specific version of Celery in the past worked but failed in the future because it didn't lock one of its sub-dependencies down well enough (vine) which caused a breaking change.

This stuff happens all the time and it's a nuisance. I never experienced issues like this with Ruby and other languages that have the idea of a lock file built into their package manager. IMO it's a desperately needed feature that should be built into pip.

Semver promises are insufficient in a world where supply chain attacks are increasingly common. Pulling untested and invalidated code in at every project build is how that transitive dependency on a package that was taken over for a small window wrecks your development team. You should never be pulling in new code by surprise, it should always be something I’m aware of and signing up for.

The Rust ecosystem is good evidence that locking doesn’t kill semver. Semver is still widely used and has all of its meaning.

> You shouldn't need to use lock files if you're using semver properly.

You mean "if all your dependencies are using semver properly". Yes, you're right, and you've found the problem.

I've experienced multiple minor and patch (according to semver) updates that broke APIs and behavior, and I'd guess most devs have as well.

I think semver makes sense to humans. I can derive a lot of meaning from it when I see n.n.n. But when it comes to the software supply chain, it's just too rickety. Frankly, when you lose customers after a new deploy broke one of your dependencies, "but the dependency author didn't respect semver" isn't an excuse.

I say this as someone who strongly pushed `dependency>=1.3.2,<1.4` until that happened to me. My argument was "security updates", and now I just don't care. The software supply chain is too chaotic, and you have to be defensive against it.

I agree, but they also include hashes.

The difference between virtual environments in Python and node_modules in JavaScript is minor I would say. It’s the same concept. I believe composer has something similar. It’s a directory where packages go and they are not part of the system wide installation.

I agree that lock files are useful and it’s a pity that pip does not offer them.

I just use venv but don’t fix dependencies. We often develop new packages and fixing every dependency’s version sounds like unnecessary effort.

With sensibly written setup.cfg files, we just “pip install” all our packages with no issues. Pip’s dependency resolver has come a long way since 2019.

What do you list in your requirements.txt? If it's just the top level dependencies, you could have versions change of your transitive dependencies which can matter and break things. If it's all dependencies, you lose the context of what you need because you use it and what you need just because it's a dependency of a dependency.

Some projects used to have two files for this, effectively managing a lockfile manually with pip freeze, but then it's nice to have a wrapper around this pattern and that's where the first gen like pip-tools came from, and then stuff like poetry/pipenv is aiming to streamline that (and avoid manual use of companions like pyenv for specific Python versons) even more.

I think the same

Some tools, like poetry do add value to your development process (namely, better management of dependencies)

pipenv really does sound like something for people that are too lazy to activate their envs before running stuff

Totally agree on this.

Reply on news.ycombinator.com