Changes to the pip dependency resolver in 20.3 | Hacker News Reader

Changes to the pip dependency resolver in 20.3

pip.pypa.io

76 pointsdi29 comments

Huge kudos to pradyunsg and the rest of the team.

In a world where we focus on the shiny new things, it is so easy to ignore the legacy systems upon which they are built.

This will be the end of a 7+ year GitHub issue: https://github.com/pypa/pip/issues/988

Python is an old language (older than Java) and its packaging systems are often the butt of many jokes. Its fantastic to see investments and improvements in this area. It isnt easy with Python, given that the build process is often executing code "setup.py" so resolving dependencies statically isn't possible without running Python.

Yup. Kudos to the maintainers and volunteer contributors who continue to work on these tools.

As some of the other comments here show, it's a thankless task.

Dependency resolution and separation of direct and transitive dependencies is why I started using Poetry.

https://python-poetry.org/

Same. It does everything pipenv does but it can actually install Ansible.

Pipenv also takes > 45 minutes to lock dependencies sometimes once you get some decent sized deps in there, like pandas and pyspark

Poetry is good, probably the best for Python, but it is not without issues:

- 1.1 is not compatible with envs created with earlier versions.

- The parallel installer in 1.1 is enabled by default but has race conditions and must be disabled.

This pretty much sums up my experience of poetry. Some great ideas, but an astoundingly buggy implementation that shows no sign of improving. We tried it at length in my day job and gave up in the end.

Yikes, thanks for the heads-up. Do you have a link to the first issue? I'm not finding it.

Poetry is still rough around the edges but I think the core experience is great and it's well on its way to be a very popular tool.

Before using it at work, we are waiting on waiting on https://github.com/python-poetry/poetry/issues/2610 (alternate repository not getting used for transitive dependencies) and ideally this https://github.com/python-poetry/poetry/issues/1556 (disable SSL verify for alternate repositories)

If you don't use your own PyPi, for a bunch of internal packages, it works great imo. One more wish item would be having absolute path dependencies instead of only relative path.

Also a big plus is the abstraction poetry provides over virtualenvs. I can just do `poetry install && poetry shell` to enable virtualenv from nothing. This is a nice upgrade over virtualenvwrapper.

Definitely not a fan. :-( Sometimes you know what you want and the package itself is wrong.

For example, I have a dependency that specifies pandas at ^0.25.1, and now I can't install it alongside pandas 1.0+ without forking or getting a PR upstreamed (even though I know it works with all versions of pandas just fine.)

In the node ecosystem, yarn actually recognizes that this situation exists and provides a mechanism to override, along with some good descriptions of situations where it's necessary[1].

[1] https://classic.yarnpkg.com/en/docs/selective-version-resolu...

I agree. This doesn’t seem like it’s going to fix dependency management in python, just replace the existing problems with arguably greater ones.

For example, this seems like _terrible_ behavior. Even more, it’s inconsistent with the philosophy behind the new change that installing dependencies shouldn’t create conflicts.

> This also means that, when you run a pip install command, pip only considers the packages you are installing in that command, and may break already-installed packages. It will not guarantee that your environment will be consistent all the time.

We should all answer the survey about that, because it is really, really crazy.

Conda dies the same thing and it's super frustrating.

The survey is now closed

Also not a fan. I like the fact that pip is relatively "stupid". It means you can use pip-tools if you want proper to have a fully resolved graph, but pip should always do what you want.

Poetry is neat but the Unix philosophy wins, as usual. Each tool should do one thing and do it well. Pip was much better at doing what pip does than poetry is. Not any more.

To hell with the Unix philosophy on this topic. Poetry is amazing, and as a user of python for 15+ years I would much rather have one tool like poetry solve the problem of "python dependencies and packaging", than 25 tools to solve the myriad of sub problems hidden in it.

That's great. It's your choice and I would happily use poetry to contribute to your projects. But why change pip? We can have both ways. Pip+pip-tools has been around for a lot longer and it's been working very well for freezing dependencies. Even better than npm/yarn.

I get you, though pip isn't particularly changing, at least not in the context you seem to be implying? The dependency resolver is a good move, implementation seems poorly conceived though from the sound of the other comments.

Then switch to poetry, why break pip?

The dependency I referenced actually uses poetry, and the reason it's borked is because "poetry add pandas" back when 0.25.1 was current meant effectively adding "poetry==^0.25.1" to requirements.txt, even though the one line of actual code that uses pandas works on pretty much any version I've ever seen.

From what I understand, the issue with the previous behaviour is that, although it enabled you to bypass the issue of that specific package, it could also install conflicting versions you didn't know about.

For your specific use case, wouldn't it be better to have a flag `--override-constraints=` where you provide explicitly overridden constraints only for the packages you're interested in?

I think this is good. However, my impression from trying to install more complex projects is that a main issue is a cultural one: Python package developers are happy to introduce breaking changes if they think they will not break for too many of their users.

For example there are packages which transitioned to being python 3 only without changing the package name, or changing the version major number. This breaks the assumption that packages use semantic versioning, which many tools use. And that's arguably less of a problem for that shiny new startup company which will, with 95% likelihood, have gone bust five years from now, and also not for the FAANG companies, but it is a big problem in scientific institutions which do not have money or personnel to constantly rewrite their software.

PyPI, and many of these projects, predate semver. If you assume PyPI is universally semver, that's a mistake on your end

> In older versions of pip, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, in pip 20.0, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver, instead, outright rejects installing anything if it gets that input.

Shouldn't there be a --force flag of some kind? I know there's another flag, but the name "legacy-resolver" makes it sound like it won't be around forever.

The situation isn't so much asking PIP to revert to an old resolver, so much as it is telling PIP that I know what I'm doing and it should get out of the way.

Do you always know what the legacy behaviour will do though? Nobody stops you from solving this in a more permanent / explicit way: clone repo with too strict version bounds, change it, install from your branch.

The "--force" / legacy option can't tell a difference between "broken solution that will work for you" and "broken solution which won't".

> The "--force" / legacy option can't tell a difference between "broken solution that will work for you" and "broken solution which won't".

But probably it doesn't have to. "--force" implies that the user wants to go on and manage the consequences of what happens. Typing it requires actively adding that flag, which is a conscious decision on the part of the user. If the user wants to check by themselves if it works or not, that could be their prerogative.

So happy that Python is taking more steps to improve its dependency resolution. The previous behavior has bitten me countless times. Also happy that they decided to allow a workaround, since countless projects depend upon the 'broken' behavior:

> Temporarily use the old resolver when necessary. If you run into resolution errors and need a workaround while you’re fixing their root causes, you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver

This is an important improvement to #python #pip dependency management. In the past I would find subtle failures due to silently incorrect package installation. This will cause pip to fail explicitly and let the programmer decide what to do.

For some reason scroll is broken on that site. Using a mouse it barely moves at all. Using a trackpad it is usable but way slower than any other websites. (I'm using FF on macOS if that matters)

Just don't mess with scrolling!

Edit: Fixed on reload.

Why did they break backwards compatibility in a minor release, though? :(

Reply on news.ycombinator.com