Ask HN: Pip, Poetry, etc. for Python package management? | Hacker News Reader

Ask HN: Pip, Poetry, etc. for Python package management?

11 pointsjzombie24 comments

I was personally using poetry to manage packages, as I thought it was easier to set up than pip, but never liked the fact that poetry doesn't use pip under the hood, as pip just seems more standard.

So, my question is, what do you prefer, and why?

View on HN

Pip.

Generally, reliable python setup is really easy (at least on linux, don't know about other platforms).

1. Don't use the prepackaged system python. You don't want package installs interfering with stuff your distro needs. Build python from scratch from source after apt-get or pacman installing the necessary dependencies for full functionality, then use make altinstall. It will leave the binaries in the major.minor version format without overwriting the python3 symlink.

2. Use get-pip.py to install pip by invoking it directly with the python version you just built. This will alias pip and pip3 to point to the correct version. Optionally, you can uninstall any package manager installed pip for safety. The pip executables should be in .local/bin

3. Never run pip as sudo, so that it install packages in user home directory (.local/lib)

4. If you need version control, use virtual environments.

I started a project, not sure if it's a good idea (but it is helpful to me), which helps to make pip/venv a little easier to work with.

It's a set of Bash commands that wrap pip, basically.

https://github.com/jzombie/pipper - Package Installer and Python Project Environment Runner

Virtualenvwrapper is another convenient tool - all you need after setting it up is type "workon x"

Just checked it out. Yes, that's very similar indeed, and I'll probably draw some inspiration from it.

My use case is to primarily help me understand pip/venv a bit better and do more experiments with bash.

Also have a look at pip-tools. https://pypi.org/project/pip-tools/

Thanks!

I use Poetry because it makes my life easy and works just like Ruby's bundler and npm in Nodejs.

Some people hate it because they forced people to upgrade a while ago. It's childish to dismiss such an amazing tool because of one bad decision on the maintainers part.

Poetry is indeed a great tool. It's what I use the most, and seems the easiest to set up.

Love this question as it highlights pythons largest shortcomming:

Out of all the modern languages they have the worst dependency story by a mile.

Most of the issues with python dependencies stem from:

- Global package installation is the default behaviour

- Libraries sub dependencies must be compatible with other installed library subdependencies

- Very loose and permissive version requirements as a result

- Leading to sub dependencies breaking your program between builds

- No native support for lock files to prevent this

Poetry so far has been the best solution to this systemic problem.

Pipenv. https://pipenv.pypa.io/en/latest/

I use VENV_IN_PROJECT=1 too.

End result is reproducible installs and easily inspectable dependencies and interpreter right next to my source.

Closest thing to cargo or npm ive found.

Pip doesn’t really work. That is, it works sometimes, but instead of getting a global view of the different versions of dependencies, it just starts downloading and installing things and it can get itself stuck in a corner.

Some developers use pip all the time. I have some little projects that I do in PyCharm and I add packages with the IDE which uses pip and it is fine.

I worked at a place where our projects were pretty complicated and pip could not solve them consistently.

My RSS reader YOShInOn uses conda for package management because CUDA “just works” with conda plus you get numpy and the like compiled against the MKL libraries for a nice boost. Conda is slower than it should be, I think it was a serious mistake that they compress packages with BZ2 (I had a system that even installed a local copy of all the CUDA libs with conda packages I made but decompression just takes forever.). Still, the solver is logically sound.

The worst problem of it is that the packages you want might not be in conda, you can get away with pipping in a package or two but long term that’s the road to ruin. If you do want to use packages that aren’t in conda your beat way forward is to build conda packages for them which is usually not hard —- to really get control of the build of any egg packages you need to make wheels out of them anyway which is basically the same.

It is a little ugly but YOShInon has a sidecar application which doesn’t have any machine learning in it but instead links the mastodon API client and other things that aren’t in conda so YOShInOn just runs that executable when it wants to post to Mastodon.

It's too bad that there are multiple package managers for Python that all have different approaches, but it is nice that there are at least ways around that, as you've mentioned (just seems like a lot of complexity to do something that seems like it should be easy; installing/upgrading/removing packages).

Well... It's a slightly more complex problem than you might think it is. For instance, installing "just one more" package might involve changing the dependency which is not compatible with another dependency that was chosen, it might be possible to change that dependency to a compatible version, but half-baked tools like "pip" frequently die in this situation.

Python's problem has been that the community has been too tolerant of solutions that are 90% correct, confusing that with "has 90% of the features somebody might want". There's nothing more time-consuming and expensive than working with code that is incorrect... I mean, the value of a sort routine that works 90% of the time is not 90% of the value of one that works all the time, but 0%. To save 15 minutes of thinking, some people will spend 15 years pushing a bubble around under a rug.

A lot of people are really disgusted with Python because of these kind of problems, in fact the place where we were struggling with Python builds didn't share my faith that the problems could be solved and gave up on the project.

Like this quote

https://www.npr.org/sections/itsallpolitics/2013/10/28/24129...

however, Python really does do the right thing in the end and I'm quite excited that pypi is now publishing the dependency lists separate from the rest of the wheel because that's a key item for a correct solver because you really want all of that info to make the right decision. Back in the Datalogue era I had found that you can fake this because wheels are really ZIP files and you can read a file inside a ZIP file with at most three partial http requests. Similarly you really have to stamp out eggs because you can't know what they depend on without running the setup.py.

Given that the foundational changes have been made it is just a matter of time before we see a package solver for Python that is 100% correct like the solver for Maven.

please stop spreading this FUD

You know the Jehovah's Witnesses came to my door the other day and told us that we shouldn't read anything about them online except their official web site.

I am still using Python for my personal projects, something I wouldn't do if I didn't think Python was worthwhile. I've seen a lot of people who've been burned and won't come back. If we're not going to be honest about problems people are going to keep getting burned and keep leaving.

No need to be sarcastic-by-allegory.

Are you using `pip freeze` / `pip list --format=freeze` to capture the state of a working environment?

What size system are you working on that you get into difficulties in upgrading a subset of project dependencies? It's often easier to upgrade by re-capturing the currently-contemporaneous set of dependencies, (and then diffing the new output of `pip freeze` with the previous one to see what's changed), than it is to manually follow the dependency graph when updating a single component.

As always, YMMV.

pip and virtualenv because I have always used it and have never have run into issues at scale.

Curious to know your thoughts on this, then: https://github.com/jzombie/pipper

Coming from a "Poetry background", this makes my life easier to not have to remember the pip/venv commands.

What's the benefit to one using this over Poetry [which already has all of these commands built-in]?

Are you using it to avoid Poetry for one of the use cases it doesn't handle well? (Otherwise, I don't see why one would use it.)

Primary reason is that it works "out of the box" for Python packages that don't use Poetry.

Of course, it could include a way to "convert" a pip-based package to Poetry, or whatnot.

Can you give more specific examples of what you mean by "packages that don't use Poetry" (AFAIK one can install any Python package with Poetry)?

Like, I know developing a package with Poetry can be painful.

And I also use pipx for use cases like running CLIs installed globally.

Or maybe you mean contributing to a third-party open-source project that doesn't use Poetry and the maintainers don't want to migrate to it for some reason.

I don't see anything like that mentioned in the readme. Maybe it would help to document the intended use cases?

If I come across a package that relies on pip, and I intend to fork it, introduce changes that might affect the requirements.txt file, and potentially submit a pull request to the original repository, I find it more straightforward to stick with pip itself. This is because Poetry, while versatile, doesn't seamlessly integrate with requirements.txt (it can be configured to work with it, but it doesn't write back to it).

In essence, I see pip as the more standardized option, given its inclusion with Python. However, I initially turned to Poetry because I prefer its commands over those of pip. What slightly disappoints me is Poetry's divergence from pip's workflow.

I appreciate your suggestion in the README! At the moment it's definitely more of a personal endeavor rather than trying to persuade anyone to use it... however, it doesn't intend to introduce anything new, it's just a CLI wrapper around standard Python tools.

I like PDM, personally. It's like Poetry, but feels a bit more ergonomic, supports a means of storing deps other than virtualenvs (a rejected PEP submission that basically works like node_modules), and never did that BS Poetry did where it failed peoples' builds until they upgraded or w/e

Perfunctory note: Python is garbage, in part because of its "devops". Also, the ecosystem seems like it's full of trash coding standards. Also, it's slow. I can't believe we are all continuing to use it :smiley_tear:

I originally wasn't a huge fan of Python myself; avoided it for a number of years as much as possible, but have really grown into it lately as I've been doing experimentations in ML.

Have since started using it for other projects basically to just get more familiar with it; and also to take myself rather less seriously and just try to keep things as simple as possible.

Reply on news.ycombinator.com