Hacker News Reader

Comment by fb03 | Hacker News Reader

Genuine question: If I am starting a Python project NOW, which one do I use? I have been using pipenv for quite some time and it works great but locking speed has been problematic, specially after your project grows large enough (minutes waiting for it to lock without any progress warning at all).

Should I just upgrade to Poetry or should I just dive headfirst into PDM? Keep myself at Pipenv? I'm at a loss.

Thanks in advance!

View on HN

I have never experienced any issues with a virtualenv and pip.

python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt

Python standard library is great and its a nice languate if you like the syntax but aside from a few constants like Django, Flask, and Pandas, the ecosystem feels like it is slowly turning into a fragmented mess.

> python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt

... bless my `zsh` shell history for these incantations. I don't think I have any hope of remembering it -- probably because of all the old virtualenv incantations!

Kind of agree with pipenv though. It's painfully slow, but it abstracts away having to worry about various requirements files (eg: dev vs prod) and the .lock keeps things consistent.

I use this same setup but with a requirements folder. Inside I have a base.txt, dev.txt and tests.txt where dev.txt starts with '-r tests.txt', etc.

It's very handy for having different environments but doesn't lend itself to a lockfile.

Same, I use two small bash scripts in my projects bin/ directory, "venv-create" for the creation of the .venv/ and "venv-python" for running the effective version of Python from the .venv/ -- this sets environment variables such as PYTHONPATH and PYTHONDONTWRITEBYTECODE and provides a single approach for running project files and packages.

I get versioned requirements files for the project base requirements, and also for each (version, implementation) of python, in case they are changes, and this has proven to be reliable for me.

It's all about finding the minimal-but-complete convenience / ergonomic solution over the, err, inconvenience of packaging. I also marvel at when I attempt to explain these things to experienced programmers, I only manage to convince them 50% of the time at most.

If you're building a package, pip install -e . is preferable to -r requirements.txt. Most projects don't need and shouldn't use requirements.txt. The only ones that do are where you're shipping a whole tested environment out, like a docker image for deployment. And in that case you need to be using something like pip-tools to keep that requirements for up to date.

That's what I use as well. It works great, it's built in and it's easy to use and understand. Only issue is when you upgrade the version of Python you're running. In that case you might need to rebuild your virtualenv, but that's super easy.

I use the same solution to have multiple versions of Ansible installed.

If you need to run multiple version of Python, then virtualenvs might not be enough, but that's honestly not a problem I have. New version of Python, great, let's me just rebuild my virtualenv and get back to work.

One of the most important rules I have regarding working in Python is: Never, never ever, install ANYTHING with the global pip. Everything goes in to virtualenvs.

PDM author here, if anyone want to know the advantage of __pypackages__ over virtualenv-based tools(venv, virtualenv, poetry, pipenv), here it is: The fact that virtualenvs come with a cloned(or symlinked) interpreter makes it vulnerable when users want to upgrade the host interpreter in-place unless you keep all the old installations in your system, which is what pyenv is doing. You can imagine how many interpreters, including virtualenv-embedded ones are on your machine.

You can regard __pypackages__ as a virtualenv WITHOUT the interpreter, it can easily work with any python interpreter you choose as long as it has the same major.minor version as the packages folder.

Use Poetry. It does everything you need and PDM is a bit too new still.

Great. I have heard "anecdata evidence" that sometimes poetry fails to install a combination of packages or something along those lines, did you find any of those shenanigans in your own experience?

(Not GP) Occasionally, but still use poetry.

The issues happen because poetry is strict about version constraints. Pip used to be lax, so some packages could get away with poor/overly restrictive constraints..

Now pip is strict as well, and poetry is also getting a lot more common, so maintainers have had pressure to fix most of these issues.

I've run into this a couple of times. Typically it's when starting a new project and gradually adding dependencies. By default, when adding (poetry add <dep>), poetry adds the most recent version of the library, and in pyproject.toml that becomes the minimum possible version. If you then later try to add something that depends on that library, but isn't up-to-date on the most recent version, poetry will (correctly!) error out because it fails to find a compatible solution. For example, something like this:

    poetry add ingredients  # most recent version: 1.0
    poetry add cake         # requires ingredients, but <1.0

So here, poetry would add ingredients to your pyproject.toml with version = "^1.0.0" in the first step. In the second step, when you tried to add cake as a new dependency, it would say it can't find a version of cake compatible with your pyproject.toml's requirement for ingredients.

The solution is to change pyproject.toml to be more permissive about the version of ingredients, resulting in it downgrading to a compatible version. Or submit an issue and/or PR with the maintainers of cake to bump up the supported versions.

In PDM, add command supports a `--unconstrained` option to relax the version constraints to find a solution. You don't need to edit the file manually.

Another way it happens: I was stuck on python 3.6 for a while and it meant that I couldn't just poetry add pandas because the latest version isn't compatible with 3.6 and it won't look for other versions.

No shenanigans for any library-ish code with few dependencies that targets relatively modern Python.

I've had one or two fundamental version conflicts with a 5+ year old application with 100+ dependencies and a decent amount of legacy stuff. They were a pain in the ass, and the sdispater's stance on not allowing overrides is a pain in the ass. We ended up forking the upstream libraries to resolve the version conflict.

With all of that, poetry is amazing and a huge step forward. I'd advocate it wholeheartedly.

I've spent the last year or so comparing these packages. Initially, I liked poetry, but I began to run into issues related to client cert auth with private repositories. I and several other users who encountered these problems submitted pull requests to address them, but found that it was nearly impossible to get any of them reviewed, much less merged.

Out of frustration, I decided to try pdm instead, and although pdm doesn't handle pushing to pypi repositories, Twine works fine for it, and I have had a pain-free experience using pdm for dependency management.

I like the idea of poetry, but it's unusable if I can't get bugs fixed, and I gave up after nearly a year of trying.

> but I began to run into issues related to client cert auth with private repositories.

Yeah, I think that's the worst thing about private infrastructure in 2021 is getting those certs correctly installed and seen by the software and both sides.

I'm thinking that tailscale or something like it may be a better alternative in the future where authorization is at the link layer before the tcp connect can happen.

As others have said, this is related to how Poetry does dependency resolution. If you say your program uses Python "^3.8", you would expect any Python version in that range to work, but instead, ALL Python versions in that range must work.

If one of your dependencies won't work with 3.9, the dependency graph cannot resolve. The author of Poetry said that this is because Poetry is meant for libraries, where this makes more sense.

Except Poetry's recommendations/decisions around version capping doesn't make sense for libraries. https://iscinumpy.dev/post/bound-version-constraints/

As a Python infra person, I would refine that to:

If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in pyproject.toml regardless.

Poetry is much more focused on managing dependencies for applications than dealing with libraries that have to be used by other libraries or applications. See this deep discussion for some timely/relevant examples: https://iscinumpy.dev/post/bound-version-constraints/

Can we mix Poetry and PEP621 though?

I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.

Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.

It went final this March though. So Poetry might needs more time to think about the transitioning plan.

Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.

I know you didn't mean it that way, but "since 2021"..? :D

I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running.

I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!

I've seen others make the same reference, but I've not understood it.

My package is a Python module and a set of command-line tools based on that module.

Is my package a library? Or an application? Because it seems to be "both".

It also includes several C extensions, using either the Python/C API manually or Cython. I would love to switch to something more modern than setup.py, but none of these new back-ends seem to support this. For example, I just checked - flit explicitly focuses on "pure Python packages with no build steps" "and leaves the hard things up to other tools".

Do you expect other people to add your project as a dependency of their projects, and call into your API? Then it's a library.

If the only expected users of your API is your own project, it's an application.

And yes, if you need to build native extensions, setuptools is still your only real option. It's not great, and results in hundreds of packages that all find their own definition of insanity. But given just how crazy the C/C++ toolchain ecosystem is, especially when you have to deal with Windows, macOS, and the diversity of Linux, no one else has been will to tackle that problem because setuptools already exists.

Following that definition, the project I mentioned is a library.

But quoting PEP 517 "Much experience suggests that large successful projects often originate as quick hacks".

I have another project which started as a command-line tool only, meant for the end-user to build, maintain, search, and update a data set. It was not designed to be called by another program, neither through import nor popen. Following the guideline, I should use poetry.

It then acquired a limited Python API so search functionality could be incorporated in a web app. One of my users paid for that feature. I don't know if others use it.

This turned my package into a library. Does that mean I should switch from poetry to flit?

There seem to be several people who have tackled that problem, including emscons, PyQt-builder, and mesonpep517, which all defer to an existing third-party build system. I don't use or have experience with any of the underlying build systems, and version numbers like 0.28.0 and 0.2 make me wary.

Are there others? Finding such modules on PyPI is not easy! A search for "517" is almost useless. Could there be a new trove category for such packages?

> Poetry is much more focused on managing dependencies for applications than dealing with libraries

The linked article is quite long. Do you have a quick example? I've built lots of apps and libraries using Poetry and haven't run into any dependency issues, and you can specify versions constraints in the same way as Flit, so it's hard to see how they'd be different in that regard.

Edit: I think I see what the issue is--Poetry sets upper version bounds by default via its ^X.Y.Z and ~X.Y.Z syntax and advocates for always specifying upper bounds in their docs.

So, it's not that Poetry can't be used for a library, but you'll have to know that you can, and perhaps should, use different version syntax in some cases (i.e., the standard >=X.Y syntax).

Sold everyone on using poetry a few months ago and now red faced as we have a litany of problems and time waste due to it. We are now sitting on some bleeding edge branch because specific dependencies cannot work at all without some new fangled feature and everyone wishes we were just using plain virtualenv as we had much less problems with that.

You would also not have any real control over what gets installed with just venv. So everyone might think the grass was greener on the other side, but it might not actually be, if you value reproducible environments.

What you've never spent hours trying out n! orderings of requirements.txt until you get the results you want/can live with? You have not lived until you've done this. /s

Dep Hell is real.

In other words, you work with people who have no clue what’s going on.

Thanks, that's good advice!

Is Poetry used a lot by Python devs? What package management a lot of Python devs use?

I mostly code on JavaScript and (obviously) use NPM a lot, and it makes me wonder.

I use it for all of my projects, yes. I haven't really had many problems, just one or two that were easily resolved.

I've been using Poetry exclusively for the past few years. I had a couple small problems with it early on, but lately I haven't run into any issues.

Just go for virtualenv and pip. Virtualenvwrapper to have handy shell tools.

Alternatively, pyenv and pyenv-virtualenv for shell integration and seamless virtualenv activation.

To be fair, I'm not saying there's anything wrong with virtualenvwrapper, just that I've never used it and for my purposes the above solution works well.

This is our setup - fits perfectly, don't even miss `workon`

That's been my thing, but I do nothing of interest.

Maybe 3.11 can make python packaging less of a beautiful disaster.

Just go docker

This doesn’t solve dependency management? All it does is it separates your env and you can install what you need there. But installing with pip is still subject to version incompatibility etc.

Poetry is good. We've used it 3 years in production without issue of large amounts of time to solve the dependencies.

    $ poetry init # to init the pyproject.toml file.
    $ poetry add <packages> 
    $ poetry shell # to activate inside the virutalenv.

Lock in the poetry.lock file after any change to your project dependencies, and other people can duplicate the project using after doing a git pull.

    $ poetry install

I tried them all a few months ago. Poetry was the best in my opinion.

Do you suffer from requiring different versions of the same library, or have some other kind of non trivial dependencies?

This will mark me out as a Luddite but I am still quite happy with a 5 line setup.py and “pip install .”

https://news.ycombinator.com/item?id=29446715

  $ cat setup.py
  from setuptools import setup
  setup(
    install_requires=['arrow'],
    packages=['dogclock'],
    scripts=['scripts/dogclock'])

I just started a Python project, using PDM. So far, I like it a lot. If I hit a show-stopper, no big deal, it's pretty easy to switch.

Pipenv is garbage. Don’t use it.

I’d just use pip and requirements files if you can. It’s doubtful that your requirements are sufficiently complex as to require a more complex resolver, although that depends on your ML needs.

Reply on news.ycombinator.com