Hacker News Reader

Comment by j88439h84 | Hacker News Reader

Poetry is a modern packaging system for Python. It's really good. I hope everyone converges on it.

Poetry is...a little quirky. I've seen it go into infinite loops trying to resolve projects with, like, 10 dependencies. I'm not really sold on it being better than just making a virtualenv and pip installing.

How do other people replicate your venv? How does production replicate your venv?

Pipenv still works for me with everything version pinned.

Create setup.cfg (it is better than setup.py, since it is declarative) listing immediate dependencies, use requires.txt as a lock file (I prefer to use pip-compile from pip-tools).

You pin all your direct dependencies in requirements.txt, run "pip freeze >requirements-frozen.txt" to get a list of the versions of all dependencies (direct and transitive), and then you ship that.

Replicating is just "virtualenv --python=python3 venv && . ./venv/bin/activate && pip install -r requirements-frozen.txt". Am I missing something here? Yes, to be absolutely sure, you need to have the same versions of python, pip, setuptools, gcc (for C extensions), C library dependencies, etc. on both systems that do the build, but that's the case for pretty much any build system out there if you want repeatable builds.

And if you you just don't feel like dealing with that, you can just use Docker (aka "put my machine in a tarball and ship it").

This is worse than how it’s done on most modern languages.

That's why we use Poetry.

yeah, that's what i tried to use for a while too. The problem is that so much of the ecosystem doesn't use it, and the python community fails to converge on a solution here the way other language communities do.

Different commands, but it's pretty much the same in similar languages: Ruby uses bundler configuration instead of virtualenv and gemfiles lock instead of requirements.txt. JS uses npm/yarn and lockfie. Go uses go.mod and does a lot of assumptions instead of virtualenv. (these days, it was worse)

Modern languages do pretty much the same, just have a nicer wrapper for it.

It's not "pretty much the same". Have a separation between Gemfile and Gemfile.lock is a huge improvement over just having requirements.txt. Having good, easy to use commands, that manages this stuff for your in a way that works well for the vast majority of use cases (i.e. bundler), is far better than the ill-defined, ad-hoc, mess that is requirements.txt management.

Gemfile-gemfile.lock is setup.py-requirements.txt.

And yes, as I said, a nice wrapper for management of it is missing in python. The main ideas/approach is the same though.

It’s really not the same. Gemfile lists the direct dependencies of your project with variable specificity of explicit versions. Gemfile.lock lists all dependencies after resolving your direct dependencies and locks them to specific versions. requirements.txt is some bastardization of both of these combined, and setup.py is neither of these it’s for distribution of modules/packages which Ruby has as well, but it’s not your gemfile.

edit: also, how do you manage dev dependencies with requirements.txt and setup.py?

You can put dev deps in extras in their own "dev" block. Similar to how gemfile uses groups. Use requirements-dev.txt for them. Requirements for is a full list of dependencies after resolution and their exact versions. (as produced by freeze) The main difference is that requirements is "as installed in current configuration", while gemfile.lock is "for any groups configuration".

Again, not as polished experience, but the model matches.

But it’s all ad hoc which is problematic. You’re solving it yourself, the tool isn’t solving it. It’s like saying that C is the same as python, you can just write your own garbage collector in C!

> You pin all your direct dependencies in requirements.txt, run "pip freeze >requirements-frozen.txt" to get a list of the versions of all dependencies (direct and transitive), and then you ship that.

What?! Seriously? That is not how it's supposed to work. Requirements.txt is already meant to be the frozen dependencies (like the output of pip freeze). Why change that? I much prefer how pip-tools does it with a new file, requirements.in that contains the unpinned direct dependencies. I thought poetry did the same thing but with pyproject.toml?

Sorry, I was a little terse — obviously you're right that you're not going to ship the virtualenv itself. As other people have said, the intent was that you create a requirements file (and/or maybe a setup.py if you need that functionality). What I meant is that between fiddliness and sluggishness of Poetry, and the marginal hassle of maintaining a requirements file and creating your own virtualenvs, I'm not sure Poetry is the clear winner at the moment.

Thanks. My main point is that bundler, maven (less so), and npm are all better than either venv/pip/reqruirements.txt or poetry. Additionally, it's also better that those language communities have mostly converged on those solutions.

That will probably be solved by the new pip resolver within the next few months.

Did that new resolver finally happen?

The pip SAT-based solver was supposed to land in 2017 or 2018 after a GSoC, and nothing came of it. Is it finally going to use a different heuristic than "#yolo" for picking which version of dependencies to install?

Ya, funded PSF grant https://wiki.python.org/psf/Pip2020DonorFundedRoadmap

Unfortunately, you need to:

- discover it

- install it properly

- run it properly

- make it create a venv with the proper python version

- potentially import data from existing setup.py/cfg

- make sure you are always in the right dir when you use it

- not be on windows (poetry shell doesn't work there)

- IDE integration is not great

- people have to chose between this, pip, pip-tools, flit, pipenv, dephell, conda...

And you cannot use it if you are not "in a project". But not all venv are project related.

> - not be on windows (poetry shell doesn't work there)

Ok that's a bug, probably fixed soon. Otherwise, most of these are true of many popular packaging tools in other languages.

Well, not pip and venv. It's provided on windows and mac automatically. venv forces you to chose the version of python. venv doesn't care if you are in project mode or not. IDE integration is great.

We've recently standardized on using Poetry at JP Morgan Chase for projects in our Organization. We're betting on it becoming a standardized tool everywhere.

I just started up a project using this. It’s pretty good so far. The documentation on common tasks is a little bit lacking. VS Code integration is sort of broken in a few random ways (for example if I try and do a find all function definition it just hangs). I also wish there was a npm style script section in the definitions. I do think it really just works though and I think it’s a great solution compared to requirements.txt.

> I also wish there was a npm style script section in the definitions.

Is that like [tool.poetry.scripts] ?

I don't think so. From their docs: > This section describe the scripts or executable that will be installed when installing the package

I want something such that I can run `poetry run script_name` and it will run that command line in the poetry context. Right now I either rely on my bash history or use a bash script to mimic the behavior of npm / yarn scripts. I want to turn this `poetry run python project_name/main.py` into an alias like `poetry run main` or `poetry run dev`.

That's what tool.poetry.scripts does.

O thats great. So just weirdly worded on the docs / I didn't understand. Thanks!

I echo the sentiments about documentation. It works great but they really need to verbosely document every command option and usage. They seem like they’re trying so hard to keep the website clean and minimal, but docs are not the place for that.

Where is that XKCD about new standards...

Ah yes: https://xkcd.com/927/

Also https://xkcd.com/1987/

Anything with Easy or Simple in its name, isn’t.

SNMP

Reply on news.ycombinator.com