Poetry – Python dependency management and packaging
python-poetry.org
python-poetry.org
Dependency resolution might be a hard problem to solve concurrently, but it would be great, if poetry did not only use a single core, to speed up the process.
As I'm writing that we are coincidentally and unfortunately plagued by random CI failures occurring when Poetry resolves the dependencies: https://github.com/actions/virtual-environments/issues/1343
Wheels and eggs are packaging formats, which Poetry (and pip) installs. You can think of poetry as like yum/dnf or apt and wheels/eggs as like .rpm or .deb files. (You can think of pip as like the rpm or dpkg command, in the sense that it's a lower-level tool but it also installs wheels/eggs, but that analogy isn't perfect.) The existence of Poetry does not obsolete either wheels or eggs.
Neither wheels nor eggs replaced setup.py files - the standard way of building a wheel is to run "setup.py bdist_wheel". You can think of setup.py as like Makefile. You would not say that tarballs have replaced Makefiles.
At least a few months ago, you needed to install an entire compilation suite if you wanted to install a pure-Python source distribution (sdist) that was created with Poetry on Alpine. Somewhat by design, sdists created with Poetry require Poetry to install. Poetry itself is dependent on the cryptography package which, like all like all packages that use native-C extensions, suffers from a lack of musl-compatible wheels.
I see some discussion at https://github.com/pypa/manylinux/issues/37 but I don't totally follow why it died off.
For comparison, with Ruby you use RVM or Rbenv plus a Gemfile with Bundler and you’re done. There’s no debate to speak of.
The seemingly equivalent nightmare that I see in JavaScript inspires me to go use Ember if only because its creator was instrumental in forming the modern Ruby tooling ecosystem, and I hear that ember is equally unlikely to make me want to stab my forehead with a fork in frustration.
https://github.com/rubyjs/libv8#do-i-get-a-binary would like a word about that. You get source or binary depending on your platform / version.
Judging by what I’ve read here and experienced firsthand with the state of python tooling, this seems like it can be a worthwhile trade off
The reason this has never become a problem in Ruby world is because Ruby community is centered around Rails, of which a requirement to deploy on non-x86_64/ARM64 is pretty much non-existent. Ruby itself doesn't even support non-glibc (e.g. compiling Ruby on musl requires a patch).
(I have contributed a tiny bit to the manylinux stuff, so I'm curious where the problems are, but I'm definitely not active in any sense... it does seem like a couple of these issues boil down to "there should be more people spending time on it," sadly.)
This gets pretty difficult when you are required to do it in Docker in order to produce manylinux-compatible wheels. Or at least that is my understanding.
For some of the arches I need to support I just do it on real hardware - but of course the quay.io manylinux images aren't available for those arches. It is a half baked solution.
> Is it that PyPI won't actually accept those wheels and so you'd have to self-host?
Yeah, that's basically what it boils down to. The platform tags are underspecified - especially for ARM, though this is hardly a python specific problem since few people seem to really understand what the differences are in ARM ABIs - but PyPI accepts a subset of even those, which is limiting.
The other difficulty is that the packaging infrastructure for impure wheels that aren't strictly cpython extensions is basically nonexistent. The packaging procedure for a module that imports a .so via ctypes, for example, is basically "good luck".
To digress slightly, another problem is that source distributions for such packages so frequently throw some obscure python stack trace or a "lol no gcc" stack trace when you try to install them in pip. Which to me is an unacceptable way to communicate a problem. Pip's apparent worldview that tracebacks are an okay way to communicate errors is so hostile to end users, and I see that same attitude present throughout the Python ecosystem. I mean the whole concept of venv's as standard procedure is a good indicator the packaging system is fubar. Sorry for the rant.
It's entirely possible that the Alpine user base isn't as big/important as I would think but this seems like something that needs to be solved.
This is the claim that has been made about every "advancement" in python packaging for the last 20 years. Some of us have been kicking around long enough to remember the great saviour being distutils, setuptools, easy_install, eggs, wheels, pip, pipfiles/pip-tools... The truth being that all people have managed to do is produce a horrific mess with an ever growing tower of components each of which never really solves the problem (and now people want me to add "tox" to the mix too!). I have to say that packaging is really the worst part of the python world.
I do my best these days to avoid every component of the python packaging ecosystem I possibly can and use Nix as much as possible, which really Just Solves The Problem.
But because of this, Nix can only have things depend on packages it has provided itself. It doesn't assume anything about the host system apart from the kernel. So yes, installing Nix on an existing Linux system can feel like installing another distribution. But you only ever have to use as much of it as you want to. Everything lives in /nix, and to get rid of everything you just have to `rm -rf /nix`.
NixOS is a full distribution that just uses Nix for everything, including configuration management.
Anyway, recently I discovered miniconda and it seems like the best, most unambiguous way to install python. It is small (in rel terms...), it works like a charm on Windows, and I can just do something run a single command like `conda install pygments` and I'm all set!
Just not sure why I would ever need to look into something like poetry, although I heard conda and poetry can complement each other somehow? (as do pip, easy_install or whatever is called and all the others haha :-p).
Miniconda is (almost?) stand-alone though. It is a minimal install and you admin your packages through the command line.
I'd say, just uninstall conda completely from your system and try from scratch with miniconda [1].
Not sure I understand why you're having so much trouble. It's literally two commands:
$ python3 -m venv venv # Create virtualenv
$ ./venv/bin/pip install pygments
$ ./venv/bin/pip install -U pip setuptools wheelI never got where the virtual env is, what it does and why I need it.
Why can't I just `pip3 install xxx` and be done with it?
Python dependency management is a nightmare, and it seems like it always will be due to the insane number of so called "easy" solutions.
- bundle --path env_directory
- gopath
- any other language's vendoring solution
In Ruby for example you can do "gem install foo" or add foo to Gemfile and use bundler. Those work similar to "pip install foo" or adding foo to pyproject and using poetry.
It was worse and complicated, but for years now it seems like complaining about venv is closer to a meme, since people go away and do pretty much the same thing in other languages. (Or don't have that option in something like C)
The fact that they're a somewhat hidden stateful context is still complicated and confusing.
The fact is when I retrieve a Python project, I still have no idea what command I should run to install the dependencies and setup everything. With Go modules it's just `go get`, and with JS projects it's just `npm install` or `yarn install`
Even if I'm not gonna get deep into python, I like to feel like I understand what's going on. With python it just involves a lot more effort than with other ecosystems, even if you think that these things are easy to learn.
Compare with ruby: `gem install the-gem`. Done. No "virtual envs", no plethora of similar but different tools to do slightly the same thing / whatever, no preconditions, no step by step procedures. Simplicity is important!
Now, I do agree the Python tooling situation is kind of a mess, but there you can still do:
"pip install the-package"
And you're just as good, all the other tools are when you're trying to do stuff harder like:
- Maintain multiple development environments - Publish new packages - Build binary packages - Handle Windows
I think on Ruby you need tools beyond "gem install" to do those as well. The point is how we achieve simplicity while keeping the ability to do those things.
Ruby does have the equivalent of virtual environments, using RVM or rbenv (similar but different tools), to solve the same problems as in Python.
[1] https://github.com/pypa/pip/issues/8511#issuecomment-6651402...
I got used to that workflow with pipenv.
I agree, this doesn't seem like it makes sense to be built into a tool like poetry - it's a little too magic. And I'd have a use case for doing different kinds of runtime configuration (not environment variables) if it had a hook system.
https://github.com/python-poetry/poetry/issues/337#issuecomm...
Most people that switched from Pipenv to Poetry were probably done so during the end of 2018 throughout 2019, of which Pipenv has no releases at all[1] (non-alpha/beta version were only released in November 2018 then May 2020).
There were a rant from few years ago[2][3] regarding other issues (performance, etc.).
[1]: https://news.ycombinator.com/item?id=21781421
[2]: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
What's so bad about it? Well it's bad at being package manager for starter. Here is a quick question. How do you install packages with pipenv with the versions specified in the lock file?
* how do I use it to install i.e. service files, config files etc.? * can I use it to generate Deb/RPM packages?
I'm not sure if packaging applications is as much of goal as packaging libraries, but I felt that it was missing stuff to do the former.
I have a ton of respect for the author. It's really good code and solves a hard problem in elegant ways. But it's a problem that we shouldn't let ourselves have.
It's like a patient talking to a therapist:
patient: I'm really depressed.
therapist: Why do you think that is?
p: Well, I've been having an affair with this woman, and my wife found out about it.
t: How does she feel about that?
p: She's pretty angry at me. It's affecting our relationship.
t: How is it affecting your relationship?
p: Well, she doesn't want to have sex with me anymore, and things have gotten a little weird with the kids.
t: How do you feel about not having sex with your wife?
p: It's depressing.
t: And the kids? How do you feel about them?
p: They'll grow up and understand eventually.
t: Here's a pill you can take every day that will make you feel better.
There are two ways to address this kind of situation. The first way is to give the patient a pill to fix the symptom of depression. The second way is to fix the behavior that's causing the depression.
Poetry (and Cargo and other package/dependency managers) are a little pill you can take to make you feel better about stuff.
But there's another school of therapy. I'm maybe going to sound a little like Zed Shaw here, but there's the "Don't fucking do that" school of therapy.
It's possible to fix the underlying behavior that's causing the pain instead of taking a pill. It's harder and requires more work, sure. But in the long term it is a better, more stable solution.
Since I'm already on a bit of a rant, I'll go ahead and say it out loud: agile is the source of many of these types of problems. You start out with good intentions and then one day you end up married to a thing that was only supposed to be a proof-of-concept, but it got shipped because product team and velocity, and now your life is hell, and that POC now has kids, and you're legally responsible for them, and fuck it, just give me a pill, doctor.
Poetry is a brilliant solution to a problem we shouldn't create for ourselves.
Examples of what I really have messed up if I have to use poetry would be also nice. I have trouble coming up with concrete examples myself.
Finally! A standard for managing dependencies in Python.