A Poetic Apology: Or Why Should You Use Poetry to Manage Python Dependencies
muttdata.ai
muttdata.ai
The reality is that there are tons of Python libraries out there with poorly specified (or just slightly out of date) dependencies, and it's super annoying to have your package manager just give up and say "sorry, not installing that!" when you encounter them. This is an issue when working on applications with lots of dependencies, where I know certain packages will work together despite their setup.py files claiming otherwise.
There are many reasons you'd want to install a version of a library that disagrees with one of your other dependencies...security updates in subdependencies, maintainers who are slow to test against new versions or don't even use their own libraries anymore etc. The answer in these cases shouldn't be limited to "work with the package maintainer to get the dependencies changed, or fork it". Yarn, for example, has a simple way to handle this situation (and a clear understanding of why it's necessary baked into the docs!) (3)
[1] https://github.com/python-poetry/poetry/issues/697
[2] https://github.com/pypa/pip/issues/8076
[3] https://classic.yarnpkg.com/en/docs/selective-version-resolu...
Poetry is much better than Pipenv and... text files with lists of packages to install. I introduced it at my workplace because of that, but I'm terrified that we'll at some point need some poorly maintained 3rd party dep and end up with a hack that was worse than anything we had before (like, a poetry wrapper that modifies the lock file after every operation).
* Poetry for dependency management
* pyenv for Python version
* Black for automated code formatting
The .python-version file automatically sets my Python version. I run black . to format the entire project. I can run poetry add XYZ to add a dependency to the project. poetry publish deploys my open source projects to PyPi.
I wrote this blog post that explains how to use Poetry + Pandas + Jupyter if you're interested in learning more: https://mungingdata.com/python/jupyter-workflow-poetry-panda...
Poetry-like programs have existed for a long time. I used Bundler with Ruby when I started programming in 2012 and was shocked with the state of Python dependency management when I was experimenting with the language a few years ago.
The ecosystem has made huge advances and now mypy is making it even better. It's great to see all these languages adopting all the amazing Scala language features!
I find it way simpler than before, it was worth learning to use those tools.
Poetry is not perfect, mind you, but it's the absolute best we have in the Python world. Whenever I jump into a new project, switching it to Poetry is one of the first things I do so I have a sane working environment for it.
> The only reason to use pip-tools is because you already know it and you don't want to switch (yet).
Actually, I settled on pip-tools _after_ trying Poetry :-)
> pip-tools is squarely worse than using Poetry.
News to me. Like I said in my original comment: "I personally still find Poetry and Pipenv to be a little heavy-handed for my preferred workflows." A tool that gets out of my way and lets me achieve my goals is a better tool for me. Since Poetry can't meet me where I am, pip-tools is better _for me_.
“I prefer pepsi because coke is too sugary.”
“The only reason you should drink Pepsi is if you already have some. It’s a squarely worse drink than coke.”
poetry works well enough but it seems to bring more complexity with it.
When it works, it's damn near perfect.
When it doesn't work, and that usually involves a problem with a downstream dependency that isn't technically Poetry's fault, it's a massive hassle and sometimes impossible to manually override.
But most python packages are not shipped as environments. They are packages to be installed into user's environments. So you shouldn't be locking your environment when you test it, you should be installing it into a fresh environment like a user would do. You can test against specific major versions of your dependencies, but this is something handled well by tox and doesn't need any locking. In fact, locking will only make your life more difficult as you'll be targeting specific minor versions.
Long story short, use pip-tools if you're building docker images or otherwise deploying a tested environment. Don't use it for regular Python packages.
Poetry uses Pip, right? Where does setuptools fit? distutils? What are wheels, eggs?... Where does Poetry fit vis-a-vis virtualenv?...
I started a new job last year at a place that has a large percentage of their codebase in Python after many years of working with OO and relatively "modern" PHP. Of course, taking PHP's reputation online to heart, you can imagine my glee! To know that I would soon be working with a language that has a little more respect, clout, and professionalism!
Oh, how wrong I was.
One of the first things I noticed was the package management disaster, along with the virtual environment and local/global package issues. Just shy of a year later, and I'm absolutely flabbergasted that Python doesn't get utterly roasted online for how bad and fragmented some of the tooling is. I could go on about other frustrations I have with the broader Python ecosystem, but I'll just leave it at the point that Python folks shouldn't cast _too_ many stones in the direction of JS or PHP without knowing what the current state of tooling in those languages are.
I'll note that my other complaint is the quality of many open source libraries are a significant step below their counterparts in PHP, Ruby or JS. Django, boto and presumably the data science libraries are top notch, but after that even heavily used projects like celery lack the level of polish that major projects in other languages have.
For example stuff like dependency confusion i.e. public package with potentially malicious code overwriting private package are impossible in Composer by design.
https://twitter.com/seldaek/status/1359417655762055171
But pip, npm and RubyGems failed to protect against dependency confusion. So Apple, Microsoft and dozens of other companies pulled malicious code.
https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...
We are very lucky to have awesome Jordi Boggiano in the PHP ecosystem.
He is doing gods work on Composer and Packgist.
The above works fine. I honestly don't know what poetry adds aside from a different way to invoke the same tools.
Let's says if one of packages that you depend on decided to depend on some third package. The maintainer of that third package decides to stop maintaining it and gives it to someone else. This someone else decides to inject bitcoin mining into the third package.
How will you know with pip that you are not pulling malicious code into production?
This is super common in the DS/ML space, and until recently, pip was actively unhelpful (i see you have numpy version x, let's install version y silently because you definitely hand review all the transitive dependencies before installing a new package).
Otherwise, I have no idea how to handle cross language dependencies well. My guess would be that operating system package managers ought to handle it, but I never tried to work with something like Debian repositories. Judging by what I've heard from other people it's a lot of pain.
If they don't you are expected to have a working C/C++ compiler set up which is definitely not ideal. At least on linux it's often as simple as installing a build-essentials package. No clue what that'd be like on other OSes since I don't use them.
I think my issues are really with pip and it's dependency resolver. Give me conda every day, it may take a while but it handles compiled dependencies and doesn't break my environment without at least telling me.
And yes, I use docker containers now.
Security by "it hasn't happen yet" is not a security.
But that seems to be an issue with every other tool in Python too.
https://python-poetry.org/docs/configuration/#virtualenvsin-...
Chiefly, the inability of the combination of poetry and pip to create editable installs of a module is a drag. Using "poetry shell" is ok, but many of the users of the module I maintain are accustomed to using pipenv or plain old virtualenvs and I'm as loathe to try to induce them to use some other tool for this as I would be if the tables were turned.
The other more serious issue is that the modules I take care of are internal things in a private PyPI repository, and poetry unfortunately has a few bugs in this area. Working fixes are in PRs that have been sitting for months, and I've attempted to contact what I think is the project maintainer without any response. In my opinion, that's absolutely fine; this is a volunteer effort, I certainly don't expect the volunteers to do heroics to support me, and I can come up with some way of getting the patches that I need into the tool.
But at the same time, the ideal tool for dealing with Python packaging is going to need to be able to coordinate this sort of stuff with the pip maintainers, and to have some mechanism to support corporate users and all that boring stuff. I want to be enthusiastic about this one because it definitely seems to knock off a lot of sharp corners in python packaging, but I'm not sure if its got legs.
I'm still optimistic that these things will get fixed, but it's not going as smoothly as I'd hoped.
There's a second issue (this one isn't cert related) where repos that return a 403 when a given package doesn't exist causes the solver to abort, which is problematic if your private repo is Artifactory, since the solver interprets this as an auth misconfiguration and won't fall back to the primary repo.
Based on some comments in the threads for those PRs I wondered if maybe the delay in merging was that some changes to the way repositories are configured, but it'd be great to have a temporary workaround for the time being.
Just checked the commit history of the repo and looks like pipenv is back on track. Do you have any additional context?
I feel like a lot of the issues people put forth as a reason for poetry and pip alternatives are a bit contrived, and more this is how npm|yarn|whatever do it expectations.
#!/bin/bash
docker run -v "$(pwd)":/req -w /req python:3.9.1-slim-buster \
sh -c "pip install -r requirements.txt \
&& echo \"#$(date)\" > requirements-frozen.txt \
&& pip freeze >> requirements-frozen.txt"