We've got about 15 repos, with the largest repo containing about 1575 files and 34MBytes of .py source, 14 current developers (with about 40 over the last 10 years) - and they really are quite proficient, but haven't demonstrated any interest at looking at anything outside pip/virtualenv.
Is there a reason to look at poetry if you've got the pip/virtualenv combination working fine?
People who use poetry seem to love it - so I'm interested in whether it provides any new abilities / flexibility that pip doesn't.
Fuzzy specs == effortless upgrades according to your risk tolerance for a given library (major version for boto3, minor version for pandas, something like that).
Poetry gets you the combination of the two: Let your dep versions float, and easily revert back to a previous deterministic build using the version-controlled lockfile if something breaks.
Lots of people like pip-tools, it would feel a lot more lightweight and closer to pip than Poetry does.
Pipenv exists but... steer clear for a multitude of reasons.
Personally I like that Poetry centers itself around the pyproject.toml standard. I also think that its usability and the enthusiasm of both the maintainers of the users is going to really carry it more into the Python mainstream in the coming years.
vine==1.1.4
urllib3==1.25.10
wcwidth==0.1.7
So, if I understand correctly, with Poetry, we might be able to say, "Keep Pandas at 0.23.4 and sqlalchemy at 1.2.7 but figure out all the other dependencies for us and load them."
Or, even better, "Keep Pandas at 0.23.x and sqlalchemy at 1.x.x but figure out all the other dependencies for us and load them."
The advantage here is security patches in underlying libraries come for free, while we focus on porting code for the really high-level + important Libraries which aren't always backwards compatible (Pandas)
Also - if we want to stick with specific versions, that's also possible with the lockfile - so every library will be exactly the same as the one in a build that worked.
The thing I don't understand - is when I do:
pip install pandas==0.23.4
pandas==1.0.3
pytz==2020.1
six==1.14.0
$ pip install -r requirements.txt
Collecting pandas==1.0.3
Using cached pandas-1.0.3-cp38-cp38-manylinux1_x86_64.whl
(10.0 MB)
Collecting pytz==2020.1
Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting six==1.14.0
Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting python-dateutil>=2.6.1
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227
kB)
Collecting numpy>=1.13.3
Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl
(14.5 MB)
Installing collected packages: six, python-dateutil, pytz,
numpy, pandas
Successfully installed numpy-1.19.4 pandas-1.0.3 python-
dateutil-2.8.1 pytz-2020.1 six-1.14.0
Though, poetry is actually quite good. There are still some things that I wish it had, like plugin support (for example I really miss setuptools_scm) or being able to use it for C packages.
But if your code is pure python it is great from my experience. The dependency resolver is especially good.
> Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. Their syntax and contents is nearly identical to Requirements Files. There is one key difference: Including a package in a constraints file does not trigger installation of the package.
> Use a constraints file like so:
python -m pip install -c constraints.txt ERROR: You must give at least one requirement to install (see "pip help install")
I'm in the same boat as you in that I'd like to keep using pip but the lack of a lock file is very dangerous because it doesn't guarantee reproduceable builds (even if you use Docker).
In Ruby, Elixir and Node the official package managers have the idea of a lock file. That is the only reason I ever look into maybe switching away from pip.
Running a pip freeze to generate a requirements.txt file doesn't work nicely when you use a requirements.txt file to define your top level dependencies.
I've been bitten by issues like this so many times in the past with Python where I forgot to define and pin some inner dependency of a tool. Like werkzeug when using Flask. Or a recent issue with Celery 4.3.0 where they forgot to version lock a dependency of their own and suddenly builds that worked one day started to break the next day. These sets of problems go away with a lock file.
Use setup.cfg to define your top level dependencies. Use requirements.txt as your "lock" file. But even then you won't get reproducible builds across different OSes, or with different non-Python things installed on your machines. Use Docker images to guarantee staging and production will be identical.
This is a nice guide on how to use it: https://www.lutro.me/posts/better-pip-dependency-management
Docs: https://pip.pypa.io/en/stable/user_guide/#constraints-files
You can also, thanks to the weird way requirements.txt works, put the line "-c constraints.txt" in requirements.txt. In that case you don't have to specify it when you run pip.
That should apply the constraints when installing packages. I don't know if there's also a way to validate what's already installed.
> Including a package in a constraints file does not trigger installation of the package.
Maybe I'm not following something but how do you get all of this to work like a lock file in other package managers?
Let's use Ruby as a working example:
1. You start a new project and you have a Gemfile.
2. This Gemfile is where you define your top level dependencies very much like a requirements.txt file. You can choose to version lock these dependencies if you'd like (it's a best practice), but that's optional.
3. You run `bundle install`
4. All of your dependencies get resolved and installed
5. A new Gemfile.lock file was created automatically for you. This is machine generated and contains a list of all dependencies (top level and every dependency of every dependency) along with locking them to their exact patch versions at the point of running step 3.
6. The next time you run `bundle install` it will detect that a Gemfile.lock file exists and use that to figure out what to install
7. If you change your Gemfile and run `bundle install` again, a new Gemfile.lock will be generated
8. You commit both the Gemfile and Gemfile.lock to version control and git push it up
At this point you're safe. If another developer clones your repo or CI runs today or 3 months from now everyone will get the same exact versions of everything you had at the time of pushing it.
The top level dependencies go in requirements.txt and trigger installation of those packages. Everything else goes in the constraints file, which constrains the version that will be installed if something triggers an installation of the package, but it doesn't by itself trigger the installation - it only locks/constrains the versions.
Otherwise pip freeze won't find any dependencies.
So you end up having to run something like this:
pip3 install -r requirements.txt
pip3 freeze > requirements-lock.txt
pip3 install -r requirements.txt -c requirements-lock.txt
That is a lot more inconvenient than running `bundle install` and if you use Docker it gets a lot more tricky because a new lock file would get generated on every build which kind of defeats the purpose of it, because ideally you'd want to use the existing lock file in version control, not generate a new one every time you build your image.
`pip-compile` from `pip-tools` is my go-to for this.
We use requirements.txt + Docker/k8s to lock in the OS. All of the versions of python modules are defined like:
six==1.11.0
sqlalchemy==1.2.7
squarify==0.3.0
What type of dependencies aren't covered by this (I genuinely am a novice here so would love to be informed where this runs into problems)
Second, how do you separate dev dependencies from prod dependencies, and how do you update a dependency and ensure all of its transitive dependencies are resolved appropriately?
pip freeze > requirements.txt
Dependencies and transitive dependencies are guaranteed to be resolved/ensured because we list everyone one of them out in the requirements.txt file.
There are other shortcomings, but those are the big ones.
The dependencies of the dependencies of what you have listed aren't guaranteed to be locked.
For example, let's say for arguments sake you were using celery.
When you install celery, by default these deps will also be installed: https://github.com/celery/celery/blob/master/requirements/de...
Those aren't very locked down. If you install celery today you might get vine 5.0.0 but in X months from now you might get 5.9.4 which could have backwards compatibility issues with what celery expects.
So now you build your app today and everything works but X months from now you build your same app with the same celery version and things break because celery isn't compatible with that version of vine.
This happened a few months ago. Celery 4.3.0 didn't version lock vine at all and suddenly all celery 4.3.0 versions broke when they worked in the past. That was tracked at https://github.com/celery/celery/issues/3547.
Docker doesn't help you here either because your workflow might be something like this:
- Dev works locally and everything builds nicely when you docker-compose build
- Dev pushes to CI
- CI builds new image based on your requirements.txt
- CI runs tests and probably passes
- PR gets merged into master
- CI kicks in again and builds + tests + pushes the built image to a Docker registry if all is good
- Your apps use this built image
But there's no guarantee what you built in dev ends up in prod. Newer versions of certain deps could have been built in CI. Especially if a PR has been lingering for days before it gets merged.
A lock file prevents this because if a lock file is present the lock file gets used, so if you built and included a lock file in version control, then CI will build what you pushed from dev, so the chain is complete from dev to prod for guaranteeing the versions you want. That is how it works with Ruby, Elixir, Node and other languages too. They have 2 files (a regular file where you put your top level deps and a machine generated lock file). A lock file in Python's world would translate to what pip3 freeze returns.
https://pip.pypa.io/en/stable/user_guide/#constraints-files
pip install celery==4.3.0 -c constraints.txt
Where constraints.txt defines exact versions for everything
I don't understand what dependencies everyone keeps talking about (which seems to be a big deal with Poetry) - when you run: pip freeze
It captures every single python module, dependencies as well. Because everything in the dependencies file is listed as: aaaaaaa==xy.z
You are guaranteed to have the exact same version.
We have all sorts of turf wars when someone wants to roll forward the version of a module, and, in the case of the big ones (Pandas) we sometimes hold off for 6-9 months before rolling it forward.
But there is something that Poetry is doing that is better than "pip freeze" - I think once I figure that out, I'll have an "aha" moment and start evangelizing it. I just haven't got there yet.
> [The new resolver] will reduce inconsistency: it will no longer install a combination of packages that is mutually inconsistent. At the moment, it is possible for pip to install a package which does not satisfy the declared requirements of another installed package. For example, right now, pip install "six<1.12" "virtualenv==20.0.2" does the wrong thing, “successfully” installing six==1.11, even though virtualenv==20.0.2 requires six>=1.12.0,<2 (defined here). The new resolver would, instead, outright reject installing anything if it got that input.
[0]: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...
But the old resolver actually installs both packages, even though it should just abort.
1. Packages are downloaded in parallel. This means dramatically quicker dependency resolution and download times.
2. Packages can be separated for development versions production environments.
3. Poetry only pins the packages you actually care about, unlike a pip freeze. One application I work on has 15 dependencies, which yields a little over 115 packages download. pip freeze makes it impossible to track actual dependencies, whereas poetry tracks my dependencies - and the non-pinned packages are in the poetry.lock file.
The rest is nice, but the above is essential.
Unless this has recently changed?
The methodology of specifying your core dependencies, but also having locked version of your dependency's dependencies works really well.
AND you can easily export to requirements.txt if you prefer to use that in production.
“Added 17000332 packages (including is-even) 875 vulnerabilities found, have fun with that info. Yours truly, NPM”
I can publish is-even on PyPI if I want, is that Pip's fault?
pip + venv + requirements.txt doesn’t solve this out of the box while most languages have common tools that do. Either they’ve rolled their own way to manage these things, or they’re rolling the dice every time they deploy.
$ wc -l requirements.txt
118 requirements.txt
alembic==0.8.8
amqp==2.2.2
anyjson==0.3.3
azure-storage==0.36.0
backports.shutil-get-terminal-size==1.0.0
billiard==3.5.0.2
I don't really understand the "Dependencies" thing (Or the difference between dev dependencies/transitive dependencies)- we literally list every single module in our environment, and its version - It's not clear to me what other dependencies there could be in a python development environment.
I do note we have three requirements.txt files, a requirements.txt, requirements-test.txt, and a requirements-dev.txt. So, presumably there is a need for different requirements that you've identified that I don't understand. So there's that.
This is the main complaint, most modern languages have a standard set of tools and flows for achieving this. Python doesn’t, and everyone does it a bit differently, and when starting a new project, you have to hand roll your own flow.
Or, use something like poetry but the python community as a whole doesn’t have a commonly used solution.
You probably have a bunch of scripts that do what poetry does (either that, or you repeat the same commands over and over A LOT).
Switching to poetry might have some initial overhead, but a big upside is that you stop using custom, internal tooling, and use something industry-standard. Importantly, it makes it easier for you to understand external projects (since you're familiar with the standard tooling), and faster to onboard newcomers.
Package management is terrible work. Nobody appreciates it. It's extremely complex, it has to work in enormous numbers of configurations, and very minor errors can have catastrophic impact to security or availability of systems.
At the macro level, this seems like a bit of a self-fulfilling prophecy: if all the senior and principal engineers using Python don't care to take a look at something else, then it's not too surprising that new solutions don't end up sticking around. That isn't too say that there does necessarily need to be a change in the Python community's choice of package manager, but the rationale for not even considering looking at other options doesn't seem super compelling.
Also I can use it for all software regardless of what laguage it's written in, not to mention having to learn multiple half-baked package managers per laguage!
And lastly any non trivial software project will need dependencies outside of the world of its own package manager anyway, so why not go all in properly with rmp/deb & make everything easier for your users & your future self.
Poetry is still rough around the edges but I think the core experience is great and it's well on its way to be a very popular tool.
Before using it at work, we are waiting on waiting on https://github.com/python-poetry/poetry/issues/2610 (alternate repository not getting used for transitive dependencies) and ideally this https://github.com/python-poetry/poetry/issues/1556 (disable SSL verify for alternate repositories)
If you don't use your own PyPi for a bunch of internal packages, it works great imo. One more wish item would be having absolute path dependencies instead of only relative path.
Internal doesn’t mean secure
Switched to poetry for one of my libraries as a test a few weeks ago, noticeably more painless!
One other thing I liked about it is the community, I distinctly remember digging into the setuptools source code once to find something that was undocumented. With poetry, that was one Discord message away.
The poetry lock file also seems to get ignored for git dependencies. Say I depend on package Foo, on branch Bar, as a git dependency. At install time I get revision 1, which gets added to the lock file. Now let's say the head of branch Bar moves to revision 2. If I re-run poetry install, I now get revision 2, even though revision 1 is still mentioned in the lock file. The solution is simple: depend on revisions / tags, rather than on branches (and this sounds like good practice anyway), but it is surprising behavior.
As a fan of pip's simplicity, I hope this won't become the case.
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
curl | interpreter
Which uses pip-tools under the hood. It even has a pre-commit-hook to make sure your packages lock files are up to date