A not so unfortunate sharp edge in Pipenv
0xda.de
0xda.de
- pipenv install updates your lockfile on install exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead
- pipenv's dependency resolution is the slowest among all the tools in this space including poetry, pip-tools, hatch and so on
- pipenv has consistently failed to do its job to simply install dependencies because of issues like the one described in the article where it fails to resolve changes in hash simply because a new wheel was added for the platform
- Tons of longstanding bugs that fail to be resolved in any sort of timely manner like this one: https://github.com/pypa/pipenv/issues/2413
I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead.
As an outsider, Python packaging seems like a shit show, sorry. This bug in pipenv is bad, yes. But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.
Maybe pip-tools is reliable and has a reasonable maintainer, but having to manually create venvs is mediocre UX.
PDM sounds interesting because it doesn't need venvs under the hood, thanks to a recent PEP.
It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.
The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks.
(I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)
That never stopped npm to be way better than all of them...
>The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
So, the "solution" is "just give up"...
So it lands in a similar space to what the parent describes - mostly a good dependency manager.
"This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved.
In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble.
IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...
Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085
> Not sure why "symlinks" would be involved.
If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable.
> What NPM does is leaps and bounds ahead
Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.
Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.
But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management...
(1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...
With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv.
The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.
Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen.
Or maybe that changed recently?
* If the package.json file can be satisfied by the lock file (i.e. if the two files match) then the lock file will be used unchanged, even if there are newer versions available. This is (mostly) identical to the `npm ci` command.
* If the package.json requires more recent versions of a particular package than are available in the lock file then this package will be updated (but other packages will remain the same) and a new lock file will be written. This happens if, for example, the package.json file has been changed by hand, and so the lock file is no longer in date. In contrast, the `npm ci` command exits and fails to install the dependencies.
* Similarly, if the package.json file has new dependencies that aren't in the lock file, it if dependencies exist in the lock file that are no longer present in package.json, then these packages will be installed or deleted and the lock file updated. Theoretically, only the relevant dependency and its dependencies will be affected, but because if the deduping algorithm, NPM may make other changes if it thinks it can simplify the dependency tree. Again, in this situation, `npm ci` will just fail.
There is one other minor exception. The `npm ci` command will never make any changes to the lock file, but `npm install` may still choose to update the lock file with additional metadata (for example if the lock file was created with an older version of npm). This will never change the versions, but it may look a bit messy in the commit history, particularly if different people on a team are using different versions of npm. As a result, I tend to use npm ci as my default anyway.
However, as a role of thumb, if `npm ci` is able to successfully install dependencies, then running `npm install` will always install the same dependencies. The only difference is when the package.json file disagrees with the lock file, in which case `npm install` will probably do what you want, unless you want some very specific results in which case you still have the npm ci command available.
(This has been true since at least npm v5, although I think there were some bugs in the initial implementation.)
npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm.
I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.
I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards).
But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.
But that's more a fault of the library ecosystem and culture, than the design of npm/yarn
Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.
I’m having difficulty finding a reference on this, could you elaborate?
To name some issues i've had with poetry(and keep having):
- Updating packages does not properly take effect sometimes. It will sometimes keep using old code, and only deleting the entire underlying venv fixed it. In this instance, I was using a wheel package locally. - It has incredibly poor UX and will just vomit raw python errors when you use it slightly incorrectly. - There's so many bugs in its handling of CLI args and getting a typical python None error is very common - It is ridiculously slow and doesn't cache things properly
I could probably go on but I'd have to sit and think a bit about it, and that's pretty much the opposite of what I want to do: forget all about python and poetry.
"Slow" doesn't just mean "slightly annoying", mind you.
I was once tasked to properly package up an Airflow installation. Not being a Python guy, I tried doing it with pipenv because we were using it for other projects.
It was impossible to resolve the dependencies because the resolver would routinely throw a timeout error. In the end I gave up and switched to Poetry. That was still a bit slow (I think Airflow just has a messed up dependency graph...), but it wrote the lockfile in a reasonable amount of time and just worked thereafter...
e.g. in a recent project the entire package configuration is this pyproject.toml file:
[project]
name = "vera"
version = "0.0.0"
requires-python = ">=3.10"
authors = [
{ name = "Doxin" }
]
dependencies = ["moderngl==5.7.3", "pysdl2==0.9.14", etc etc]
[project.optional-dependencies]
dev = ["pre-commit==2.20.0"]
[tool.setuptools]
packages = ["vera"]
python3.10 -m venv venv
source venv/bin/activate
pip install -e vera[dev]
Environment variable tweaks are for local-only specific fixes, not for reproducible/distributable configuration.
At my work, we use environment variables all the time for configurations, e.g. for telling a kubernetes pod where to find the database. In your opinion, what is the better option(s)?
Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration, as environment variables are designed to be short-lived, memory only.
A good old configuration file is the best. You can version it, distribute it, it's explicit, possibly structured, easily documented.
That doesn't mean that there is no room for environment variables, but these should be for local-only hacks and tweaks.
I totally agree with this.
> A good old configuration file is the best.
But the issue with this is that you often don't know what your configuration is ahead of time. Sometimes it is only generated just before execution of your code.
You can encrypt sensitive configuration fields with tools like `mozilla/sops` [1], which will reach out to your KMS or secrets manager of choice to encrypt/decrypt sensitive fields on the fly.
This way, you are minimizing the splitting of state across secrets managers and your code. Your configuration is stored at code, code reviewed, and versioned, which has devops and security benefits of its own.
I say this as someone who has done an unfortunately large amount of work in the python packaging and python package distribution space and still get bitten every few months
`npm install` doesn't do that anymore since at least a year
I was burned by pipenv before (naively trying to use it because a certain prominent member of Python community hyped it up, and I hadn't known he'd gone off the rails). I can find no redeeming qualities to it whatsoever, and it scarred me enough that I don't want to touch other newer tools (poetry, pdm, whatever) with a 10ft pole.
I find solace in pip and virtualenv just working and not trying to be too clever.
I pin all my immediate reqs, and if there's a conflict with indirect deps (which happened maybe once or twice), I figure out the version I need and pin that manually.
For packaging I rely on specialized tools that do just that (build, twine), and don't need one-with-everything spaceship of a tool.
I also develop in Node and have suffered much more grief by npm. I lost count of the number of times where I had to rm-rf node modules and npm cache, rerun and hope for the best.
* greater PEP compliance with respect to poetry (and no upper boundary constraints on packages)
* support for scripting
* parallel installation
Here is the opinion of somebody way more qualified than me:
Pip allows your requirements file to omit versions. That means: you can manually add your root requirements into your requirements.txt. Then you can
pip install -r requirements.txt
pip freeze > rewuirements.txt.lock
pip install -U -r requirements.txt
and it will update your dependencies. If anything breaks, you can uninstall it. If it all works out, you can lock again.
This way I can stay within pip tools and speed up package installs. One negative is having to manually fix dependency conflicts. But that is a price I'd be willing to pay for ease if development and deployment.
Edit: Also note that the article makes an excellent point concerning supply chain attacks which pipenv would have successfully mitigated in this particular case.
Unfortunately they create another learning curve, another leaky abstraction layer, another point of failure, etc etc...
manually adding root dependencies. What if you forget?
Not locking. What if you forget?
Locking invalid versions. What if you don't check all is compatible first?
How do you handle dev requirements? requirements_dev? but then how do you lock?
The good thing is: you got rid of dependency walking and slow locking. As with everything, there is a tradeoff that I was willing to take.
I'm pretty sure you need an extra step, since pip freeze will put "==" constraints in the lock file. I do the same thing, but I have this extra step to replace "==" with ">="
sed -i "s/==/>=/g" requirements.txt.lockThis is why I don't use the lock file to update dependencies in the first place. You have to use the requirements.txt for updates. The lock is for deploying, or initializing to last stable revision.
requirements.txt also ensures that you only install what you need.
This is why every 3 months or so I completely delete the venv and recreate it.
According to the last comment there it should be fixed:
> I don't think this should be an issue in the current 2022.4.8 release because of the requirement to use index restricted packages for locking. Please advise if you think this is still a concern and we can revisit it.
But apparently it is still an issue (assuming the author did use a recent enough pipenv version)
From the article:
> Overall this is pipenv working exactly as it should.
I don't agree, IMO pipenv should just install from the source package and ignore the newer wheel, maybe showing a warning about the new wheel.
Edit: Just noticed that the author agrees in the end:
> I suppose the behavior I’d like to see here is that if my Pipfile.lock has hashes in it for a distribution, then even if additional distributions are available, pipenv sync should be allowed to continue to install from the distribution corresponding to the “trusted” hash.
I prefer to use https://python-poetry.org/ instead of pipenv, since it has less issues in my experience. But I wonder how it would behave in this situation.
My first instinct would be to rebase the entire project tree onto a new commit that follows the original addition of the dependency. This isn't a great solution, mostly because git doesn't make such an operation easy, but also because there's something a little spooky about a mass history rewrite like that.
- Pipenv?
- Nice deterministic builds
- Dogslow to lock
- Has some quirky defaults (pipenv install also updates other packages)
- Pretty
- 10x locking speed
- Heard (in several places) problems about it failing to deterministically resolve a dep tree in some edge cases, which makes me a little scared (am I going to hit the dep jackpot at some point?)
- Hatch?
- Bawl in a corner?
Thanks in advance.
I'm very happy to answer any questions about it right here or as GitHub issues.