Poetry – Dependency Management for Python – v1.0
github.com
github.com
Now we use pyenv with poetry to run all our projects in Python with venv in project instead of decided by poetry default settings. We changed the settings with:
poetry config settings.virtualenvs.in-project true
Now everything is the way we wanted, with Pipenv many times faced issues that the project worked with requirements.txt but not in Pipenv. Didn’t have this issue moving to poetry yet.
Also with emacs support and automated deployment tools the predictability of poetry made our team easy to work with and publish Python application modules.
Our developers use mac, windows and Linux for development and each one have different location of venv with poetry, if we start gunicorn or uwsgi they may make mistakes. With predictable venv location all can work the same way from within project directory.
* VS Code configuration so much easier as you can point to the correct env using a relative path in settings.json and check that in
* Predictable paths to execute tasks / build
* No outside of development tree vens that you forget to clean up or forget to deactivate
* No different to node_modules, .git or similar when it comes to "polluting" the local project tree as you can .gitignore it
As well as ignore config in codecov, black, flake8, mypy and so on...
poetry config settings.virtualenvs.in-project true
poetry config virtualenvs.in-project true- I tried using the recommended installation method of
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
- Using the above method, but replacing `| python` with `| python3`: The installation works. But trying to invoke poetry gives this error:
/usr/bin/env: ‘python’: No such file or directory
- Manually changed shebang in `$HOME/.poetry/bin/poetry` to `#!/usr/bin/env python3`: Now poetry runs successfully.
I mean, it works, but it is fragile (shebang probably needs to be manually reset at every update), not beginner-friendly, and not ergonomic. Installing using pip3 has the same issue. In comparison, pipenv is much easier to install. I was hoping this issue would not be present for poetry v1.0, but unfortunately it is.
Being able to rely on python being python 2 and python3 being python 3 is pretty useful
My recommendation is to either create an alias for `python3`, or even better use `pyenv`: https://github.com/pyenv/pyenv
If poetry did nothing but simply the process of publishing and updating python packages, it would still be an amazing tool.
I still don’t feel hungry for new dependency management solutions (requirements.txt + virtualenv works reliably for me) but I do like the idea of refactoring setup.cfg and setup.py for simpler PyPI publishing.
pipenv never felt quite right. poetry has felt really close.
I'd love to know if anyone has made this jump and their experience.
Before that I tried to use pipenv, but it's been too buggy and unpredictably slow with installs. I've always had bad experiences with virtualenv etc., and mostly stayed away from Python development due to that. With Poetry that has changed immensely, as I now have the reproducibility in environment I'm used to from Rust/Ruby/Node.
All that said, I think there is still quite a way to go for Poetry. E.g. I had (hard to reproduce) bugs where if the install was canceled, I had to purge the whole virtualenv, as it was unable to recover itself. I'm pretty opimistic about the project as a whole though!
Yes, I haven't encountered any of the slowness in dependency resolution that Pipenv has with Poetry. It does still feel a bit slow compared to package managers from other languages, but not in a buggy way, so I would attribute it to not being that optimized yet.
[0] https://github.com/python-poetry/poetry/issues/571
edit: gah, still hitting baffling errors:
$ poetry env use /usr/local/bin/python3
[NoCompatiblePythonVersionFound]
The specified Python version (3.7) is not
supported by the project (^3.7).
Please choose a compatible version or loosen the python constraint specified in the pyproject.toml file.
$ poetry env use /usr/local/bin/python3
[NoCompatiblePythonVersionFound]
The specified Python version (3.7) is not
supported by the project (>=3.7).
Please choose a compatible version or loosen the python constraint specified in the pyproject.toml file.
Having lived more in the JS ecosystem for the last several years, my ideal workflow would be a copy of how the Yarn package manager works:
- Top-level dependencies defined in an editable metadata file
- Transitive dependencies with hashes generated based on the calculated dependency tree
- All dependencies installed locally to the project, in the equivalent of a `node_modules` folder
- All package tarballs / wheels / etc cached locally and committed in an "offline mirror" folder for easy and consistent installation
- Attempting to reinstall packages when they already are installed in that folder should be an almost instantaneous no-op
PEP-582 (adding standard handling for a "__pypackages__" folder) appears to be the equivalent of `node_modules` that I've wanted, but tools don't seem to support it yet. I'd looked through several Python packaging tools over the last year, but none of them seemed to support it yet (including Poetry [0]).
The only tool that I can find that really supports PEP-582 atm is `pythonloc` [1], which is really just a wrapper around `python` and pip` that adds that folder to the path. Using that and `pip-tools`, I was able to mostly cobble together a workflow that mimics the one I want. I wrote a `requirements.in` file with my main deps, generated a `requirements.txt` with the pinned versions and hashes with `pip-compile`, was able to download and cache them using `pip`, and installed them locally with `piploc`.
Admittedly, I've only tried this out once a few weeks ago on an experimental task, but it seemed to work out sufficiently, and I intend to implement that workflow on several of our Python services in the near future.
If anyone's got suggestions on better / alternate approaches, I'd be interested.
[0] https://github.com/python-poetry/poetry/issues/872
That's basically what a virtualenv is, the python version of a node_modules directory. You just need to explicitly activate/deactivate to move in and out of that environment, instead of it being based on the directory tree. And the requirements file is equivalent to the "dependencies" section(s) of package.json
Just browser and JavaScript silence the errors and might result in things cannot be seen. This is not the case with Python “No errors must pass silently unless explicitly silenced”.
I guess the node_modules approach is fine for Node, since the native binding story is quite bleak and no very people are deep into it anyway. But for Python this would be a complete no-go given how pervasive extension modules are. I don’t know what went through Ian Bicking’s mind when he first designed virtualenv, but so many aspects of it just reflects that tight cross-language integration Python is so good at (the bin/include/lib structure, copying/symlinking binaries into place, setting up env vars to trick foreign build systems into working, etc.). I wouldn’t want it any other way even if I could go back in time and involve in its original design. Well, maybe a few minor improvements, like offer a more beginner-frieldly interface than activate scripts, or more heavily promote running the binaries directly, but the design is just… correct.
pip wheel -w ./wheelhouse -r requirements.txt
Then you “lock” and it builds a dependency tree with the newest version of everything that satisfies your loose requirements.
Then you can choose to install dependencies from your lockfile, which is exact frozen versions OR you can re-run the lock later, which will again attempt to update everything to the latest versions that satisfy your loose constraints
Alternatively you can build a docker image using Bazel (for reproducibility) and use tools like Droot to chroot into the image then run.
https://cx-freeze.readthedocs.io/en/latest/
It's worked pretty well for us on both Linux and Windows environments.
Anyway, the solution is simple: put
black = { version = "*", allows-prereleases = true }
in your pyproject.toml file. Or use --allow-prerelease as suggested by others.
It just says that he doesn't like the pipenv cli, and some decisions made, but doesn't say why or how poetry is different.
The only thing it goes into detail with is a bug in pipenvs dependency resolution, and how their resolver avoids that bug.