Generate pip requirements.txt file based on imports of any project
github.com
github.com
This was written 10 years ago when I was struggling with pulling and installing projects that didn't have any requirements.txt. It was frustrating and time-consuming to get everything up and running, so I decided to fix it, apparently many other developers had the same issue.
[Update]: Though I do think the package is already at a level where it does one thing and it does it good. I'm still looking for maintainers to improve it and move it forward.
And 10 years later this is still a common problem!
BTW, this tool is not a dependency manager. Many sibling comments seem to misunderstand the case and promote unrelated tools.
If I find a script/notebook/project with no requirements.txt, I usually know when it was created. Being able to get the versions that were selected back then on the author's machine would be great for reproducibility.
Even if you have all versions as of the time of the last modification to the code, you don't know if the dependency resolution happened at that point in time, or if the environment was set up years prior and never updated.
Nevertheless, this is what you are looking for: https://pypi.org/project/pypi-timemachine/
Agreed that in a perfect world I would never need this, but...
uv pip compile --exclude-newer=2019-12-02
I’ve found pip-tools [1] to be a nice middle ground between `pip freeze` and something heavier like poetry. It’s especially helpful in showing you where a library came from, when a library installs other libraries it depends upon.
One could still have a typo in their requirements.in file with pip-tools, but changes there are much less frequent than imports in any Python file in your proejcf which could be a daily occurrence in some codebases.
Edit: the main goal was to generate optional extras for different entry points in a large codebase, find missing or extra imports, resolve dependencies across multiple repos, and see which files reference the mapped packages. Ideally if you have many internal repos which are not published and you cannot correctly use dependency resolution, then you can generate requirements before you pass them onto something like UV.
(I'm a poetry user myself, but I can see the writing on the wall that poetry is probably not going to be the winner of who ultimately ends up in the PEP.)
[0] https://packaging.python.org/en/latest/specifications/pyproj...
[0] https://peps.python.org/pep-0621/ [1] https://peps.python.org/pep-0020/
Is that correct?
https://docs.astral.sh/uv/pip/compile/
In other words, bring the thinking here. Whether it's new thinking, or decade old thinking, it's well time to converge. We've had decades of trying it like Heinz catsup varieties. Worth trying more wood behind fewer arrows.
Yes exactly
https://github.com/mamba-org/mamba
In other words, bring the thinking here. Whether it's new thinking, or decade old thinking, it's well time to converge. We've had decades of trying it like Heinz catsup varieties. Worth trying more wood behind fewer arrows.
Iff it must be the conda ecosystem, pixi (https://pixi.sh/latest/) is a much better pick.
But conda-style packages (or anything with install time dependency resolution really) also have the issue of being completely untestable. That makes them unsuitable at least for user-facing package management, if you care about your packages not randomly breaking without warning.
I'd rather see every language converge on a single package manager that implements functional package management, i.e. guix or nix. One can dream...
I've found that there really only are two kinds of packages I want to install: those I want globally (e.g. CLI tools like git, git-annex, DataLad, whatever shell enhancing you want, etc.) and project-specific dependencies (anything required to get going with it on a freshly installed system with the smallest amount of assumptions possible).
The former is sufficiently addressed by `pixi global` and other package managers, the latter by pixi projects. Notably, conda environments are a bad fit for both (not global, not really updatable, not automatically tied to a project, not rebuildable due to missing lock files, ...).
You could just as well install it either globally with pixi global, or as a project dependency in a pixi project. In this case, the latter.
> I also have a few shared envs, for quick tests
Fair, I just use temporary directories with pixi projects for that and thus no longer see a point in conda envs for this. It has the added benefit that all my temporary directories are in the same location and easy to get rid of.
> general utils
I would want those to be available at all times, so conda envs aren't fit for that purpose. Instead, I use pixi's global subcommand.
> Poetry for project management.
That limits you to lock files for python dependencies only, unfortunately. In a project relying on browser automation, I would want the browser version to be recorded in the lock file as well. Pixi does that.
---
I still don't really see why you found pixi lacking. It addresses all of your problems, maybe with the exception of globally installing multiple and non-CLI packages. But again, conda isn't any better as you are generally advised not to install stuff into the base env and conda has no other facility for globally installing general utilities.
Mamba doesn't even interact with the official python packing ecosystem... It is purely a conda replacement and conda packaging is not Python packaging (it's a more fundamental fragmentation than choosing a particular tool). So weird choice to not fragment Python dependency management.
If you depend on both conda and standard Python packaging (e.g. PyPI.org) you can use Pixi: https://github.com/prefix-dev/pixi?tab=readme-ov-file#pixi-p.... Pixi attempts to bridge the conda world and the Python package world, so it's possible to rely more deeply on standards based packaging but still use conda tooling when you need it.
It doesn't do anything I couldn't already do with other tools. It just combines those other tools' functions into one cohesive, ridiculously fast, ergonomic tool. I won't say anything bad about Poetry. I used that prior to uv. I don't want to return to it though.
Is this really the case for Nix, or is it actually widely adopted, and this adoption is underreported?
If it's not actually widely adopted, what do you think are the biggest obstacles in Nix's way?
You realize that this project predates uv by roughly a decade, right?
I don't think it is good idea to merrily write 10s of import statements and end up with loads of dependencies.
> Looking for maintainers to move this project forward.
So not sure how maintained it is.
https://github.com/pypa/pipfile "This format is still under active design and development", last commit "2 years ago". I think this is dead.
Pipenv was last updated 10 hours ago. Looks like it's still an active project to me.
But in any case, I'd vastly prefer the standard pyproject.toml over some file specific to mostly one tool.
> This repository contains the design specification of the Pipfile format, as well as a proposed implementation of a parser for the specification which can be used by Pipenv and, in the future, any other consumer (e.g. pip)
It should probably be updated to reflect whatever the current status and goals are.
I have no idea how well it works.
in python are there any libraries where the pip install command is different than import?"
ChatGPT: Yes, there are some Python libraries where the `pip install` command differs from the module name you use for `import`. Below are a few examples of such cases:
1. *`pip install opencv-python`* - Import as: `import cv2`
2. *`pip install beautifulsoup4`* - Import as: `from bs4 import BeautifulSoup`
3. *`pip install PyMySQL`* - Import as: `import pymysql`
4. *`pip install python-dateutil`* - Import as: `import dateutil`
5. *`pip install python-dotenv`* - Import as: `import dotenv`
6. *`pip install google-auth`* - Import as: `import google.auth`
7. *`pip install Pillow`* - Import as: `from PIL import Image` '''
It seems that the project hardcodes the import to pypi names though. So it has a that sweetspot of being more impractical while still being insecure. https://github.com/bndr/pipreqs/blob/master/pipreqs/mapping
If you install this in a company that takes security seriously, you should get warned or fired pretty much.
It's one thing to import code from a random developer on the internet to do your job, we get a pass on that if it makes us more productive, but to import code that helps us with importing code? Red flag.
If the project were to create a directory that maps python namespaces to pypi tools, sure. But it's designed as a "tool" that "automates" a pesky little inconvenience of a missing requirements.txt file. The readme explains in no way how that mapping works or any security considerations.
It doesn't give me the impression that the author or the users would care about auditing the requirements.txt file at all. (Also, what would auditing the dependencies entail? Reading the source code of the dependencies? Doubt it.)
Just use venvs, or install separate Python versions for development e.g. via pyenv. You likely don't want to use the relatively outdated Python version shipped with Debian anyways.
- reproducible python package dependency
- reproducible python version
- package locking
- PEP compliant package management configuration via project.toml
- automatic venv management
- project configuration and management including build scripts in one place
All you need is
- pdm init # starts preparing env
- pdm add package # install package locally, project wide
- pdm install # reproduce
With pyproject.toml being the main config file you can also define build script , run scripts like you can do with `npm run dev` , using `pdm run dev` for example. Also you can define all the development tools and their config along with it.
We used PDM , so it streamlines package and depedency, project management and venv config all in one pyproject.toml - and you can generate requirements.txt and pdm.lock which in turn builds your venv , can easily share with team (just one pyproject.toml file to share) .