Uv: Python packaging in Rust
astral.sh
astral.sh
> Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows.
Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim.
> uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.)
> uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions.
This is great to see though!
I can understand it being a flag on these lower level, directly invoked dependency resolution operations.
While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537
How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
> Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution.
> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.
The living version of PEP 440 has a bit on how pre-releases are handled[1]. The basic version is that the installer shouldn't select them at all, unless the user explicitly indicates that they want a pre-release. Once opted into, they're ordered by their phase (alpha, beta, rc) and pre-release increment (e.g. `.beta.1 > `.alpha.2`).
[1]: https://packaging.python.org/en/latest/specifications/versio...
* Try running `python -c 'import this' | sed -n 15p`
When it comes to package managers, as you noted, the situation is even more ironic, as nicely depicted in https://xkcd.com/1987/.
Early versions of go didn’t solve things well, but go has had this fixed for a long time now. Any arguments about packaging issues are likely related to older out of date problems people had.
The only real problem people have these days is around private non open source packages. You can do it, but you need to learn how to set environment variables.
>>> from __future__ import braces
File "<stdin>", line 1
SyntaxError: not a chance $ LC_ALL=C NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/release-13.10.tar.gz nix-shell -p python26 --run 'python -c "from __future__ import braces"'
File "<string>", line 1
SyntaxError: not a chanceYou will notice that every package management solution from all your favourite languages will have its own set of tradeoffs.
And quite a few where the tradeoffs are minor enough to be well worth some sanity & consistency.
C++ is mostly unmanaged in practice which is obviously bad (though Python is the only "modern" language where I regularly see published projects without so much as a requirements.txt so very comparable). However looking at the actual options for C++, Conan seems pretty clear cut & not awful in itself.
So... not as bad as Python by any measure imo.
Like plenty of people just use pip or poetry and are fine. It’s totally bad faith to claim that this is all orders of magnitude worse
And it's always there. If your IDE didn't generate it your maven archetype did.
What's the source of truth in any random pip project that I should look to as a package manifest? requirements.txt? Setup.cfg? Pyproject.toml? Setup.py? What if they're all there? Do they all agree? Should I cascade or defer to one? Merge the trees? What if none of the above exist? Pip certainly doesn't ensure any does.
As for parsing them, requirements.txt is the most barebones near-informationless manifest out there, and setup.py is init code - good luck trying to automate reading the dep tree reliably without side effects.
SPM's manifests being non-declarative is also super problematic for 3P tooling but again - it's one well defined target. Decisions are clear, if a little cumbersome. There's nowhere near the mess of indecision and reduplication you see in python projects.
Lock files alone are a proven piece of technology that pretty much just works across all modern package managers, and yet the best that pip can recommend to date is `pip freeze > requirements.txt`, which is strictly inferior than what is available in other package managers because there's no hash verification and no distinction between transitive dependencies and explicit dependencies.
That pip still doesn't have an answer for lock files is a sign of a deep problem in the Python package management world, not the natural result of packaging being hard.
[0] https://docs.npmjs.com/cli/v10/configuring-npm/package-json#...
Hopefully uv can make this the default behavior. It seems like the majority of users are not aware of its existence because it’s an optional flag.
Here's what they say about requirements:
> Requirements files are used to hold the result from pip freeze for the purpose of achieving Repeatable Installs. In this case, your requirement file contains a pinned version of everything that was installed when pip freeze was run.
Here's what they say about constraints:
> Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. ... In terms of semantics, there is one key difference: Including a package in a constraints file does not trigger installation of the package.
> ... Write a single constraints file for your organisation and use that everywhere. If the thing being installed requires “helloworld” to be installed, your fixed version specified in your constraints file will be used.
> Constraints file support was added in pip 7.1. In Changes to the pip dependency resolver in 20.3 (2020) we did a fairly comprehensive overhaul, removing several undocumented and unsupported quirks from the previous implementation, and stripped constraints files down to being purely a way to specify global (version) limits for packages.
This sounds like something vaguely similar to a lock file, but they seem to intend it to be used globally at the organization level, and they're certainly not pushing it as the answer to locking dependencies for a specific project (they specifically recommend requirements for that). Maybe you can use it that way—although this Stack Overflow answer says that the 2020 update broke it for this use case [1]—but even so it doesn't answer the fundamental need for lock files unless everyone is actually on board with using it for that, and even the pip maintainers don't seem to think you should.
[0] https://pip.pypa.io/en/latest/user_guide/#requirements-files
[1] https://stackoverflow.com/questions/34645821/pip-constraints...
ironic that pip freeze literally generates a requirements.txt with ==
constraints.txt certainly did not exist back when I was doing python.
Conversely Ruby packaging has been a solved problem for a decade, when the python community has been extremely resistant to conceptually similar solutions for the longest time on strange ideological grounds, and came around only recently.
In short, python-dev leadership just doesn't care. Can't fix that with technical solutions.
Age is part of the problem, but it's not the whole problem. Ruby manages to have a package manager that everyone loves, and Java has settled on a duopoly. Meanwhile Python has someone making a serious attempt at a new package manager every few years.
> Meanwhile Python has someone making a serious attempt at a new package manager every few years.
Many People have many attempts at many things in many communities. Doesn't mean they are important. Heck, vim and zsh have also dozens of different package-managers, and they have less use cases and problems to solve then python.
All the more reason it should have been so much better if it has already had time to learn from & iterate on past efforts.
Don't get my wrong, there's plenty of examples of projects ignoring well-known best practices (see the community backlash at Cargo's approach to name spacing or Homebrew loudly ignoring community feedback on FS permissions for 10+ years), but at this stage in 2024 you'd think we'd have gotten a little further than we have.
I feel uv should provide a way to install different python versions to truly cover an end-to-end tool. The current approach of searching for existing virtualenvs of conda envs helps but I was hoping to completely remove the need for another package/dependency manager.
(Taken from the docs)
If a --python-version is provided to pip compile (e.g., --python-version=3.7), uv will search for a Python interpreter matching that version in the following order:
- An activated virtual environment based on the VIRTUAL_ENV environment variable.
- An activated Conda environment based on the CONDA_PREFIX environment variable.
- A virtual environment at .venv in the current directory, or in the nearest parent directory.
- The Python interpreter available as, e.g., python3.7 on macOS and Linux. On Windows, uv will use the same mechanism as py --list-paths to discover all available Python interpreters, and will select the first interpreter matching the requested version.
- The Python interpreter available as python3 on macOS and Linux, or python.exe on Windows.> I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.
npm -> Yarn was life changing performance-wise.
I wonder what pip -> uv is.
I’m loving the pace of release with this crop of speed obsessed projects and I cannot wait for astral to tackle typing.
Installation times (MM:SS):
# No cache
- uv: 01:05
- pip: 01:56
# Cache:
- uv: 00:02
- pip: 00:42Npm has improved greatly.
Npm features follow a two-step process: (1) Get implemented in Yarn (2) Get implemented in npm.
By the way, the creator of mamba started his own company at https://prefix.dev/
They want to essentially leverage the conda(-forge) infrastructure to build a new cross-platform, cross-language, cargo-like package manager: pixi
Using only „Pythons native tools“ like pip and venv simply works nowadays so good that I wonder about the purpose of many tools like poetry etc. etc.
https://www.bitecode.dev/p/back-to-basics-with-pip-and-venv
https://www.bitecode.dev/p/relieving-your-python-packaging-p...
pip and venv work fine, but you have to get them first; and that can be a struggle for unseasoned python devs, especially if you need a version that's not what your distro ships, and even more so on Windows and macOS.
I use micromamba [1] specifically, which is a single binary.
[1] https://mamba.readthedocs.io/en/latest/user_guide/micromamba...
pip install works, but pip's dependency management doesn't seem to (for Pytorch, specifically) which is why projects that have pip + requirements.txt as one of their installation methods will often have separate pytorch installation instructions when using that method, though if the same project supports conda installation it will be a one-stop-shop installation that way.
That’s interesting — I’ve also had difficulties with PyTorch and dependency resolution, but only on the most recent versions of Python, for some period of time after they’re released. Picking Python 3.9 as a baseline for a project, for example, has been very reliable for PyTorch and all the related tooling.
After fiddling with different solutions for years and having to start fresh with a new Python install, I've been using nothing by miniconda for years and it just works
Python is used by loads of scientists, academics, and other non software engineers. Those people need an easy way to use python.
If you want to have a platform that allow to manage Python on all machines, including allowed packages and version, integrated with ldap, with auditing capabilities, they are pretty much the only game in town.
And big companies want that.
They were never big outside Windows shops.
Today I would say anaconda brings more problems than it solves (https://www.bitecode.dev/p/why-not-tell-people-to-simply-use), but at the time, we didn't have wheels for everything, and it was a life saver for anybody that wanted to use c extensions.
So anaconda became first popular because it solved a real end user problem, then it moved on to be the corporation providers because it already was well known.
It was a very good strategy.
PyPI distributed wheels (you can obviously build wheels however you like, but that doesn't mean they'll run on others' systems) may at some point get close to what conda does (likely by reinventing it in an ad-hoc fashion), but there's enough mindshare (especially around its target of "data science", where all the tutorials are around using conda) that I don't see it disappearing.
[1]: https://www.anaconda.com/blog/announcing-python-in-excel-nex...
Improving Python is especially cheap compared to the productivity that could be unleashed. Surprised it isn't done more often. Only microsoft has shown significant interest, which is a shame. Perhaps changing.
We need a permanent solve for python package management, governed by the python steering council.
> TL;DR: uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows.
(.venv) lijok@mbp icebreaker % python -m uv pip install --upgrade uv setuptools wheel
error: unexpected argument '--upgrade' found
tip: to pass '--upgrade' as a value, use '-- --upgrade'
Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE>>
For more information, try '--help'.
(.venv) lijok@mbp icebreaker % python -m uv pip install -- --upgrade uv setuptools wheel
error: Failed to parse `--upgrade`
Caused by: Expected package name starting with an alphanumeric character, found '-'
--upgrade
^
alias pip="/usr/bin/uv pip"
Arguably they'd get significantly further choosing a specific and narrow use-cased and nailing it (e.g. deploying WSGI/ASGI apps via OCI containers, with the ability to have live reload during dev), rather than introducing another option whose main selling point is speed, which while nice, is hardly going to resolve why we have the different use-cases and hence solutions.
I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards.
(Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)
``` black @ git+https://github.com/psf/black ```
We also support editable installs for local directories, like you mentioned:
``` black @ ../black ```
The thing we don't support is using _editable_ installs for Git and direct URL dependencies, like:
``` -e git+https://github.com/psf/black ```
In my experience, these are really rare, and we didn't see a reason to support them.
As conda/pip interop is just not great. And even micromamba (C++ implementation of conda) is relatively slow to resolve compared to pip.
But agreed either work together or create a type checker. I use mypy currently but it definitely slows down my editor.
def _deep_merge(updates: dict[str, str]) -> None:
for key, update_info in updates.items():
if isinstance(update_info, dict) and "values" in update_info:
value = update_info["values"]
- error: Argument of type "Literal['values']" cannot be assigned to parameter "__key" of type "SupportsIndex | slice" in function "__getitem__"
Type "Literal['values']" cannot be assigned to type "SupportsIndex | slice"
"Literal['values']" is incompatible with protocol "SupportsIndex"
"__index__" is not present
"Literal['values']" is incompatible with "slice" (reportGeneralTypeIssues)It took me a great amount of starring to figure out that changing the signature of updates to dict[str, dict] was what it was complaining about.
Hence they will always trigger user errors, and their image will be stained by it.
Rip (https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda).
And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM.
A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools.
I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow.
As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.
Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.
Being compatible with the mess that exists is where the difficulty comes from.
The last 5 or so years I've mostly done work in rust... so I guess I'm a rust developer now.
Do the last 5 years invalidate my opinions, thoughts, or skills w.r.t. python somehow?
edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?
really curious on the reasoning behind this decision :)
Mostly it’s useful right after a new release where prebuilt wheels aren’t available for all packages yet and you have users who may care or not about this.
If you only have single requirements file, you get forced to choose which platform to support. With multiple, you can break out if needed. These breaking changes and deadlocks are rare. It’s still good to have an escape hatch.
says:
> uv does not yet produce a machine-agnostic lockfile.
so maybe the non-portable requirements.txt is just a first milestone
> Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more.
A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.
Sure, it’s rendered a bunch of projects obsolete, but it’s because it’s so much better than the predecessors.
Obviously it would be pretty crummy to end up in a situation like pyright where MS really have leant in to their embrace, extend, extinguish strategy.
That is true for every single development tool in the Python space.
Yeah. Probably precisely the same people that call everything EEE.
You’ll always find people acting alarmist about anything.
I agree that we have no idea how it'll play out. But I've seen nothing but good faith from Charlie and others, and I see no reason to preemptively condemn their attempt to build a business.
In this case they are replacing Python code with Rust which might exclude large part of Python community from being able to contribute.
(Or another framing: only the tiniest fraction of the Python community was contributing to CQA tooling in the first place. It's not clear that prioritizing the interests of the 99.9% of Python programmers who -- rightfully! -- will never modify these tools makes sense.)
I'm not sure what the maker of that video wants, does he want money to be poured back into the community, or for no one else to make money?
Global dependencies by default (fixed by venvs). Shipping precompiled binaries that are not portable across platforms or python versions (kinda fixed by wheels, I forget the PEP number). No lock files with checksum verification by default (kinda fixed by requirements.txt with require hashes, but not really). Also, there's a bunch of weird shit that can happen if the pip version and python version disagree due setup tools.
It's got a great interface for installing stuff. But the likelihood that what it installs breaks something on your machine is pretty high the longer you use it.
Edit: the real, fundamental problem with pip is that it can't be saved. Too many dockerfiles and server provisioning scripts are relying on it - it's interface, semantics, CLI output, etc. It's not 100% broken, which is why it's still useful. But any true fix has to be done by creating a separate tool.
Pip is also strongly tied to the specific Python version. If you upgraded Python interpreter on a system, congrats all virtual environments and all PyPi packages are broken now. Hopefully you saved the minimum set of requirements and they don't have some catastrophic dependency breakage. You run pip install and pray.
What? Native binaries? Building them on the fly while installing a pip package? Oh you're brave.
Misleading. Continue to use the older version, or packages will need to be installed in a new folder for the new version. Recreate venvs and/or simply reinstall any --user.
* Defaults to installing packages globally instead of per project.
* requirements.txt feels very much like a second class citizen when you contrast it with package.json, Cargo.toml, build.gradle, pom.xml, or most other dependency systems in other package managers.
* No native support for lockfiles short of pinning every version of every project in requirements.txt. This solution is inferior to what is available in ~every other package manager, because a pip freeze doesn't distinguish between explicit dependencies and transitive dependencies.
I'm sure there are others, but they're all along the same lines—pip is strictly inferior than basically every other package manager out there for other languages. Things that people expect to be part of a package manager have ugly hacky workarounds instead that aren't uniformly applied across projects.
I was able to build useful things with pip + virtualenv
https://stackoverflow.com/a/34631976
Pip fails with some kind of "not found" error and then you have to use some other package manager to supply the missing thing.
Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed.
sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.
The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).
One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.
Does this show deficiencies of Python as a general purpose language?
They also seem to have money for rapid development which trumps poor community projects.
Maybe it shows the challenges with using python for deployment and distribution of binaries.
uv, rye they want to install and manage python, so they try to solve the bootstrapping problem - should a tool that installs python require python to be installed, etc?
https://news.ycombinator.com/item?id=32116649
Python packaging has gotten asymptotically worse in the last 90 days from my experience having used the language for 20 years now and working on a multi-platform triumvirate of Lin-Mac-Win that has not changed in twenty years.
I can honestly say that, given how daily failures in the packaging ecosystem such as pip failing to build egg, pkg-config, and similar problems, I'm happy to see a solution from rust, because I sent an intern into the polars underworld 2 years ago to great success and have only grown my own reliance on rust infrastructure more broadly since.
This may very well be the only packaging innovation in python that I actually look forward to trying tomorrow. I'd already be working in rust full-time if I could only get past the syntax that reminds me too much of perl after bathing in the semantic whitespace light for this long.
Of course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.
Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity.
I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.
For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.
For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only?
Just downloading a library to a local source code directory should not cause arbitrary code to run on your system, and there should be safeguards in place so developers are not one typo away from a malicious package pwning their entire system. Supply chain attacks remain a huge area of concern.
Instead, when I "ctrl+f security" the homepages of any of these new packaging systems, I get 0 results. Not good.
I think you're thinking of this post[1]. The code being searched for wasn't malicious, just buggy :-)
[1]: https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
Heh, I forgot about the anime catgirl part.
Please keep doing what you're doing!