uv: Deduplicate all files in the wheel cache
github.com
github.com
But it has always had two major issues:
1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")
2. For people with a lot of different environments the cache grows significantly more than pip
I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent.
It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic.
It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different!
Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
> The only advantage of uv is to have support for parallel async extraction.
This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.
> If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv.
I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL.
> When the user edits one file, all files are modified simultaneously
This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common.
> Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks
We also explored this (see https://github.com/astral-sh/uv/pull/19694) and the linked pull request has a table comparing to this strategy.
> We have optimization for empty files (0 bytes) because there is nothing to write and checksum.
Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
FYI: The extraction of files is largely python code that doesn't release the GIL. (cf. the zipfile class from the python interpreter has large layers of abstraction with a massive overhead in python code).
Regardless, python packages are thousands of tiny files, so pip never gets to release the GIL for any meaningful duration.
If you were writing an app that only extracted large GB files, you could take advantage of some I/O operations and some zlib operations freeing the GIL for a bit. Unfortunately pip is the opposite use case, lots of tiny files.
> Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
Optimizing empty files was actually quite worthwhile for pip, because about 10% of python packages are empty init files.
This might not give the same result for uv though. pip is fully linear, every single open/read/write/stat operation we removed was a direct performance gain. uv does parallel async IO, you could very well remove 10% of filesystem calls and barely affect the overall duration. :D
TIL. I ran some benchmarks and confirmed this is the case for many small files as you'd see in wheels — the GIL is released in a meaningful way for larger files though. Thanks!
> This might not give the same result for uv though.
Yeah, I built a prototype and ran some benchmarks. It makes a big difference if the entire wheel is empty files but for any real world examples it's within noise of the baseline.
As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.
uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:
* We did a lot of work to make version parsing twice as fast: https://iscinumpy.dev/post/packaging-faster/
* Since that blog post I made typical version parse three times faster on top of that: https://github.com/pypa/packaging/pull/1082
* Also since that blog post version filtering has gone through multiple optimizations and in some cases will be more than 30x faster e.g. https://github.com/pypa/packaging/pull/1105, https://github.com/pypa/packaging/pull/1111, https://github.com/pypa/packaging/pull/1120
At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).
But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance. It's great to see all the improvements happening in pip performance regardless :)
This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.
> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.
Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.
Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.
That sounds like it would be relatively easy to demonstrate: You could tweak uv to perform downloads/extractions sequentially and then compare its runtime with pip. Has any such comparison been done?
- https://docs.astral.sh/uv/reference/environment/#uv_concurre...
- https://docs.astral.sh/uv/reference/environment/#uv_concurre...
The original request to add a concurrency variable in uv was requested by me because, among other things, I wanted to measure the difference between uv and pip: https://github.com/astral-sh/uv/issues/3311
I'm not really following this performance discussion as I think it's gone off the rails.
Maybe the only advantage in a particular area (installing)? Because there are many other advantages.
The rich lock file for instance allows for much better cross-platform tooling. I can build a linux Docker container from a macos build host, for example, without VMs or any other emulation - simply using the cross-platform details in uv.lock and the correct tooling. I can even cross-compile numpy and other native wheels (linux -> macos, macos -> linux).
Other locker tools provide similar cross-platform information (Poetry, PDM), but pip is still lacking.
Using hints or knowledge at the app level is a much better experience if the app can tell the FS that two files are identical. The FS doesn't have to worry about hashing blocks within the file, correcting alignments, etc.
Hardlinks have long been a well-documented feature of many file systems. I don't see how their use in a context users largely do not meddle with is too confusing or complicated.
Anyways it can be both? It could let the filesystem handle it if it detected the FS has file or block dedupe and fallback to hardlinks otherwise. It doesn't but that could be done if it showed it was worth it.
Maybe it's because I've been recently playing around with deduplicating subtrees in a different domain, but I don't see what makes this feature too complex to be worth the cost. Sure, symlinks and hardlinks require some additional care compared to plain files, but it's easy enough to add tests for those cases.
...per unique TB written to datasets with deduplication enabled, of course, not the pool as a whole (even though that ram cost is indeed at the pool level).
there are plenty of useful things to be done with zfs dedupe at practically no cost if you design for it ahead of time. unfortunately for some reason people parachute into every thread where zfs comes up to say you can never ever turn it on, so people are taught not to even try.
for instance, i have a pool with plenty of datasets that have dedupe off and only a crucial few with it enabled: the ones that are nfs-mounted media storage volumes for fediverse instance containers that would otherwise all duplicate all media they see in common, which is most media posted to any of them and a good bit of the remote media received by any of them. the size of the dedupe tables in memory is purely a function of the number of blocks written to those few datasets and i expect to either sunset this laptop from 2012 or migrate the pool to a fresh one long before i would be worried about the memory cost of deduplication swelling a few orders of magnitude to where i'd have to care.
uv has been absolutely black magic for everyone in the way it just works and works much faster, but here you are telling us that actually no it's somehow not that good
I’m not sure at what uv excel at but reading comments it seems uv is not only package manager but also venv manager.
This one being downloading to an offline wheelhouse and installing from that.
The other one being having a shared named global environment ;o).
P.S. I'll have to remove this as an important feature nab has that uv doesn't when I make the announcement nab is no longer experimental, aha.
I tried `uv --preview-features content-addressed-cache` and I get an error:
`uv --preview-features content-addressed-cache error: 'uv' requires a subcommand but one was not provided`
Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing).
If I've got a file named:
DSC98731-b3-7b39197a22.JPG
- if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues)
- if any other file has the same Blake3 7b39197a22 hash, it's a duplicate
- if that 7b39197a22 checksum is in my database, "things can happen".
It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
I wasn't very clear.
Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18.
This is one great benefit of using .zip files as file formats, you get this for free.
Though fair enough, it could offer it as an opt-in thing.
Whereas your sqlite data are your data, and if they're corrupted they can be lost forever or propagate the issue to backups.
I agree.
> Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage.
I mostly run ext4 (desktop, laptops, etc.) but for my main server at home, it's a ZFS (mirrored) tank on an old server with ECC RAM.
And backups. So much backups.
I wonder if something changed, with todays SSDs, memory bottlenecked CPUs, and hardware accelerated CRCs.
FWIW, RAM bit-flips are much more common today, on consumer devices where SQLite is used a lot, since today's memory operates at the limit (see RowHammer).
Do you have it in a public repo you could share?
But I know others did similar thing so maybe there are public repos out there. But in any case: it should now be the kind of thing relatively easy to vibe-code if it's for your own use.
Sounds like it's worth another look at your settings to make sure they are right.
Compared to what? Fast is relative. Compared to pip it's miles ahead.
If you're running Python scripts, I'd recommend using `uv lock --script <path>` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades.
(I work on uv)
pip install git+https://github.com/some-org/repoThe one use I can give it is for running scripts with inline dependencies. I have found it a noticeable improvement over pipx there. But that's more due to needing to parse dependencies every single launch, if it's something I use regularly I end up just installing them normally instead, and it's faster than either.
None of it is unique to `uv` I believe, but it does it all, reliably, is easy to install, and easy to use. In terms of DX, for my use cases, it beat poetry, pipenv, pyenv (for python version management) and just using pip
One could also make a comparison to another popular tool, conda, which is glacially slow, but that would arguably be unfair since conda does things uv doesn't.
More generally, it's still a nice thing, though. It's just a little less friction. Even a few seconds slower can nudge you towards different behavior
Disk isn't free, especially now.
I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there.
The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc.
Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
A week later I was cleaning up disk space again...
Or (nasty) bugs hard to debug
It's a tradeoff between wasting a little time everyday vs. wasting equally little time every year or so but also having the mental load of cleaning up plus maybe existential dread vs. spending more money to never have either problem. I do understand the last one isn't an option for everyone, but if it is, it's absolutely worth it.
This could happen with a user asking their editor to show definition, and then not realizing the result was in some other package, changing it, and saving it, but only if the editor's normal atomic replace mode is disabled. So perhaps that sounds farfetched. But if an AI agent wants to patch some package in a venv, I've seen them bypass the editor and use command line tools to make edits often enough to be concerned. Like corrupting the cache for one package version is already not ideal, and this could potentially affect multiple versions now?
I suppose is this happens the right fix is to nuke the cache completely?
Funny to list OpenAI as an example at the end of the blog post ...
We just use containers in CI instead now for matrix testing and it’s so much more clear what is going on with the test env.
The pip ecosystem (including uv) doesn't encourage this. Although you could and many have, the build tools don't support it as first class.
For why? Ask what happens if you want an updated version or specific version of CUDA? The library author must add support for runtime linking or users would have rebuild the entire conda package.
I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.