notatallshaw

Fun fact!

The original request to add a concurrency variable in uv was requested by me because, among other things, I wanted to measure the difference between uv and pip: https://github.com/astral-sh/uv/issues/3311

I'm not really following this performance discussion as I think it's gone off the rails.

That's excellent news for uv, and I think covers one of two major use cases I often see where uv does not cover standard packaging workflows.

This one being downloading to an offline wheelhouse and installing from that.

The other one being having a shared named global environment ;o).

P.S. I'll have to remove this as an important feature nab has that uv doesn't when I make the announcement nab is no longer experimental, aha.

> it improved a boto3 case by 3x (30s to 10s) and our "standard" solve benchmark by 2x. It's plausible some of those gains have been reduced by other optimizations in our solver since then though.

This makes a lot of sense because pubgrub makes heavy use of version comparison, compared to simple DFS algorithms like the one resolvelib uses. A lot of Pubgrub optimizations come from finding clever ways to not need to keep comparing versions.

> But this was just one example optimization, we do other low-level things, like zero-copy deserialization from our cache. The point is not that we do specific things, but that we have more levers to pull to improve performance.

Oh yes, I agree with the general point, I was just picking on the specific example for a fun exploration of performance optimizations.

Also, FWIW, I have in my professional career, not OSS work, implemented zero-copy deserialization from cache in pure Python, there are many surprising levers in Python when you are willing to explore the weird corners of the standard library.

> We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code.

As a complete aside, uv can and does do this, but for this particular optimization I'm not sure how much absolute time it ends up saving compared to pure Python in real world resolution scenarios.

uv's total memory usage isn't that much leaner than pip's, and for parsing speed it turned out that the library pip uses, packaging, was just very unoptimized at the time uv launched. This has been significantly addressed since then:

* We did a lot of work to make version parsing twice as fast: https://iscinumpy.dev/post/packaging-faster/

* Since that blog post I made typical version parse three times faster on top of that: https://github.com/pypa/packaging/pull/1082

* Also since that blog post version filtering has gone through multiple optimizations and in some cases will be more than 30x faster e.g. https://github.com/pypa/packaging/pull/1105, https://github.com/pypa/packaging/pull/1111, https://github.com/pypa/packaging/pull/1120

At this point large dependency resolves in pip are spending very little of their time doing things in packaging, like version parsing. The main non-IO time spent in large resolves is now in the core resolver, resolvelib, which I hope to one day replace with my experimental resolver nab: https://github.com/notatallshaw/nab. Nab scales to large resolves much more efficiently than resolvelib (in fact I've cross-ported some of the algorithmic efficiency gains to uv already ;o)).

As a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can.

But it has always had two major issues:

1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download")

2. For people with a lot of different environments the cache grows significantly more than pip

I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.

Or an obscure CalVer that no one realizes - /pip-maintainer

FWIW, break-system-packages comes directly from pip's interface, it's not a novel addition from uv.

If Astral/OpenAI are willing to support uv pip in the long term, why do you think they should use a stick as well as a carrot to get people on uv's top level interface?

While uv's standard workflows probably works for 80+% of people, at least if they learn them, I certain do a lot of stuff with uv pip and pip that can't be replicated.

Also both uv pip and pip support installing from pylock.toml files, which are fully reproducible.

> Their infrastructure (pip) getting worse and worse

Hi, I'm a pip maintainer, can you please report any issue you think is happening in recent versions of pip: https://github.com/pypa/pip/

I particularly work on performance and stability and think we've been making big strides over the last two years

> getting everyone to install private venv's for every app, leading to missing security updates, because updates just break everything.

venv's stop your OS tooling from breaking, Linux distro maintainers pushed Python packaging to not allow installing into the system packages by default, and they weren't wrong.

All these other problems are solved with proper use of a lock file (which are now standardized, and you can generate and install from them in the latest version of pip), and/or use a project tool (uv, poetry, etc.).

> Running pip install lightning is all that is needed to activate

FYI, pip added cooldowns in 26.1:

  * https://discuss.python.org/t/announcement-pip-26-1-release/107108
  * https://ichard26.github.io/blog/2026/04/whats-new-in-pip-26.1/

To use:

  * CLI: pip install --uploaded-prior-to=P1D ...
  * Env Var: PIP_UPLOADED_PRIOR_TO=P1D pip install ...
  * Config: pip config set global.uploaded-prior-to P1D

The note at the end of the post indicates the user asked Claude to review their own chat logs. It's impossible to tell if Claude used or built a a performance harness or just wrote those numbers based on vibes.

Pip maintainer here, to do this in pip (26.0+) now you have to manually calculate the date, e.g. --uploaded-prior-to="$(date -u -d '3 days ago' '+%Y-%m-%dT%H:%M:%SZ')"

In pip 26.1 (release scheduled for April 2026), it will support the day ISO-8601 duration format, which uv also supports, so you will be able to do --uploaded-prior-to=P3D, or via env vars or config files, as all pip options can be set in either.

The threat actor was sophisticated enough to spam GitHub issues with dozens of different accounts. I imagine they could completely overwhelm PyPI with unauthenticated reports.

> I had no idea how to contact anyone related to PyPI

https://pypi.org/security/:

> If you've identified a security issue with a project hosted on PyPI Login to your PyPI account, then visit the project's page on PyPI. At the bottom of the sidebar, click Report project as malware.

As a pip maintainer I don't think that's really true. The resolver in both pip and uv are fundamentally sequential and single threaded, you can't really queue up or split out jobs.

What uv does is parallelize the final download of packages after resolution, and batch pre-fetch metadata during resolution. I don't think these benefit from async, due to their batch nature classic multi-threaded download pools are probably the better solution, but I could be wrong!

Experiments have been done on the former in pip and didn't find much/any improvement in CPython, this may change in free threaded CPython. For the latter we currently don't have the information from the resolver to extract a range of possible metadata versions we could pre-range, I am working on this but it requires new APIs in packaging (the Python library) and changes to the resolver, and again we will need to benchmark to see if adding pre-fetching actually improves things.

pixi offloads PyPI ecosystem stuff to uv, but pixi is conda first. The team were actually the first to build a Rust based Python package resolver (rip), but after uv was released they migrated to uv's resolver (Python package resolvers are hard and a lot of work to build and must be tested against the whole ecosystem).

On a widely used open source project I maintain I've been seeing PRs in the last month that are a little off (look okayish but are trivial or trying to solve problems in weird ways), and then when I look at their account they started opening PRs within the last few weeks, and have opened hundreds of PRs spread over hundreds of repositories.

My understand is Astral's focus for ty has been on making a good experience for common issues, whereas they plan for very high compliance but difficult or rare edge cases aren't are prioritized.

Compliance suite numbers are biased towards edge cases and not the common path because that's where a lot of the tests need to be added.

My advise is to see how each type checker runs against your own codebase and if the output/performance is something you are happy with.

AI is somewhat helpful but I'm not interested in a company finding a way for me to pay to do my volunteer OSS work. GitHub Copilot offers a permanent free subscription for OSS maintainers.

I previously ignored a free offer when Claude reached out to me as an open source maintainer as it was a glorified free trial. I hope this one continues beyond the listed 6 months, I am not interested in a glorified free trial and if it requires entering credit card details I won't be signing up.

I found the diagrams on the Wikipedia pages help build an intuitive understanding of each step: https://en.wikipedia.org/wiki/Conflict-driven_clause_learnin...

Also, the pubgrub-rs guide I find has a gentle ramp in introducing complexity: https://pubgrub-rs-guide.pages.dev/internals/intro

I assume lock and dependency files are in the training data, so predicting version number tokens have high probabilities associated with them.

As a pip maintainer I primarily think about resolution and resolver performance. I got involved when pip introduced it's current resolver around late 2020, and got my first PR landed in mid 2021.

I recently became a packaging maintainer, from working on fixing edge case behavior around specifiers and prerelease versions.

When I did some recent profiling I noticed that A LOT of time was being spent in packaging, largely parsing version strings. I found a few places in pip and packaging that reduced the number of Version objects being created, Henry really ran with the idea of improving performance and made big improvements. I'm excited for this to be vendored in pip 26.0 coming out at the end of January.

If anyone is interested the next big improvement for pip is likely to implement a real CDCL (Conflict-Driven Clause Learning) resolver algorithm, like uv's use of pubgrub-rs. That said, I do this in my spare time so it may be a year or more before I make any real traction on implementing that.

> At one company I worked at, we had a system where each deploy got its own folder, and we'd update a symlink to point to the active one. It worked, but it was all manual, all custom, and all fragile.

The first time I saw this I thought it was one of the most elegant solutions I'd ever seen working in technology. Safe to deploy the files, atomic switch over per machine, and trivial to rollback.

It may have been manual, but I'd worked with a deployment processes that involved manually copying files to dozens of boxes and following 10 to 20 step process of manual commands on each box. Even when I first got to use automated deployment tooling in the company I worked at it was fragile, opaque and a configuration nightmare, built primarily for OS installation of new servers and being forced to work with applications.

We're getting there https://pip.pypa.io/en/stable/cli/pip_lock/ !

Pip has been a flag bearer for Python packaging standards for some time now, so that alternatives can implement standards rather than copy behavior. So first a lock file standard had to be agreed upon which finally happened this year: https://peps.python.org/pep-0751/

Now it's a matter of a maintainer, who are currently all volunteers donating their spare time, to fully implement support. Progress is happening but it is a little slow because of this.

Unfortunately that was posted 1 month before the Faster CPython project was disbanded by Microsoft, so I imagine things have slowed.

Python has about 40 keywords, I say I would regularly use about 30, and irregularly use about another 5. Hardly seems like a "junkyard".

Further, this lack of first class support for lazy importing has spawned multiple CPython forks that implement their own lazy importing or a modified version of the prior rejected PEP 690. Reducing the real world need for forks seems worth the price of one keyword.

> I had been hoping someone would introduce the non-virtualenv package management solution that every single other language has where there's a dependency list and version requirements (including of the language itself) in a manifest file (go.mod, package.json, etc) and everything happens in the context of that directory alone without shell shenanigans.

Isn't that exactly a pyproject.toml via the the uv add/sync/run interface? What is that missing that you need?

> You let people make the mistake and have the library throw an exception if they do that, not through type checking but just through something eventually calling a method that doesn't exist.

Exceptions or crashes would be annoying, but yes, are manageable, although try telling that to new users of the language that their code doesn't work because they didn't understand the transitive dependency tree of their install and it automatically vendored different versions of a library for different dependencies, and how did they not know that from some random exception occurring in a dependency.

But as I explain in my example, the real problem is that one version of the library reads the data in a different layout from the other, so instead you end of with subtle data errors. Now your code is working but your getting the wrong output, good luck debugging that.

> I wouldn't mind a codebase where numpy objects created by dependency B can't be shared directly with dependency A without me first running some kind of conversion function on them

Given there's no compiler to enforce this check, and Python is dynamic language, I don't see how you implement that without some complicated object provenance feature, making every single object larger and every use of that object (calling with it, calling it, assigning it to an attribute, assigning an attribute to it) impose an expensive runtime check.

But maybe I'm missing something obvious.

I agree, but as a type checker it is a subjective choice, whether to be explicit and not make assumptions or whether to infer from certain certain patterns as probably correct and not be noisy to the user. Very glad to see they plan to fix this.

I assume in your example if you update the foo declaration to the following it solves the complaint:

    foo: list[str] = []

If so this a type checking design choice:

  * What can I infer from an empty collection declaration?
  * What do I allow further down the code to update inferences further up the code?

I don't know Pyrefly's philosophy here, but I assume it's guided by opinionated coding guidelines inside Meta, not what is perhaps the easiest for users to understand.