Rye and Uv: August Is Harvest Season for Python Packaging

How the hell was pip taking over an hour to produce an image for a non-compiler language?

I’m only a casual Python user. But wtf was it doing and why did it take so long? That’s bonkers.

I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.

I would really encourage you to think more on this. Good enough for you... is not really the goal. We want standard tools that help with packages and virtual environments that can scale to an organization of 2 python devs... all the way up to hundreds or thousands of devs. Otherwise, it fragments the ecosystem and encourages bugs and difficult documentation that prevents the language from continuing to evolve effectively.

As long as you don’t care about Python versions I too find pip + venv sufficient.

But there is no way for you to define which Python version your project is built against.

If you’re building a package you probably need to test against multiple versions. If you are building a “project” that isn’t an installable, distributed package but a bunch of code that is shared with a few developers and does something (run a machine learning model, push itself to a cloud function, generate a report) you probably want to target 1 and only 1 Python version.

As for the mono repo approach are you suggesting to copy paste numpy and pandas into the repo?

That has been entirely my experience too: pip is good enough for me.

> What really annoys me about Python, is the fact that I cannot easily wrap my application in an executable and ship it somewhere.

You are not wrong, but let's unpack this. What you're saying is that there is a need to make it easy for another person to run your application. What is needed for that? Well you need a way for the application to make its way to the user and to find some Python there and for that process to be transparent to the user.

That's one of the reasons why I wanted Rye (and uv does the same) to be able to install Python and not in a way where it fucks up your system in the process.

The evolved version of this is to make the whole thing including uv be a thing you can do automatically. You can even today already (if you want to go nuts) have a fully curl to bash installer that installs uv/rye and your app into a temporary location just for your app and never break your user's system.

It would be nice to eventually make that process entire transparent and not require network access, to come with a .msi for windows etc. However the per-requisite for this is that a tool like uv can arbitrarily place a pre-compiled Python and all the dependencies that you need at the right location for the platform of your user.

The cherry on the top that uv could deliver at one point is that fully packaged thing and it will be very nice. But even prior to this, building a command line tool with Python today will no longer be an awful experience for your users which I think is a good first step. Either via uvx or if you want you can hide uv entirely away.

It depends where and what you are trying to install, but generally for every sensible deployment target there's a tool (sometime more than one) which handles building a installable binary for that target (e.g. an OS specific installer). There's now even tooling to deploy to unusual places such as Android or iOS or in the browser. Naturally in some cases specific packages won't work on specific targets, but because there are standard interfaces if the code is capable of being run somewhere, the tooling for that target should be able give you something that works.

That battle has already been lost for Python. Microsoft owns Python, they just don't make it public.

This is how I came to believe this is the case:

Few years ago I wanted to write Python bindings to kubectl. I discovered that in order for that to work cross-platform, I need to make CGo use the same compiler on all platforms as does Python. Unfortunately, on MS Windows, CGO uses MINGW while Python uses MSVC. I wrote to Python dev. mailing list (which still existed at that time) and asked why did they choose to use a proprietary compiler for their "open-source" project. The answer I received in a round-about way was that MSVC was a historical choice, which cannot be presently changed because MS provides Python Foundation with free infrastructure to run CI and builds, and it also provides developers to work on Python (i.e. MS employees get paid by MS to work on Python interpreter). And that they are under orders not to drop MS tools from the toolchain.

Year after year the situation was getting worse. Like in a lot of similar projects, success created a lot of ground for mediocre nobodies to reach positions of power. Python foundation and satellite projects like PyPA started to be populated by people whose way into these positions was not through contributing any useful code, but rather writing pages of code of conduct. This code of conduct and never-ending skirmishes around controlling positions eventually led to some old-timers leaving or being outright kicked out (latest such event was the ban of Tim, the guy who, beside other things, wrote Tim sort, which is a somewhat famous feature of Python).

Year after year MS was pushing its usual agenda they do in every project they get their hands on: add crapload of useless features for the sake of advertising. Make the project swing every way possible, but mostly follow the fashion trends as hard as possible. This is how Python is now devoted to adding as much of ML-style types as possible (in the language with a completely different type system...), AoT compilation and JIT (in the language that's half of the time used to dynamically glue native libraries...) and so on. Essentially, making it a C#, but without curly braces.

MS is smart enough to understand that publicly announcing their ownership of Python will scare a lot of people away from the technology, so they don't advertise it much. But they keep working on ensuring developers' dependency on their tooling, and eventually they will come to collect on their investment.

npm was always a company, right? Whats the rugpull? Has anything stopped working?

The PSF and PyPA are more than welcome to get their act together and tell a good packaging story for Python. Unfortunately all they’ve had to offer is a billion contributor blog posts essentially saying “the system we set up for ourselves limits our ability to be helpful, and that’s not our fault, somehow.”

They’re all so caught up in their internal systems and politics[1] that they don’t seem to know why they’re actually there anymore.

So if someone’s gonna go do an actually-good job and capture the market in the way that Astral is, then that’s exactly what we as a community deserve.

[1]: I mean internal politics. This is a weird alt right “DEI HIRE!” rant.

I learned recently that the “authority” in the name PyPA - Python Packaging Authority - was originally intended as a joke: https://discuss.python.org/t/remove-the-authority-from-packa...

I stopped worrying about what PyPA said when they endorsed Pipenv above other options, largely due to what seemed to me to be personal relationship reasons. Pipenv was disastrous at that time. It meant well, but at work we found ourselves waiting an hour for it to update its lockfile, and we had a relatively small number of widely-used dependencies in each repo. It just didn't work.

As a practical matter, I hugely appreciate the hard technical work PyPA does. However, I don't concern myself much with which toolsets they're recommending at the moment. Use what the community uses and don't worry about the "official" suggestions.

As a somewhat outside observer I don't quite understand what the PyPA actually is. In some sense there is an unclear number of participants and it's not even sure to which degree the PyPA relates to core python or the PSF.

I think the real endorsement that could help would be the core Python project itself. In a perfect world the official Python tutorial would start with "here is how you install python" and it starts by installing uv, the same way as the official Rust docs point to rustup and cargo.

I hope strongly that the PSF will manage to establish some sort of relationship with Astral which would enable that to eventually be a reality.

I frankly don’t give a fuck about PyPA at this point. They’ve proven themselves to be largely irrelevant in this context. Anything they get their hands on just seems to rot, so I’d unfortunately like them to stay far away from this. It pains me to say this. It’s against my philosophy. But it’s merely a reflection of the current state of things. I’ve been working with Python full-time for a decade. Other packaging ecosystems have essentially lapped Python’s.

At this point I also don’t care about nuances like “is it execution issues with PyPA, or that their set remit is faulty?” I’m sick of getting drawn into that stuff, too.

Forking doesn't inherently lead to further fragmentation: the level of fragmentation post forking can still be much lower than before consolidating on the rug-pulled tool

(also, how many more decades does this imaginary community need to create a great dominant tool?)

It may be easily forkable due to the licence choice (MIT or Apache), but the choice of Rust limits the number of people who can actually contribute.

Isn’t npm VC-backed?

At least uv doesn't lead to an absolute clusterfuck of poetry conflicting with virtualenv. Or broken package.toml format with minor poetry changes. Or amazingly dumb "sources" which doesn't work for transitive dependencies and leads to even longer resolve times in case of multiple indexes.

Are there any specifics you can share about what you found difficult to do in uv?

We just recently migrated to uv from pip-tools and it was pretty seamless. uv pretty much just plugs into the standard python tooling workflow.

you probably want Rye (as uv is too low-level for now), no?

To which insanities are you referring?

Should have ended with "I use Nix, btw"

The harsh reality is that this sort of tooling requires a lot of publicity to be successful. Nobody is going to try an unknown package manager from some random developer, but people will enthusiastically adopt anything pushed by a "famous" developer like Ronacher.

Not all good things gain traction. It definitely helped that Armin is well-known in the Python community. Where did you advertise your new tool?

People aren’t writing new package managers for rust every other week. JavaScript still sees some but less.

Perhaps people are trying to solve real pain points, and by getting closer to solving them things feel nicer!

Who are "those people"?

Rye was started by Armin as a collection of existing tools (mostly) with a frontend similar to cargo.

Then Astral came out with uv which aims to be a frontend into a collection of their own tools similar to cargo.

Armin and Astral agreed for Astral to take over Rye some time during uv development with (I assume) the goal for uv to fully replace Rye.

Use uv. As of 0.3.0 it covers most of rye now anyway. Especially if you’re writing projects and not consumable libs/apps (I haven’t used uv for anything other than package management so far).

It was a prototype. Lessons/code folded into uv.

That's entirely appropriate imo - system packager should be running pip or whatever and then putting the output where it wants, it's not for setuptools or whatever to declare that.

What path would you put in setup.py anyway? A different one for different distro preferences, a different one again for Windows, for macOS?

You can still use the setuptools backend when using declarative tooling, I use it in projects I can't fully port to pyproject.toml.

The big problem is that Python is used in a lot of different contexts and has too many valid ways in which people are able to build, compile, release non-Pythonic extensions in their package. Original way for packaging code was effectively an executable script which made some things easy, but other things much harder. Modern efforts are trying to limit the flexibility of the package definitions without breaking all of this legacy code. I unfairly think of Ruby as only for Ruby on Rails, so only ever dealing with the web domain.

Here is a story about the nightmare it takes to compile the Fortran code that supports much of SciPy (backbone numerical computing library with algorithms for wide swaths of disciplines) https://news.ycombinator.com/item?id=38196412

It's not. Just that Guido never cared about packaging, so it was left to a ragtag unpaid motley crew to piece together and later learn from industry practices that solidified a decade or two after they started.

Python's import system, as it is, makes some things slightly more complicated due to the need to use virtual environments (copies of the interpreter with separate library paths) as opposed to something like node_modules. This could easily be changed, if the powers-that-be actually cared about packaging. Packaging is handled by a separate group, which likes design by committee, and which enables the creation of separate third party tools instead of standardising on one good tool (uv, rye, and poetry, are all popular tools made by people who are explicitly not members of the packaging group).

jax itself is just pip installable, so this should be drop-in. CUDA is it's own headache, and that's where mamba (now pixi[1]) really shines.

[1]: https://pixi.sh/

Are you saying that because this was more of an opinion piece?

As the author of Rye, a package that was heavily influential on the development of uv, Armin is very well positioned to share his opinions about this.