PYX: The next step in Python packaging
astral.sh
astral.sh
We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place.
> Beyond the product itself, pyx is also an instantiation of our strategy: our tools remain free, open source, and permissively licensed — forever. Nothing changes there. Instead, we'll offer paid, hosted services that represent the "natural next thing you need" when you're already using our tools: the Astral platform.
Basically, we're hoping to address this concern by building a separate sustainable commercial product rather than monetizing our open source tools.
Only viral licenses are forever.
But, to refocus on the case at hand: Astral's tools don't require contributors to sign a CLA. I understand (and am sympathetic) to the suspicion here, but the bigger picture here is that Astral wants to build services as a product, rather than compromising the open source nature of its tools. That's why the announcement tries to cleanly differentiate between the two.
>bigger picture here is that Astral wants to build services as a product
What services? pyx? Looks nice but I doubt my boss is going to pay for it. More likely they just say "Whatever, package is in PyPi, use that."
UV, Ruff, Ty. Again, maybe they can get some data/quant firm who REALLY cares about speed to use their products. Everyone else will emit a long sigh, grab pip/poetry, black and mypy and move on.
Some companies specify that packages need to be hosted on their internal repository/artifactory, and for these companies, pyx might be a good tool.
MongoDB had a CLA from the start, didn't it?
> Astral's tools don't require contributors to sign a CLA.
That's a pretty vital difference!
> That's a pretty vital difference
Not really when the license is MIT/Apache. They can create a closed source fork and continue development there.
I do like your strategy, hope it works out!
Because FWIW CPython is not GPL. They have their own license but do not require modifications to be made public.
jfrog artifactory suddenly very scared for its safety
> or is one simply paying for access to Astral's particular server that runs it?
pyx is currently a service being offered by Astral. So it's not something you can currently self-host, if that's what you mean.
... Then how can it make decisions about how to serve the package request that PyPI can't? Is there not some extension to the protocol so that uv can tell it more about the client system?
[1]: https://packaging.python.org/en/latest/specifications/simple...
> honoring the normal 503/691 ones.
Embrace
> pyx can service specialized requests
Extend
... ;)
Unfortunately, the integrity of employees is no guard against the greed of investors.
Maybe next year investors change the CEO and entire management and they start monetizing the open source tools. There is no way of knowing. But history tells us that there is a non-trivial chance of this happening.
(Even with an entirely private setup, there is the risk that it will encourage too much developer attention to shift to working within that silo and thus starve the non-paying community of support, although I think this risk is less, given Python's enormous breadth of usage across communities of various levels of monetization.)
In the HPC space there are already Easybuild and Spack which make all the compiler tool chain and C and Fortran library dependency stuff very easy. They just haven’t taken off outside as they aim to solve cluster management problems but Spack is easy to self serve.
So even though they went commercial, they left pretty good things behind for the Open source community.
@woodruffw I love your quote above that commits you to your open source base and I'm rooting for you. But how about an approach that commits you to this sentence in a more rigorous and legal way, and spin off your open source tooling to a separate community-based entity? Of course, upon that you can continue to maintain sufficient representation to make Astral's commercial products the natural progression and otherwise the model remains the same. That would be a significant transfer of control, but it is that very transfer that would get a overwhelming response from the community and could really unblock these great tools for massive growth.
I work a lot with LLVM/Clang and whilst i know Apple and Google are significant contributors I feel confident that LLVM itself exists outside of that yet accept that e.g. Apple's contributions afford them weight to steer the project in ways that match their interests in e.g. Swift and Apple tooling.
Many ideas are being added to recent versions of pip that are at least inspired by what uv has done — and many things are possible in uv specifically because of community-wide standards development that also benefits pip. However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). That has a lot to do with why I'm making PAPER.
For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process. It remembers entire wheels, but finding them in that cache requires knowing the URL from which they were downloaded. Because PyPI organizes the packages in its own database with its own custom URL scheme, pip would have to reach out to PyPI across the Internet in order to figure out where it put its own downloads!
FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved.
I'm not saying here that pip doesn't have some gnarly internal details, just that the bigger thing holding it back is the lack of maintainer resources.
> For just one example: uv can quickly install previously installed packages by hard-linking a bunch of files from the cache. For pip to follow suit, it would have to completely redo its caching strategy from the ground up, because right now its cache is designed to save only download effort and not anything else about the installation process.
I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy.
That said, I do think there are other good examples to your point, like uv's ability to prefetch package metadata, I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
> FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved.
Fair enough. I'm sure if someone were paying me a competitive salary to develop my projects, they'd be getting done much faster, too.
> I actually think this isn't a great example, evidenced by the lack of a download or wheel command from uv due to those features not aligning with uv's caching strategy.
I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files.
> uv's ability to prefetch package metadata,
You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays.
> I don't think we're going to be able to implement that in pip any time soon due to probably the need for a complete overhaul of the resolver.
Ugh, yeah. I know the resolution logic has been extracted as a specific package, but it's been challenging trying to figure out how to actually use that in a project that isn't pip.
I'm talking about the fact there is no `uv pip download` or `uv pip wheel`.
I'm sure we'll discuss when you add this feature to uv, but I personally find uv's cache already grows too big too fast, so adding more to it makes me concerned.
> You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays.
Yeah, one of the way uv speeds up resolving is it pre-downloads .metadata files and checks if their requirements are identical to versions it had already checked, so it can quickly rule them out. It's a clever use of a collector and an advanced resolver.
No. This something people tell each other, that it's a lack of resources, but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, where they actively ignore and refuse to work on things that affect users every single day of usage.
There are features in FreeCAD that are straight up just broken that you hit every single day of using FreeCAD. When a simple low cost fix is suggested you get immense pushback, because of how "impure" the fix is despite being entirely cosmetic, also being reversible and having no long term impact on maintenance.
Progress happens when you sidestep those who block progress. That's what realthunder was doing and that's what Astral is doing with uv. No more bullshit excuses.
This is what forks are for. Alternately, money would probably help. Otherwise it's a bit rude to impose your priorities on others.
Perhaps some larger projects lose track of what they intended to prioritize. This is also a resources problem. There's nobody available to do management (and probably also not a social structure internally that would allow for it).
> because of how "impure" the fix is despite being entirely cosmetic, also being reversible and having no long term impact on maintenance.
The developers will always be in a better position than end users to assess the long-term maintenance impact.
I agree, but I also think sometimes that people think it's obstinate maintainership and it's not. For example, large PRs are hard to merge from a contributor who is fresh to the project because there is so much careful review that needs to be done, it's very resource intensive.
That said, one of my goals of becoming a maintainer has to be to try and making submitting a PR more friendly. Feel free to submit a PR to pip, I will be happy to help you get over any teething issues.
At best developers may be in a better position to assess long-term consequences. The history of major software projects is also the history of many projects making technical decisions that look good in the short-term but which turn out to impede actually achieving the desired features/performance/etc, and thus lead to major refactors/rewrites.
Well uv, as an OSS project, has come along with around 2 orders of magnitude more manpower than pip and has solved a lot of problems
> but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, where they actively ignore and refuse to work on things that affect users every single day of usage.
I wasn't making a statement about OSS projects in general, but I agree that pip has prioritization problems but I would argue that it stems from the lack of resources.
The people who do donate their spare time are in fact only likely to donate that time on things that interest them. If there were resources to have someone act like a project manager, and other to follow their lead, then the prioritization problems could be fixed, but there aren't those resources.
> No more bullshit excuses
Then contribute your full time hours to pip, or other OSS that needs fixing, rather than arm chair commenting on hacker news.
Conda has been able to do this for years.
imo, if pip's preference is to ship broken functionality, then what is/is not shipped with pip is not meaningful.
(And for the record: it isn't their fault at all. `pip search` doesn't work because PyPI removed the search API. PyPI removed that API for very good reasons[1].)
And this is part of the interlocking considerations I mentioned: there are private indices that supply the XML-RPC API, and breaking them doesn't seem justifiable[1].
Edit: fixed the link.
[1]: https://github.com/pypa/pip/issues/5216#issuecomment-1235329...
It's so wild to me that so many people are apparently against making a user-friendly update. The whole thing seems very against pep8 (its surprising, complicated, non-specific, etc)
Nobody wants anything to be user-unfriendly. You're taking a very small view into Python packaging and extending it to motives, when resources (not even financial ones) are the primary challenge.
I think the existing error message is useful:
$ pip search foo
ERROR: XMLRPC request failed [code: -32500]
RuntimeError: PyPI no longer supports 'pip search' (or XML-RPC search). Please use https://pypi.org/search (via a browser) instead. See https://warehouse.pypa.io/api-reference/xml-rpc.html#deprecated-methods for more information.
It either used to do fewer of those, or my memory is Swiss cheese, but I remember needing to search for the why's and alternatives.
I think part of the confusion here (which is reasonable IMO) is that pip has a huge version tail, and so people report error messages and behaviors that have been fixed for years. This is one of the challenges that I think gets under-appreciated about pip's maintenance.
Pypi is powered by warehouse which has around 3 developers maintaining it[0]. They're doing an awesome job, but the funding and resource available to them are probably substantially less than Astral could have with a paid offering.
I was surprised to see the community here on HN responding so cautiously. Been developing in python for about a decade now- whenever astral does something I get excited!
The issue is, track record is not relevant when the next investors take over.
With uv the worst case is it goes closed source in a few years and we all switch to a fork.
With pip the best case is that maybe in 10 years they have improved it to fix all the issues that uv has already fixed, and you only spend 10 years using shitty software and putting up with endless bugs and paper cuts.
Microsoft – who invented embrace, extend, and extinguish – own NPM, but I don’t see people wringing their hands over them in every thread that mentions NPM. But you mention Astral here or on Reddit and people line up to tell you it’s only a matter of time before they fuck people over. Why the disparity?
Astral on the other hand has basically been rewrites in Rust of existing community-based open source tools, for which there is always the question of how such work is funded. PYX (which is an interesting choice of name given the conflicts with pyrex/cython filenames) from what we can see here appears to be in a similar vein, competing with PyPI and making changes which seemingly require their client (uv) be used.
Anaconda/ContinuumIO was also treated with similar suspicion to Astral, so I don't think it's Astral in particular, it's more they both are operating in the part of the ecosystem where it is comparatively easy to lock out community-based open source tools (which the Python ecosystem appears to have been better at setting up and maintaining than the JS ecosystem).
Astral are not causing issues though. Why does “as long as Astral is not causing issues” not apply?
> Anaconda/ContinuumIO was also treated with similar suspicion to Astral
I haven’t observed this. I have seen condo talked about a fair amount but any issues have always revolved around it being awkward to use. But practically every discussion here or on Reddit about Astral has FUD.
Anaconda/ContinuumIO (the company) was absolutely treated with suspicion, see e.g. https://www.mail-archive.com/numpy-discussion%40scipy.org/ms... (and you'll find many such threads around that time on mailing lists/forums that were where the scientific python community was), and while the sky didn't fall in, their history hasn't been spotless. In many ways Astral is the more "webby" version of Anaconda/ContinuumIO, and so assuming Astral will behave (and evolve) in a similar way to Anaconda/ContinuumIO seems to me at least to be a rational thing to do?
pyx doesn't compete with PyPI; it's a private registry that companies can use e.g. to host internal-only packages, or to provide curated views of things like PyPI for compliance reasons.
> making changes which seemingly require their client (uv) be used
That's an explicit non-goal: "You won't need to use pyx to use uv, and you won't need to use uv to use pyx."
As soon as there is a commercial interest competing with the open source project at the same company the OSS version will begin to degrade, and often the OSS community will be left in the dark about this. The startup I was at had plenty of funding, far too many engineers, and still removed basically every internal resource from the oss project except one person and drove out everyone working on the community end of things.
I would also recommend avoiding working for any open source startup if your goal is to get paid to contribute to a community project. Plenty of devs will take a reduced salary to work on a great community project, but most of the engineers I saw definitely got the "bait and switch" and moved immediately to commercial projects.
I feel like having a working python environment is not a great requirement to managing your python environment.
Also, i tend to run pip like 4-5 times max (except initial package installations) during whole project lifetime, or not at all if packages are in distro already. Its speed is ok for that. So I'm not sure what "should" i miss.
most of the stuff in the python ecosystem have a core built in C, including the language itself.
Hopefully at the point the community is centralized enough to move in one direction.
But the increased resource usage is real. Now around 10% of our builds get OOM killed because the build container isn't provisioned big enough to handle uv's excessive memory usage. I've considered reducing the number of available threads to try throttle the non-deterministic allocation behavior, but that would presumably make it slower too, so instead we just click the re-run job button. Even with that manual intervention 10% of the time, it is so much faster than pip it's worth it.
(I work on uv)
We run on-prem k8s and do the pip install stage in a 2CPU/4GB Gitlab runner, which feels like it should be sufficient for the uv:python3.12-bookworm image. We have about 100 deps that aside from numpy/pandas/pyarrow are pretty lightweight. No GPU stuff. I tried 2CPU/8GB runners but it still OOMed occasionally so didn't seem worth using up those resources for the normal case. I don't know enough about the uv internals to understand why it's so expensive, but it feels counter-intuitive because the whole venv is "only" around 500MB.
Did you try reducing the concurrency limit?
Debian decided you should use venv for everything. But when packages are installed in a venv, random cmake nonsense does not find them. There are apt-get level packages, some things find those, others do not. Names are not consistent. There's a thing called pipx which my console recommended for much the same experience. Also the vestiges of 2 vs 3 are still kicking around in the forms of refusing to find a package based on the number being present or absent.
Whatever c++headerparser might be, I'm left very sure that hacking python out of the build tree and leaving it on the trash heap of history is the proper thing to do.
Not a project you built, or something you're proud of? Installing packages?
He creates the software and leaves it for others to install. No wonder he does not appreciate what uv does. ;)
We want to share the wheels more smoothly though and having been looking at pyx for that. Seems promising.
This comes across as uninformed at best and ignorant at worst. Python still doesn't have a reliable way to handle native dependencies across different platforms. pip and setuptools cannot be the end all be all of this packaging ecosystem nor should they be.
First things first:
Import path, os
I love Python, the ZEN of it, and you really need to accept the fact that there are conventions - quite a lot and that bash or shell scripts are where the magic happens, like environmental variables, if you know how to secure your app.
Even the self thing finally makes sense after years of bewilderment (“Wait: not even Java is that brutal to its users.”)
Lately stumbled over poetry after really getting the gist out of venv and pip.
Still hesitant, because Windows doesn’t play a role.
I'm so glad all the Python packaging challenges are "solved"
I just want packaging to fuck off and leave me alone. Changes here are always bad, because they're changes.
Or install it with the os package manager or something simmilar
They offered two options, so you can go do the other one if it doesn't work for you.
Yeah I had that on my work computer. I just created a venv and source that in my .bashrc.
I think it's a good thing that pip finally decided to refuse overwriting distro package paths (the alternative is far worse) but making "pip install --user" not work as well doesn't make sense.
This wall of text is an abridged excursion of my installing-python-on-Linux experience.
There is also my installing-python-on-Windows experience, which includes: official installer (exe or msi?) from python.org; some Windows-provided system application python, installable by setting a checkbox in Windows's system properties; NuGet, winget, Microsoft Store Python; WSL, WSL2; anaconda, conda, miniconda; WinPython...
Also I mean I understand mise but I personally just prefer using uv for python and bun for typescript both of which can run any version of python/ (node compliant?)
I still like the project though, but I tried to install elixir using it and it was a mess man.
It seems every project out there uses a different package manager, a different version of python, a different config file to set all of that up.
Most of the time, I just have a random .py file somewhere. Sometimes it's a full project that I can look at and find out what package manager it's using. Sometimes it has instructions, most of the time not. _That's_ the situation I struggle with.
Do I just run ./script.py? python script.py? python3 script.py? python3.12 script.py? When inevitably I miss some dependencies, do I just pip install? python pip install? pipx install?
As a developer I'm sure that you just set it up and forget about it. And once things work, they probably keep working for you. But man, it really reflects negatively upon Python itself for me. I don't hate the language, but I sure hate the experience.
#!/usr/bin/env -S uv run --script # /// script # requires-python = ">=3.12" # dependencies = [ # "ffmpeg-normalize", # ] # ///
Instead, we should always choose the common denominator of the most obsolete software platform imaginable. If there is an OS that has not been maintained for several decades, then that is the baseline we should strive to support.
Using an operating system with old libraries and language runtimes is not a personal preference with the consequences of restricting oneself to older software versions, no, it is a commandment and must not be questioned.
You have to use ancient C++ standard versions, deal with bugs in libraries that have been fixed years ago, lose out on all kinds of useful improvements or you end up with retrofitting a modern toolchain on an old system (but you still have to deal with an old glibc).
It’s hell. Just make the tooling/QA good enough so that everyone can run on the latest stable OS not too long after it’s released.
pyproject.toml tries to fix it, poetry kept wanting to use their own wonky names for the tags, I'm not sure why.
Once that is standardized, venvs should be cattle and not pets. That is all that is needed. UV makes that fast by hardlinking in the libraries and telling you the obvoius (that venvs should be cattle and not pets)
This fight was poetry's to loose.
However, uv is the first alternative that has tempted me to switch. uv offers performance improvements over pip and handles pyenv use cases as well. I’ll be paying attention to pyx to see how it pans out.
Maybe Python and js people should just use composer too.
I don't think PHP is perfect by any means but I don't dread using Composer, even in contexts like WP plugins where it is definitely not a "native" technology.
This is why containers are great IMO.
It's meant to solve the problem of "well it works on my machine"
I do ship containers as a top level .exe equivalent.
And also if you mean that pyx might be hosted on uv, well I think the discussion can go towards that pyx should be made open source but honestly, I am pretty sure that someone might look at pyx and create a pyx api compliant hosted server or I am still curious as to how pyx works and what it actually truly does.
This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience.
I went with Python because I never had this issue. Now with any AI / CUDA stuff its a bit of a nightmare to the point where you use someone's setup shell script instead of trying to use pip at all.
Also, now you have two problems.
In addition to elasticsearch's metrics, there's like 4 JVM metrics I have to watch constantly on all my clusters to make sure the JVM and its GC is happy.
In-house app that uses jdbc, is easy to develop and needs to be cross-platform (windows, linux, aix, as400). The speed picks up as it runs, usually handling 3000-5000 eps over UDP on decade old hardware.
I use Go a lot, the journey has been
- No dependency management
- Glide
- Depmod
- I forget the name of the precursor - I just remembered, VGo
- Modules
We still have proxying, vendoring, versioning problems
Python: VirtualEnv
Rust: Cargo
Java: Maven and Gradle
Ruby: Gems
Even OS dependency management is painful - yum, apt (which was a major positive when I switched to Debian based systems), pkg (BSD people), homebrew (semi-official?)
Dependency Management is the wild is a major headache, Go (I only mention because I am most familiar with) did away with some compilation dependency issues by shipping binaries with no dependencies (meaning that it didn't matter which version of linux you built your binary for, it will run on any of the same arch linux - none of that "wrong libc" 'fun'), but you still have issues with two different people building the same binary in need of extra dependency management (vendoring brings with it caching problems - is the version in the cache up to date, will updating one version of one dependency break everything - what fun)
1. There are a lot of build checks for problems involving mismatches between documentation and code, failed test suites, etc. These tests are run on the present R release, the last release, and the development version. And the tests are run on a routine basis. So, you can visit the CRAN site and tell at a glance whether the package has problems.
2. There is a convention in the community that code ought to be well-documented and well-tested. These tend not to be afterthoughts.)
3. if the author of package x makes changes, then all CRAN packages that use x will be tested (via the test suite) for new problems. This (again because of the convention of having good tests) prevents lots of ripple-effect problems.
4. Many CRAN packages come with so-called vignettes, which are essays that tend to supply a lot of useful information that does not quite fit into manpages for the functions in the package.
5. Many CRAN packages are paired with journal/textbook publications, which explain the methodologies, applications, limitations, etc in great detail.
6. CRAN has no problem rejecting packages, or removing packages that have problems that have gone unaddressed.
7. R resolves dependencies for the user and, since packages are pre-built for various machine/os types, installing packages is usually a quick operation.
PS. Julia is also very good on package management and testing. However, it lacks a central repository like CRAN and does not seem to have as strong a culture of pairing code with user-level documentation.
The mmdetection library (https://github.com/open-mmlab/mmdetection/issues) also has hundreds of version-related issues. Admittedly, that library has not seen any updates for over a year now, but it is sad that things just break and become basically unusable on modern Linux operating systems because NVIDIA can't stop breaking backwards and forwards compatibility for what is essentially just fancy matrix multiplication.
Meanwhile, I didn't feel like Python had reached the bare minimum for package management until Pipenv came on the scene. It wasn't until Poetry (in 2019? 2020?) that I felt like the ecosystem had reached what Ruby had back in 2010 or 2011 when bundler had become mostly stable.
docker has massively improved things - but it still has edge cases (you have to be really pushing it hard to find them though)
venv made these problems start to disappear, and now uv and Nix have closed the loop for me.
Not saying it didn't, I've just never ran into that after a decade of using the thing on various Nixes
Too bad there is nowadays the confusion with anaconda (the distribution that requires a license) and the FOSS pieces of conda-forge. Explain that to your legacy IT or Procurement -.-
OTOH today most Linux distros don't install any dev tools by default on a clean install. And, ironically, a clean install of Windows has .NET, which includes a C# compiler.
At the same time, even on Windows, libc has been stable since Win10 - that's 10 years now. Which is to say, any binary compiled by VC++ 2015 or later is C-ABI-compatible with any other such binary. The only reasons why someone might need a specific compiler version is if they are relying on some language features not supported by older ones, or because they're trying to pass C++ types across the ABI boundary, which is a fairly rare case.
This is a joke, obviously. We've had more than 14 for years.
This itself is IMO a product of Python having a relatively healthy consensus-driven standardization process for packaging, rather than an authoritative one. If Python had more of an authoritative approach, I don't think the language would have done as well as it has.
(Source: I've written at least 5 PEPs.)
I am honestly tired of the Python packing situation. I breathe a sigh of relief in language like Go and Rust with an “authoritative” built-in solution.
I wouldn’t mind the 30 different packaging solutions as long as there was authoritative “correct” solution. All the others would then be opt-in enhancements as needed.
I guess a good thought experiment would be if we were to design a packaging system (or decide not to) for a new PL like python, what would it look like?
I don't know about Go, but Rust's packaging isn't authoritative in the sense that I meant. There's no packaging BDFL; improvements to Rust packaging happen through a standards process that closely mirrors that of Python's PEPs.
I think the actual difference between Rust and Python is that Rust made the (IMO correct) decision early on to build a single tool for package management, whereas Python has historically had a single installer and left every other part of package management up to the user. That's a product of the fact that Python is more of a patchwork ecosystem and community than Rust is, plus the fact that it's a lot older and a lot bigger (in terms of breadth of user installation base).
Basically, hindsight is 20/20. Rust rightly benefited from Python's hard lesson about not having one tool, but they also rightly benefited from Python's good experience with consensus-driven standardization.
That said, it'd be nice if pip (or some PSF-blessed successor) adopted a model more similar to that offered by poetry (and formerly, pipenv, and now, I guess, uv) at least for package locking. `pip freeze > requirements.txt` isn't fit for purpose in the age of supply chain attacks, unfortunately, and PyPI already offers a reasonably workable backend for this model, whether the community at-large agrees to this or not. There are objective benefits (biggest one being better code integrity guarantees) that outweigh the objective (largely, performance-related) drawbacks.
> The next step in Python packaging
The headline is the confusing bit I think - "oh no, another tool already?"
IMO you should lean into stating this is going to be a paid product (answering how you plan to make money and become sustainable), and highlight that this will help solve private packaging problems.
I'm excited by this announcement by the way. Setting up scalable private python registries is a huge pain. Looking forward to it!
– “client (uv) and server (pyx)” and
– “You can use it to host your own internal packages, or as an accelerated, configurable frontend to public sources like PyPI and the PyTorch index.”
is what really helped me understand what pyx aims to be.
https://news.ycombinator.com/item?id=44712558
Now what do we have here?
(I work at Astral)
I'm glad to see Astral taking steps towards that.
I'm all seriousness, I'm all in on uv. Better than any competition by a mile. Also makes my training and clients much happier.
Of course if you are on one of the edge cases of something only uv does, well... that's more of an issue.
> "An example of what this might look like (we may not do this, but it's helpful to have a concrete example of the strategy) would be something like an enterprise-focused private package registry."
https://hachyderm.io/@charliermarsh/113103605702842937
Astral has been very transparent about their business model.
The issue is that there isn't a clean business model that will produce the kind of profits that will satisfy their VCs - not that there isn't any business model that will help support a business like theirs.
Private package management would probably work fine if they hadn't taken VC money.
Since this is a private, paid-for registry aimed at corporate clients, will there be an option to expose those registries externally as a public instance, but paid for by the company? That is, can I as a vendor pay for a Pyx registry for my own set of packages, and then provide that registry as an entrypoint for my customers?
We actually support this basic idea today, even without pyx. You can run (e.g.) `uv pip install --torch-backend=auto torch` to automatically install a version of PyTorch based on your machine's GPU from the PyTorch index.
pyx takes that idea and pushes it further. Instead of "just" supporting PyTorch, the registry has a curated index for each supported hardware accelerator, and we populate that index with pre-built artifacts across a wide range of packages, versions, Python versions, PyTorch versions, etc., all with consistent and coherent metadata.
So there are two parts to it: (1) when you point to pyx, it becomes much easier to get the right, pre-built, mutually compatible versions of these things (and faster to install them); and (2) the uv client can point you to the "right" pyx index automatically (that part works regardless of whether you're using pyx, it's just more limited).
> Since this is a private, paid-for registry aimed at corporate clients, will there be an option to expose those registries externally as a public instance, but paid for by the company? That is, can I as a vendor pay for a Pyx registry for my own set of packages, and then provide that registry as an entrypoint for my customers?
We don't support this yet but it's come up a few times with users. If you're interested in it concretely feel free to email me (charlie@).
what happens in a situation in which I might have access to a login node, from which I can install packages, but then the computing nodes don't have internet access. Can I define in some hardware.toml the target system and install there even if my local system is different?
To be more specific, I'd like to do `uv --dump-system hardware.toml` in the computing node and then in the login node (or my laptop for that matter) just do `uv install my-package --target-system hardware.toml` and get an environment I can just copy over.
(As always, big fans of Astral’s tools. We should get on with trying pyx more seriously)
Is the plan to also provide accurate (curated) metadata for security and compliance purposes?
https://www.buzzfeed.com/kristenharris1/famous-people-same-n...
and yet no one bemoans this (hospitals don't consult name registries before filling out birth certificates). that's because it's almost always extremely clear from context.
> The real pyx
what makes that pyx any more "real" than this pyx? it's the extension of the language py plus a single letter. there are probably a thousand projects that could rightfully use that combination of letters as a name.
the last release is Oct 16, 2022. are we doing this like jerseys - the name is now retired because pyx won all the championships?
Some software just needs fewer updates. The actual repo had a few commits last year and even more the year before that.
I think based on the quality of their work, there's also an important component which is trust. I'd trust and pay for a product from them much more readily than an open source solution with flaky maintainers.
They had contacted a company I had worked for asking them to purchase a license, because apparently somewhere in the company some dev workflow had contacted the conda servers regularly. We never ended up tracing it down as it stopped some weeks before them contacting us, according to our network logs.
In general, you don't resort to such sales tactics unless there is good cause (significant usage, where you as unpaid vendor also have leverage) or you just end up burning potential future customers (as was the case here, as they chose a bad contact point inside the company).
Don't understand how a private company like Astral is leading here. Why is that hard for the Python community to come up with a single tool to rule them all? (I know https://xkcd.com/927/). Like, you could even copy what Go or Node are doing, and make it Python-aware; no shame on that. Instead we have these who-knows-how-long-they-will-last tools every now and then.
They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
Python packaging is (largely) solving problems that Go and Node packaging are not even trying to address.
imagine a world without: failed to build native gem extension
I don't know much about go, and I've only scratched the surface with node, but as far as node goes I think it just distributes JS? So that would be one answer to what Python packaging is trying to solve that node isn't trying to address.
From a standards perspective, it is a combination of a Python version/implementation, a "platform" and an "ABI". (After all, the glibc/musl distinction doesn't make sense on Windows.)
Aside from CPython/pypy, the system recognizes IronPython (a C# implementation) and Jython (a Java implementation) under the version "tag"; of course these implementations may have their own independent versioning with only a rough correspondence to CPython releases.
The ABI tag largely corresponds to the implementation and version tag, but for example for CPython builds it also indicates whether Python was built in debug or release mode, and from 3.13 onward whether the GIL is enabled.
The platform tag covers Mac, Windows, several generic glibc Linux standards (called "manylinux" and designed to smooth over minor differences between distros), and now also some generic musl Linux standards (called "musllinux"). Basic CPU information (arm vs intel, 32- vs 64-bit etc.) is also jammed in here.
Details are available at https://packaging.python.org/en/latest/specifications/platfo... .
And in the TS/JS world we have React.Native that has a flexible pluggable model that allows creating XCode projects with autodiscovered dependencies in C, C++, Swift and other languages.
It's also flexible enough to allow third-party products like Sentry to integrate into the build process to upload debug symbols to the Sentry servers on release builds.
So no, Python is really not unique in its requirements.
Sass, Prisma, native DB drivers, or any other project using node-gyp or Node's NAPI are valid examples.
I've never had to compile a native Node library on a POWER9 GPU cluster, but I have had to compile Python ones.
Sorry, but Python is really not that unique.
As a sibling comment posted, Node packages have no issue distributing non-JS assets. Since the very beginning, the "package.json" file had support for the "arch" and "postinstall" fields to distribute precompiled binaries or compile on install, for example using node-gyp. If you split your precompiled binaries and use optional dependencies then only the right package for your machine is downloaded.
In all package managers, distributing native code may be uncommon but it's always supported well enough (Node/npm/yarn, JVM/Gradle, PHP/Composer, Ruby/gem, Rust/Cargo, etc). What's unique about Python is how messy all the packaging handling is compared to other languages.
I don't know what guides you're reading but I haven't touched easy_install in at least a decade. It's successor, pip, had effectively replaced all use cases for it by around 2010.
It is mentioned in the "Explanations and Discussions" section [0] of the linked Python Packaging guide.
Old indeed, but can still be found at the top level of the current docs.
[0] https://packaging.python.org/en/latest/#explanations-and-dis...
> easy_install, now deprecated, was released in 2004 as part of Setuptools.
Some of those are package tools, some are dependency managers, some are runtime environments, some are package formats...
Some are obsolete at this point, and others by necessity cover different portions of programming language technologies.
I guess what I'm saying is, for the average software engineer, there's not too many more choices in Python for programming facilities than in Javascript.
Whatever I would say at this point about PyPA would be so uncharitable that dang would descend on me with the holy hammer of banishment, but you can get my drift. I just don't trust them to come out with good tooling. The plethora they have produced so far is quite telling.
That said, pip covers 99% of my needs when I need to do anything with Python. There are ecosystems that have it way worse, so I count my blessings. But apparently, since Poetry and uv exist, my 99% are not many other people's 99%.
If I wanted to package my Python stuff, though, I'm getting confused. Is it now setup.py or pyproject.toml? Or maybe both? What if I need to support an older Python version as seen in some old-but-still-supported Linux distributions?
> They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
Granted, tooling is different from the language itself. Although PyPA could benefit from a decade having a BDFL.
Your Python version is irrelevant, as long as your tools and code both run under that version. The current ecosystem standard is to move in lock-step with the Python versions that the core Python team supports. If you want to offer extended support, you should expect to require more know-how, regardless. (I'm happy to receive emails about this kind of thing; I use this username, on the Proton email service.)
Nowadays, you should really always use at least pyproject.toml.
If your distribution will include code in non-Python languages and you choose to use Setuptools to build your package, you will also need a setup.py. But your use of setup.py will be limited to just the part that explains how to compile your non-Python code; don't use it to describe project metadata, or to orchestrate testing, or to implement your own project management commands, or any of the other advanced stuff people used to do when Setuptools was the only game in town.
In general, create pyproject.toml first, and then figure out if you need anything else in addition. Keeping your metadata in pyproject.toml is the sane, modern way, and if we could just get everyone on board, tools like pip could be considerably simpler. Please read https://blog.ganssle.io/articles/2021/10/setup-py-deprecated... for details about modern use of Setuptools.
Regardless of your project, I strongly recommend considering alternatives to Setuptools. It was never designed for its current role and has been stuck maintaining tons of legacy cruft. If your project is pure Python, Flit is my current recommendation as long as you can live with its opinionated choices (in particular, you must have a single top-level package name in your distribution). For projects that need to access a C compiler for a little bit, consider Hatch. If you're making the next Numpy, keep in mind that they switched over to Meson. (I also have thrown my hat in this ring, although I really need to get back to that project...)
If you use any of those alternatives, you may have some tool-specific configuration that you do in pyproject.toml, but you may also have to include arbitrary code analogous to setup.py to orchestrate the build process. There's only so far you can get with a config file; real-world project builds get ferociously complex.
uv addressed these flaws with speed, solid dependency resolution, and a simple interface that builds on what people are already used to. It unifies virtual environment and package management, supports reproducible builds, and integrates easily with modern workflows.
Anyway, if you're familiar with Node then I think you can view pip and venv as the npm of Python. Things like Poetry are Yarn, made as replacements because pip sort of sucks. UV on the other hand is a drop-in replacement for pip and venv (and other things) similar to how pnpm is basically npm. I can't answer your question on why there isn't a "good" tool to rule them all, but the single tool has been pip since 2014, and since UV is a drop-in, it's very easy to use UV in development and pip in production.
I think it's reasonable to worry about what happens when Astral needs to make money for their investors, but that's the beauty of UV compared to a lot of other Python tools. It's extremely easy to replace because it's essentially just smarter pip. I do hope Astral succeeds with their pyx, private registries and so on by the way.
It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem.
I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
They had time to force "--break-system-packages" on us though, something no one asked for.
They aren't doing this.
> They had time to force "--break-system-packages" on us though, something no one asked for.
The maintainers of several Linux distros asked for it very explicitly, and cooperated to design the feature. The rationale is extensively documented in the proposal (https://peps.python.org/pep-0668/). This is especially important for distros where the system package manager is itself implemented in Python, since corrupting the system Python environment could produce a state that is effectively unrecoverable (at least without detailed Python-specific know-how).
- https://github.com/pypa/packaging/issues/774
- https://github.com/pypa/setuptools/issues/3548
- https://github.com/pypa/pip/issues/7953
I relied on those for a decade, maybe two.
> something no one asked for
Was being a facetious, sure someone asked for it, but it was pretty dumb. This has never "corrupted" anything, is rare (not happened to me in last 15 years), and simply fixed when knowledgeable.
Not everyone can simply fix it, so a better solution would be to isolate the system python, allow more than one installed, etc. Distros already do this to some extent.
> It’s entirely volunteer based so I don’t blame them
It's not just that they're volunteers; it's the legacy codebase they're stuck with, and the use cases that people will expect them to continue supporting.
> I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
"Invest" is the operative word here. When I read discussions in the community around tools like pip, a common theme is that the developers don't consider themselves competent to redesign the UX, and there is no money from anywhere to hire someone who would be. The PSF operates on an annual budget on the order of $4 million, and a big chunk of that is taken up by PyCon, supporting programs like PyLadies, generic marketing efforts, etc. Meanwhile, total bandwidth use at PyPI has crossed into the exabyte range (it was ~600 petabytes in 2023 and growing rapidly). They would be completely screwed without Fastly's incredible in-kind donation.
The bits of it that work are useful though
Now we have pyproject.toml and uv to learn. This is another hour or so of learning, but well worth it.
Astral is stepping up because no one else did. Guido never cared about packaging and that's why it has been the wild west until now.
0. A lot of those "tools and concepts" are actually completely irrelevant or redundant. easy_install has for all practical purposes been dead for many years. virtualenv was the original third party project that formed the basis for the standard library venv, which has been separately maintained for people who want particular additional features; it doesn't count as a separate concept. The setup.py file is a configuration file for Setuptools that also happens to be Python code. You only need to understand it if you use Setuptools, and the entire point is that you can use other things now (specifically because configuring metadata with Python code is a terrible idea that we tolerated for far too long). Wheels are just the distribution format and you don't need to know anything about how they're structured as an end user or as a developer of ordinary Python code — only as someone who makes packaging tools. And "pypy" is an alternate implementation of Python — maybe you meant PyPI? But that's just the place that hosts your packages; no relevant "concept" there.
Imagine if I wanted to make the same argument about JavaScript and I said that it's complicated because you have to understand ".tar.gz (I think, from previous discussion here? I can't even find documentation for how the package is actually stored as a package on disk), Node.js, NPM, TypeScript, www.npmjs.com, package.json..." That's basically what you're doing here.
But even besides that, you don't have to know about all the competing alternatives. If you know how to use pip, and your only goal is to install packages, you can completely ignore all the other tools that install packages (including poetry and uv). You only have any reason to care about pipenv if you want to use pip and care about the specific things that pipenv does and haven't chosen a different way to address the problem. Many pip users won't have to care about it.
1. A lot of people actively do not want it that way. The Unix philosophy actually does have some upsides, and there are tons of Python users out there who have zero interest in participating in an "ecosystem" where they share their code publicly even on GitHub, never mind PyPI — so no matter what you say should be the files that give project metadata or what they should contain or how they should be formatted, you aren't going to get any buy-in. But beyond that, different people have different needs and a tool that tries to make everyone happy is going to require tons of irrelevant cruft for almost everyone.
2. Reverse compatibility. The Python world — both the packaging system and the language itself — has been trying to get people to do things in better, saner ways for many years now; but people will scream bloody murder if their ancient stuff breaks in any way, even when they are advised years in advance of future plans to drop support. Keep in mind here that Python is more than twice as old as Go.
3. Things are simple for Go/JS/TS users because they normally only have to worry about that one programming language. Python packages (especially the best-known, "serious" ones used for heavyweight tasks) very commonly must interface with code written in many other programming languages (C and C++ are very common, but you can also find Rust, Fortran and many more; and Numpy must work with both C and Fortran), and there are many different ways to interface (and nothing that Python could possibly do at a language level to prevent that): by using an explicit FFI, by dlopen() etc. hooks, by shelling out to a subprocess, and more. And users expect that they can just install the Python package and have all of that stuff just work. Often that means that compiled-language code has to be rebuilt locally; and the install tools are expected to be able to download and set up a build system, build code in an isolated environment, etc. etc. All of this is way beyond the expectations placed on something like NPM.
4. The competition is deliberate. Standards — the clearest example being the PEPs 517/518/621 that define the pyproject.toml schema — were created specifically to enable both competition and interoperation. Uv is gaining market share because a lot of people like its paradigms. Imagine if, when people in the Python community first started thinking about the problems and limitations of tools from the early days, decided to try to come up with all the paradigms themselves. Imagine if they got them wrong, and then set it in stone for everyone else. When you imagine this, keep in mind that projects like pip and setuptools date to the 2000s. People were simply not thinking about open-source ecosystems in the same way in that era.
> They should remove the "There should be one-- and preferably only one --obvious way to do it." from the Python Zen.
First, define "it". The task is orders of magnitude greater than you might naively imagine. I know, because I've been an active participant in the surrounding discussion for a couple of years, aside from working on developing my own tooling.
Second, see https://news.ycombinator.com/item?id=44763692 . It doesn't mean what you appear to think it does.
because they're obsessed with fixing non-issues (switching out pgp signing for something you can only get from microsoft, sorry, "trusted providers", arguing about mission statements, etc.)
whilst ignoring the multiple elephants in the room (namespacing, crap slow packaging tool that has to download everything because the index sucks, mix of 4 badly documented tools to build anything, index that operates on filenames, etc.)
Its a pain in the ass to work with professionally.
I'm still not convinced that the built-in vendor support added in Go 1.5 wasn't intentionally made incompatible with the community-developed solutions out of some weird kind of spite. Why didn't they just use "./vendor/src" like all of the existing tools were using? (Remember that Go was written by plan9 folks, so making a "src" symlink didn't work. In fact, the Go compiler dislikes symlinks in most places -- pre-modules the unholy things you had to do with GOPATH were awful.)
Design by committee is hard. Especially building consensus around reconsidering what at this point should be considered bad design. Python has a few decades of that. It took decades for the to even consider starting the process of removing the GIL, which is a great example of "it's clearly broken but lets not fix it". Packaging is the same.
These guys are laser focused on just fixing things the right way and that's what it takes,
There seems to be a lot of movement around some of these topics. The Gil is being removed, python's performance issues are meling away with recent releases, and things like uv are happening. I like it.
This is not true. Discussions about removing the GIL have been ongoing for decades. There were some abortive attempts over the years but it is regarded as quite the technical challenge, and has implications for compatibility with code written in C.
> which is a great example of "it's clearly broken but lets not fix it".
That is not the attitude of the Python community.
Go and look up why you can't run scripts with `python3 foo.py` on Windows. It's like a one-line fix and they've come up with all sorts of naysaying reasons rather than just adding python3.exe (which Microsoft did years ago in their Python packages).
What happens in Go/TS when you want to use a C library underneath that's dynamically linked? Oh wait, there's no mechanism in it's package manager to support this, so it's assumed it's installed already, or in Go, C code is bundled instead. This is the problem all the Python packaging stuff above has tried to solve.
You just need to understand how to setup a venv, and for each thing you want to depend on how to install it in the venv. Put those bits in a shell script and you can clone the project, run the script and then have a working env
It will break sometimes because python and a lot of python packages have zero respect for backward compatibility, but that's python for you.
It's ugly and horrible, but you don't have to relearn the latest python packaging "standard" every other year
They have: uv
> Don't understand how a private company like Astral is leading here
They wrote the single tool to rule them all.
Though speaking as a long time developer in the ecosystem, switching between npm, yarn, and pnpm is fairly trivial in my experience. Especially after node_gyp went away
The idea that every single directory for every single project down 18 subdirectories deep should have their own copy of is-even is insanity
- there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info)
- there's no way to get hashes of the archives
- there's no way to do things like reverse-search (show me everything that depends on x)
I'm assuming that a big part of pyx is introducing a dynamically served (or maybe even queryable) endpoint that can return package metadata and let uv plan ahead better, identify problems and conflicts before they happen, install packages in parallel, etc.
Astral has an excellent track record on the engineering and design side, so I expect that whatever they do in this space will basically make sense, it will eventually be codified in a PEP, and PyPI will implement the same endpoint so that other tools like pip and poetry can adopt it.
[1]: Top-level: https://pypi.org/simple/ Individual package: https://pypi.org/simple/pyyaml/
> there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info)
Not true for wheels; PyPI implements https://peps.python.org/pep-0658/ here. You can pre-download just the dependency info instead.
For sdists, this is impossible until we can drop support for a bunch of older packages that don't follow modern standards (which is to say, including the actual "built" metadata as a PKG-INFO file, and having that file include static data for at least name, version and dependencies). I'm told there are real-world projects out there for which this is currently impossible, because the dependencies... depend on things that can't be known without inspecting the end user's environment. At any rate, this isn't a PyPI problem.
> there's no way to get hashes of the archives
This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links.
But yes, there is no reverse-search support.
In principle, separate metadata availability should still at least be possible for most sdists eventually. But I'm not the one calling the shots here.
pip install -r requirements.txt --dry-run --only-binary=:all:`
> This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links.
Also, this is an implementation detail, but the archives are stored in a content-addressable store with the path being based on the (hexadecimal) blake2b/256 hash of their contents.
Sounds like Pyx is trying to do a bit of both.
Disclosure: I have interacted a bunch with folks from all of these things. Never worked for or been paid by, though.
Python packaging for for Python only modules has never been a problem. When people say they hate python packaging they are usually talking about being able to successfully install dependencies without much thinking. But, the biggest reason that doesn't work is because of the dependencies that have to be compiled. Which brings it's own problems. Have you ever had a c dependency on node or ruby on a system that wasn't the same system they built it with? Turns out it sucks in all the languages. It's just that the amount of c-level packages in python is quite larger than say ruby. The likelihood of a problem is significantly larger.
Especially in the GPU accelerated space.
Venv hasn't always been around for example, and before it was, dependency conflicts between two things running on the same machine were hairy
Python is used heavily for DS/ML/AI, which is exactly the area where native code packages are necessary and prevalent. Worse yet is that those packages often involve GPU code, and things like CUDA bring their own complications.
If you're writing web apps in Python, dependencies haven't really been a problem for a long time now.
> Ruby is mostly used in web dev, where most if not all of your dependencies tend to be pure Ruby.
Where it breaks down in Python is when the dependencies are enormous source code projects with their own dependencies in their own right, that you can't reasonably bundle in with your own project. Conda aimed to solve that by being cross-platform. In the old Python days you'd do 'pip install X' and it would fail because it expected the package to be available, on your system, and in the PATH/LD_LIBRARY_PATH. Then wheels (python packages that could bundle libraries rather than just source code) came along and people started bundling lowest-common-denominator versions that most people could use but at non-optimal performance, and that's roughly still where we are today.
The reason python is a mess is because it became the default for academics and data scientists who have no real interest in code quality or maintainability and the rot just spread from there.
But you can if you want have a non-flat namespace for imports using PEP 420 – Implicit Namespace Packages, so all your different packages "company-foo", "company-bar", etc. can be installed into the "company" namespace and all just work.
Nothing stops an index from validating that wheels use the same name or namespace as their package names. Sdists with arbitrary backends would not be possible, but you could enforce what backends were allowed for certain users.
import io
from contextlib import redirect_stdout as r
b=io.StringIO()
with r(b):import this
print(b.getvalue().splitlines()[-1])
Sure, astral's products are remarkable and widely loved, but I would like to understand if there's a USP beyond that?
They did say they want their thing to have understanding of the code, so maybe they’ll sell semgrep-like features and SBOM/compliance on top. Semgrep is ok popular, but if it maybe bundled into something else (like the package registry itself) that might get enough people over the line to buy it.
Private registries and “supply chain security” tools individually aren’t the hottest market, but maybe together the bundle could provide enough value. Let’s see how it goes.
Full disclosure, uv rocks and is way better than poetry, setuptools and whatever complicated and non-zen way packaging python has had in the past.
Python should get rid of its training wheels :^)
(As the user in question: three times.)
> Leverage progress in the systems programming ecosystem to create repeatable builds. Turn prebuilt binaries from “sources” into cacheable artifacts that can be deleted and reconstructed at will. Institute a way of creating secondary caches that can start shouldering some of the workload.
This doesn't avoid the need for the wheels to exist and be publicly available. People running CI systems should figure out local caching that actually works, sure. But if you delete that cacheable artifact on the public PyPI website for something like, I don't know, numpy-2.3.2-cp312-cp312-win_arm64.whl, you're going to be re-creating it (and having it downloaded again) constantly. Windows users are just not going to be able to build that locally.
And you know, space usage isn't the problem here — we're talking about a few hard drives' worth of space. The number of downloads is the problem. Outside of CI, I guess that's mostly driven by end users defaulting to the latest version of everything, every time they make a new virtual environment, rather than using whatever's in their package installer's cache. I do know that uv makes the latter a lot easier.
I think that's being too kind to Linux + Mac users, they are often not generally able to do it either in my experience, especially with Python stuff which tends to need more bleeding edge deps than the local package manager can supply.
Executing a Python script in the same directory as some sort of project.json file that contains all the complicated dependency details would be a pretty good solution to me. But I'm probably missing a whole bunch of details. (Feel free to educate me).
In general I really dislike the current system of having to use new environment variables in a new session in order to isolate Py scripts. It has always seemed like a hack with lots of footguns. Especially if you forget which console is open.
Or put your dependencies inline in the script (https://packaging.python.org/en/latest/specifications/inline...), and it's just a single file script again. But with dependency support
EDIT to clarify it was just the pyproject.toml that inline metadata could replace
I was suggesting that the Python executable have this sort of functionality built in, with no need for a new session.
But I don’t get it. How does it work? Why is it able to solve the Python runtime dependency problem? I thought uv had kinda already solved that? Why is a new thingy majig needed?
uv becomes a client that can install those packages from your private registry.
I imagine pip will be able to install them from a PYX registry too.
The dependencies in question are compiled C code that Python interfaces with. Handling dependencies for a graph of packages that are all implemented in pure Python, is trivial.
C never really solved all the ABI issues and especially for GPU stuff you end up having to link against very specific details of the local architecture. Not all of these can be adequately expressed in the current Python package metadata system.
Aside from that, a lot of people would like to have packages that use pre-installed dependencies that came with the system, but the package metadata isn't designed to express those dependencies, and you're also on your own for actually figuring out where they are at runtime, even if you take it on blind faith that the user separately installed them.
This is not really my wheelhouse, though; you'll find a much better write-up at https://pypackaging-native.github.io/ .
Hrm. This doesn’t sound right to me. Any project should target a particular version of Cuda and then the runtime machine simply needs to have that version available. Right?
> a lot of people would like to have packages that use pre-installed dependencies that came with the system
Those people are wrong. Everything these days requires Docker because it’s the only way to deploy software that can reliable not crash on startup. (This is almost entirely a Linux self induced problem)
> pyx is also an instantiation of our strategy: our tools remain free, open source, and permissively licensed — forever.
> Beyond the product itself, pyx is also an instantiation of our strategy: our tools remain free, open source, and permissively licensed — forever. Nothing changes there. Instead, we'll offer paid, hosted services that represent the "natural next thing you need" when you're already using our tools: the Astral platform.
pyx itself is not a tool, it's a service.
To be precise: pyx isn't intended to be a public registry or a free service; it's something Astral will be selling. It'll support private packages and corporate use cases that are (reasonably IMO) beyond PyPI's scope.
(FD: I work on pyx.)
> but the bigger picture is that pyx part of a larger effort to make Python packaging faster and more cohesive for developers
Maybe they are only targeting dev tooling companies as a way to simplify how they distribute. Especially in the accelerated compute era.
Really useful concept especially for school.
This way, !each of my repositories has its own nix file defining the list of dependencies and it can easily be built per system as well
PYX is a package registry, and therefore an alternative to PyPI (like how JSR is an alternative to NPM).
The alternative to `pip` that Astral has built is called `uv`. Feel free to not use that either, but personally I'd consider it if I were you. It has full pip compatibility (with significantly more speed), and many other nice features besides.
Likewise with uv, speed is only one of the (many) improvements over pip.
From the guidelines [1]
> Please don't post shallow dismissals, especially of other people's work.
Edit: and turns out a smart guy is top rated comment atm
"All python packaging challenges are solved. Lesson learned is that there is not a single solution for all problems. getting more strings attached with VC funded companies and leaning on their infrastructure is a high risk for any FOSS community."
I wanted to start a business exactly like this years ago, when I actually worked in Python. I ended up not doing so, because at the time (circa 2014-2015) I was told it would never take off, no way to get funding.
I'm glad you're able to do what ultimately I was not!
the two ways of spacing the em dash on that quote is a joke about how it's not actually possible to do that? (And there's a third way in another line of the zen)
Do I buy it? Not sure. But apparently there’s more to this line than what it suggests.
1. do dev in an isolated environment like docker
2. install deps from config - ie. build a new image when adding a dep
Much ad language.
They do not explain what an installation of their software does to my system.
They use the word "platform".
[1] https://rwsdk.com/blog/saas-is-just-vendor-lock-in-with-bett...
Thankfully I only use it for OS and application scripting and keep myself to the standard library and application exposed APIs as much as possible, even if there are marvelous alternatives outside the standard library.
I'll pass. I'd rather have the battle-tested old thing, thanks.
I don't understand why people who don't do weird AI stuff would use any of that instead of sticking to distribution packages and having the occasional 1 or 2 external modules that aren't packaged.
well getting a good system is step 1
It's the default installation path for aider, for instance.
(I've seen https://xkcd.com/927/ in the comments a few times, but apparently I'm the first one who had this one in mind)
python has burned me with it's packaging so many times.
Still don’t get how they are solving what they claim to solve.
It's easy to compete with Nix tooling, but pretty hard to compete with the breadth of nixpkgs.
They explicitly mention PyTorch in the blog post. That's where the big money in Python is, and that's where PyPI utterly fails.
It would be nice if they contributed improvements upstream, but then they can't capture revenue from doing it. I guess it's better to have an alternative and improved PyPI, than to have no improvements and a sense of pride.
There is a lot of other stuff going on with Pyx, but "uv-native metadata APIs" is the relevant one for this example.
> Private registry
ouch.