Rip – Rust crate to resolve and install Python packages | Hacker News Reader

Rip – Rust crate to resolve and install Python packages

prefix.dev

135 pointsdroelf36 comments

> The above invocation will automatically download the latest version of pixi, extract it, and move the pixi binary to ~/.pixi/bin. If this directory does not already exist, the script will create it.

Please don't pollute my home directory. Follow the XDG base directory specification [1].

[1]: https://specifications.freedesktop.org/basedir-spec/basedir-...

There are even libraries for that!

https://github.com/dirs-dev/directories-rs

Blog post author here – happy to take any questions. We're looking to unify the conda + PyPI ecosystems and build a really nice cross-platform, language independent package manager (which is called pixi).

Any chance you guys, Charlie Marsh/Astral, and Armin Ronacher will ever team up? I’m really excited about next-gen tooling but would love a cargo-like “one true path”. At the very least keep the tools compatible?

Yeah, I think it would be awesome to work on top of the same foundations (rip & resolvo being two of those, and konstin's, who works with Charlie at astral, pep440_rs and pep508_rs being two others).

We'll have some more conversations around this in the upcoming weeks I hope.

I noticed you're an incorporated organisation. What's your business model?

We'll build some features sooner or later on the repository side that are going to be interesting for enterprise customers.

But we have a strong and true commitment to open source. A non-open-source package manager would definitely not work for our community!

as an arch user; i gotta say `aur/rip` is already in use; and i can only say good luck if you keep to a 3char name

Last updated: 2015. Sourceforge.

Perhaps packages that are borderline irrelevant should not squat on prime name real estate. Plus cant rip ship as rip-py and still install as rip on the bet that nobody is actually going to be trying to use these tools simultaneously?

Name reuse is a huge security concern and probably shouldn't be done even if a name is under-utilized.

rip isn't really planned as a tool that you would install on your computer. It's a low-level library.

Agreed no one uses those defunct tools like awk or roff, let's re-use them!

Who cares if every build script in the world breaks, progress waits for no one.

Also rm replacement written in rust: https://github.com/nivekuil/rip

This is really impressive work tbh. Good job.

Any plans to integrate this with Poetry?

No plans but we would be happy to explore this further if the poetry-devs want to. Maybe they'll join us at PackagingCon next week: https://packaging-con.org/

Is it different than rye (rye-up.com)?

Yeah, rye uses a Python build from indiegreg while we use the Python builds from conda-forge (which is a community project with many maintainers).

Conda-forge also produces packages for much more (nodejs, Qt, HDF5, C/C++ compilers, LLVM ...).

So we have access to a pretty big repository and a large community already :)

Are there any gains to be had in just using this for pypi packages (no conda)? Solving for a monorepo with hundreds of packages is the cause of much time lost.

Our solver implements "state-of-the-art" CDCL (conflict driven clause learning). So there is a chance that the performance is improved over "resolvelib" which is used by pip.

Once we are feature complete enough we will work on some more benchmarking.

How well does this work for projects that include dependencies that are still reliant on the old setup.py and don't have wheels published?

It doesn't work well for those yet. rip only deals with wheel files for now. The way (afaik) pip works for source dist packages is that it locally builds the wheel, and then extracts the metadata from that locally built wheel.

This is also the approach that we want to take with rip. As you can imagine, the performance will be quite bad (but nothing we can do about that). Ideally, almost all packages should ship wheels.

Ideally - however there are points where it's not possible yet.

Bindings like pycairo are a good example.

On windows a WHL can be provided.

On Linux it can't - Cairo (which pycairo binds to) is shipped by the distro, and distros are free to enable or disable different features.

Since the C bindings are linked to the cairo shared library, and that includes all the backends you can't build that and know if would work in a distro.

There are two use cases for pycairo - in one you might not care what the system provides and if the pycairo who provided its own Cairo shared object, that would be fine.

In the second use case you want to use the system provides Cairo, e.g. to work with Gtk.

It's impossible to resolve this workout changing how Cairo itself works and it's API.

The result is no pycairo whl on Linux, and users who have to install all the dependencies (which once go to pango you pull in Gtk, X, freetype etc).

Cairo isn't the only example but it's one I know.

A lot of these are libraries that bind system libraries and predate virtualenv / venv - when they were created compiling things was fine.

> Ideally, almost all packages should ship wheels.

It would be ideal but it's not always possible. I maintain a GSSAPI/KRB5 library that wraps the C libs but due to PyPI wheel policies I cannot upload a wheel without embedding those C libs which then opens up a whole bunch of problems around deps and lib conflicts :(

> all packages should ship wheels.

And then a new python version is released and virtually no packages with native modules have wheels for that new python version. (too few use abi3)

Also, good luck finding wheels for e.g. s390x linux.

pypa/cibuildwheel: https://github.com/pypa/cibuildwheel :

> Example setup: To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this .github/workflows/wheels.yml

That's great and all for packages I control, but I publish wheels already.

https://pythonwheels.com/ now lists only three packages that are not yet published to PyPI as wheels.

But do you or others build them with SLSA 3? https://slsa.dev/get-started#slsa-3

There's thousands more packages than listed on that website.

Also now you are moving the goalposts. Python wheels exist but not all Python packages are distributed as wheels.

If I am going through the effort to build wheels myself, then I don't need a resolver since I can just make sure to pick all the versions I need. Then I don't need rip to install anything.

I sure hope there’s a pnpm-style content addressable storage and space-saving capabilities

Yeah, we're working on that and use a similar model like pnpm in pixi / rattler already.

In fact, for a long time conda & mamba have created "hard-links" to a central package cache for the mentioned space savings.

Been quite happy with PDM last 6 months, that's pretty fast, how does this stack up?

https://github.com/mamba-org/rattler :

> Rattler: Rust crates for fast handling of conda packages

https://github.com/prefix-dev/pixi :

> https://pixi.sh/

If I understand it correctly, rip::pip as to what rattler::conda? Finally pixi brings them altogether as the pkg manager to rule them all. Lofty goal but I remain optimistic that something better (esp in python land) is just around the corner.

yep, that's it!

Yeah, rattler are our low-level crates to handle conda packages (similar to how rip is a low-level crate to handle PyPI packages). Both build on top of resolvo (our SAT solver, based on MiniSAT and libsolv) and both are used (or going to be used) in pixi, which is our package manager.

Reply on news.ycombinator.com