Riff, automatically provide external dependencies for Rust projects | Hacker News Reader

Riff, automatically provide external dependencies for Rust projects

determinate.systems

96 pointsgrhmc26 comments

Have you considered integrating this functionality with this nix project builder? https://bob.build/

Or this dependency installer that uses nix? https://www.jetpack.io/blog/devbox/

It is amazing to see all this user-friendly tooling come out now that is leveraging nix.

We're currently working on adding Riff to Nixpkgs, and as soon as that happens you'll be able to use it as a dependency in Bob.

As for Devbox, that's certainly an interesting tool but not really something that can be integrated with Riff due to the language difference (Go vs. Rust), but it is fair to say that the projects have related goals.

If you have any questions or comments about Riff, we at Determinate Systems will be actively monitoring this space for the next few hours, so fire away.

This looks really exciting, but I'm not clear on how this works:

  So what has happened here? Riff has:
    - read your project’s Cargo.toml,
    - determined which external dependencies 
      are necessary based on your crate 
      dependency graph,

How does Riff go from crate -> external package dependencies? Are those listed in the Cargo.toml somewhere? Does it just make a best-effort attempt to scrape the project and figure out what's referenced? Or does Riff maintain its own internal registry of which dependencies are needed by which popular crates?

Riff has an internal registry mapping crates to external dependencies. However, the registry is intended to bootstrap Riff's utility. If projects define their own Riff metadata in their own Cargo.toml, that data is preferred. The README shows how to do that: https://github.com/determinateSystems/riff#how-to-declare-pa...

Neat! Sounds reasonable

I think there is a lot of fertile ground in the area we're researching here. I'd be into talking more about where we can go, and how Nix is behind it.

Is there a newsletter I can subscribe to so that I get get notified of new posts to your blog?

I have a strange feeling when thinking about the solution especially on macOS. The nix installation needs its store located at /nix which got harder to initialize since macOS Catalina or Mojave added the root protection. In general this is no problem when installing nix on its own. But when a wrapper tool does this and breaks the setup etc that feels hrmm problematic.

I see potential setup issues coming up with weird store error messages no normal human being being able to understand.

I really like nix and nixos. But I had already a nix setup which refused to build a rust project because of some weird linkage errors with libc or something. And I couldn’t make heads or tails out of it.

But I will check out this tool to get a better feeling. A general solution to this issue would be awesome. I would prefer JNI build support as well ;)

What do you mean about when a wrapper tool "does this"? Riff doesn't install its own Nix.

I only checked the post so far and my understanding was that it uses nix. It is ironic that a tool who sets out to solve the issue of installing external dependencies, needs a preinstalled nix setup.

My understanding was that it uses nix under the hood and the user has no real interaction with this part. But come to think of it it kinda makes sense.

Managing Python projects with Nix has been unbelievably painful for me - so hoping there is Python support soon. In general super excited that this tool is coming from Nix OGs.

Have you tried Mach-nix?

Yes, when it works it is great. But when it does not it is insanely hard to debug.

> While Riff does its best to infer external dependencies from your project's crate dependencies

How does it do this? Just a list of known packages and their dependencies, or?

I'm using https://github.com/LaurentMazare/tch-rs/ (rust bindings for pytorch's c++ api). Depending on the hardware I'm building for I either need to download a cpu build of pytorch, or a cuda build of pytorch. How would this work with riff?

Is there any hope of a good cross compiling story? If so, that would be awesome (currently I just don't even try if there are significant C dependencies)!

How thoroughly are dependencies cached? Backed to the pytorch example it's a fairly big (and thus slow) download, I don't want that happening too frequently.

---

In general this looks like something that could be really useful. Using projects with significant C dependencies currently sort of sucks, and this looks like it might actually fix the problem :)

> How does it do this? Just a list of known packages and their dependencies, or?

It checks a known registry or reads package metadata.

> I'm using https://github.com/LaurentMazare/tch-rs/ (rust bindings for pytorch's c++ api). Depending on the hardware I'm building for I either need to download a cpu build of pytorch, or a cuda build of pytorch. How would this work with riff?

Riff doesn't track which GPU you use at this time, so it can't track that which build of `pytorch` to use, sorry. It might be something we make it aware of if we have enough evidence it would be helpful.

> Is there any hope of a good cross compiling story? If so, that would be awesome (currently I just don't even try if there are significant C dependencies)!

Regarding cross compiling: We've spoken about it, and I certainly believe it would be desirable. I imagine it would come in a later version. We've taken some steps to ensure we don't rule out that feature later.

> How thoroughly are dependencies cached? Backed to the pytorch example it's a fairly big (and thus slow) download, I don't want that happening too frequently.

Dependencies are cached by Nix, so they should not become invalidated very often. If your tempdir erases we may lose the generated lock file (We've been discussing where to place these perhaps more permanently) which may force another download if there was an update.

> Riff doesn't track which GPU you use at this time, so it can't track that which build of `pytorch` to use, sorry. It might be something we make it aware of if we have enough evidence it would be helpful.

Would it be possible to do something like make it depend on a cargo feature flag, or something similar?

It's probably better that this isn't automatic, since the correct choice of device isn't always obvious just based on what is available.

This looks like a really good idea for solving that pain point

> Riff is a tool that we built to help developers write software without having to wrangle dependencies or complicated configuration. It’s built to enable you to clone a project and get to work in seconds. ...

The title specifically mentions Rust projects. The main competition there is Cargo, which does exactly what the quote above claims. It's not at all clear what Riff offers over Cargo.

It is fairly clear that it's not in competition with cargo, since it is literally based on cargo. It installs dependencies that are not managed by cargo, apparently mainly C dependencies.

external means non-rust dependencies here.

I mean, it is expressed pretty clearly in the article; it uses Nix to manage external dependencies, and manages setting link paths and other build bookkeeping.

Thanks. I scanned right past the most important part for the article to get get right, mainly because it was buried in a paragraph, and wrapped in parenthesis no less:

> ... But unfortunately they’re not so great at installing dependencies written in other languages (we call these external dependencies). ...

I'm sure it's a fine project. However, the article itself is not great at explaining the value proposition.

It looks awesome, but couldn't it be a cargo plugin instead? (Or nix plugin).

Something like cargo --nix build ...

The problem with the name riff is that it's a new tool name to learn again, and nix already has name recognition. I feel that there are just too many tool names to learn in the open source ecosystem.

It could be a Cargo plugin, and we looked at doing that. There may also be some neat tricks around build.rs integration.

However, we decided to wait on those. After all: it only supports rust today, but we're eyeing a future where we support Go + Javascript too. We want Riff to work relatively consistently across these ecosystems.

Ah, riffing it I see.

Reply on news.ycombinator.com