Hacker News Reader

Comment by digikata | Hacker News Reader

The added cargo env variable pointing to the path of generated binaries is a nice quality of life fix. I think that was one of the first annoyances I ran into with cargo and binaries. It was minor but was like a little splinter that stood out from cargo's otherwise nicely integrated build and test capabilities.

View on HN

I've been using https://docs.rs/assert_cmd/ for this. (Helps spawn the binary and assert output.)

Thamks for the call out! I'm excited by the new env var and need to consider its impact on `assert_cmd`.

The real problem though is that cargo doesn't play nice as part of system where cargo doesn't own everything.

Cargo needs a lot more ways to be interrogated about the artifacts it produces and control over how they are produced.

Exactly this. It would be really great if for example when building Rust in Nix, Nix could own each of the artifacts, in order to share artifacts between tools that use the same revisions of dependencies, as well as to avoid rebuilding artifacts unnecessarily on every change to a derivation.

How does Nix handle areas with extensive package ecosystems? Python, Node et al? It seems like every distro/super packager has different compromises for this question.

I haven't investigated Python, but for Node it uses a tool node2nix that fetches the metadata for the entire dependency tree and constructs a Nix expression for each one, such that each expression represents the source for that specific dependency. It then takes the top-level package and makes a call to `nodeEnv.buildNodePackage`, which contains the source for that top-level package and declares the entire recursive dependency tree. buildeNodePackage itself rebuilds the entire node_modules folder from the declared dependencies and reconstructs the lockfile. And then it asks npm to rebuild the package.

The end result is this bypasses npm's downloading and managing of node_modules, does it all in a fully declarative fashion, and then hands control over to npm to build the package. Also, each dependency's source is cached independently by Nix, so if multiple packages use the exact same dependency, it doesn't have to be redownloaded (unlike Rust where the entire dependency tree is vendored separately per package).

> (unlike Rust where the entire dependency tree is vendored separately per package).

To be clear, Cargo downloads the source globally. The compiled output is per-project, however.

Nix uses `cargo vendor` to literally vendor the source of all dependencies as a separate derivation. Maybe cargo keeps a global cache of the source as well when doing `cargo vendor`, I don't know, but Nix does construct a derivation that contains just the vendored dependency sources and then provides that as input when building the full crate.

I'm curious why there's no similar variable pointing to a built library (e.g. a cdylib).

What's the use case?

I feel like I've been pushing my use case plenty already, so please take this as an answer to your question rather than one more push ;-)

I'd like to be able to create plugin-style cdylibs that get loaded into a host program. For instance, PostgreSQL extensions are shared libraries (typically written in C) that need to be loaded into a running instance of postgres for integration testing. I am trying to make it reasonable to write such extensions in rust (https://github.com/jeff-davis/postgres-extension.rs).

The environment variable to find the shared library would solve one problem, but there are still a couple more right behind it:

* Running "cargo test" tries to build a binary that links to the library to run the unit tests, but this will fail because there are unresolved symbols in the .so file (symbols that call APIs in the host program, in this case PostgreSQL functions like palloc()). In other words, the test binary is useless because it can't load the plugin.

* Linux seems fine creating a .so with unresolved symbols, but Mac and Windows need special linker flags. I can document that the linker flags are needed by consumers of my crate, but it would be nice if there was a better/supported way to create plugin-style shared libraries. I tried filing a bug with PR (https://github.com/rust-lang/rust/pull/66204) it was determined that it needed to go through the RFC process.

The simplest way to see the problem is to look at this crate: https://github.com/jeff-davis/unresolved

My problems don't end there, unfortunately. For my crate to really work, I also need to solve the problem of setjmp/longjmp at the edges between C and Rust (https://github.com/rust-lang/rfcs/issues/2625).

It's all good; I was asking because I legit didn't know, and couldn't think of anything off the top of my head. I don't work on Cargo, so I'm not as in-tune with its development.

So, I am still not 100% sure that I understand, mostly because each package can only have one library, so this would only work if you wanted one plugin. I guess maybe if they were all in a workspace?

Typically one postgres extension would have only one plugin library. So, the idea is that one crate would produce one plugin. I guess it's possible to have more than one plugin per extension, but it never occurred to me and I haven't seen that before.

Having the location of the plugin might also allow "cargo install" to work.

Reply on news.ycombinator.com