Cargo: predictable dependency management
blog.rust-lang.org
blog.rust-lang.org
Maybe the reason why it's hardly talked about is because it's common sense and pretty much all dependency managers support it?
Except node of course, because they have no idea what they're doing.
I'm just waiting for this to happen to bower next. AFAIK they're just a registry pointing to github. All it's going to take is someone doing a force push without thinking and we're in this same situation again.
NPM and node are two different things.
* people publish stuff inadvertently (i.e. private information/keys) * people publish stuff they are not allowed to (i.e. copyright and trademark violations) * people publish stuff you do not want to see published (i.e. stuff related to breaking legal or ethical laws)
It can be argued that users must not do that or that cargo.io doesn't have to oblige, but if they, for example, get a DMCA notice they'll still have to.
As far as satisfying support requests: obviously exceptional stuff like DMCA gets handled, but package owners publishing keys, etc. is still their responsibility and shouldn't accelerate or even guarantee removal IMO. If you publish secrets, it's not crates.io's responsibility to help you hide that mistake, and you need to be changing those secrets anyway.
What does this mean? Rubygems and npm both allow a full unpublish, not only a yank. crates.io does not. Rubygems and npm are the same in this regard.
Remember that we are not talking about npm unpublishing someone's library without their approval (that's a different issue), we are talking about npm allowing a user to unpublish a library, which crates.io does not allow a user to do. I will repeat, because people seem not to believe it: rubygems allows this as well, specifically because the maintainers of rubygems could not handle the support tickets that resulted from not providing this feature.
The idea that this is some amateurish aberration from npm is a myth. If Rust is lucky, someday crates.io will have to choose between paying someone to field these tickets or letting users unpublish code.
That is, AFAIU, the reason the rubygems.org maintainers allow it now.
http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html
"If you’ve pushed a gem with internal code, you still need to reset API keys, URLs, or anything else sensitive despite the new behavior."
And:
"...we’ve been using an Amazon S3 bucket to store the gems for years now with versioning on - so if someone does remove gems that are necessary, we can easily restore them."
So what they've really done is given developers the illusion that the unwanted gem has been removed, while introducing the ability to break everyone's workflow just like npmjs. In some ways this is worse than before; devs still need to change secrets, and if it's non-secret sensitive code they are concerned about, it's still 'out there' and the dev still has to trust that the rubygems.org people don't do something unwanted with it.
If you publish secrets, you're causing crates.io to perform copyright violations. It's not just about helping you hide the mistake, but about helping to stop further violations.
You shouldn't need to resort to a formal DMCA request to stop copyright violations.
But secrets can have many other kinds too, like accidentally publishing code which is under a restrictive license.
That's how RubyGems used to work, except you could contact the support team and ask them to permanently delete your gem version if something really sensitive and irrevocable was put into it. They had to change that due to their support log getting too big: http://blog.rubygems.org/2015/04/13/permadelete-on-yank.html
> A yank does not delete any code. This feature is not intended for deleting accidentally uploaded secrets, for example. If that happens, you must reset those secrets immediately.
Also I just wanted to give a little history because eridius's original comment made it sound like it was a novel concept.
> if that goes away so does the Cargo index
Cargo addresses the namespacing concern here http://internals.rust-lang.org/t/crates-io-package-policies/...
> This is enough information to uniquely identify source code from crates.io, because the registry is append only (no changes to already-published packages are allowed).
That makes me wonder: Is it easy or possible to replace crates.io with a self hosted repository?
Background of the question is that I know of a company where access to the standard public maven repo is forbidden. They use a commercial repository provider but I don't know if it is hosted on premises.
The feature that I'd like to see but haven't found the time to implement is delegation: "look at this index first, but if it's not here, go look at this other one". Right now, if you spin up your own crates.io and point Cargo at it, it won't have any packages... which works for some people, but not others.
I agree that this would be very helpful for a lot of people, but it is kind of the opposite of what I was asking about.
As far as I understand it the commercial repos try to solve two main concerns:
1. license compliance
2. security
Better performance and reliability are just additional benefits.
I only know the details for a certain Fortune 500 company. They don't want the builds to fetch packages from a site they don't control, and they certainly don't want "if it's not here, go look at this other one". The idea is more control about where the packages come from, not more flexibility.
I think if Cargo doesn't provide a way support alternative (possibly commercial) repos, it would be an obstacle to the adoption of Rust in the corporate world.
In maven the repo URL is configurable in settings.xml. This URL can be different for different departments of even different projects.
From what I see in the cargo source the crates.io URL is hard coded. So the DNS is the only level of redirection we have. Using varying IP addresses for crates.io for different departments or even projects wouldn't fly, at least not in the world I live in.
>ignore the broader OSS ecosystem
It's not about that either because the commercial repos contain very much the same OSS packages as the standard repo but don't present all of them to everyone all the time. Take for example a car company: GPL3 for in-house projects that are just used by the employees are discouraged but somewhat tolerated. GPL3 for projects that run in the car are a big big no no. You want to be certain that no dev ever introduces GPL3 source into anything that is in the car. You want your build to fail if any of packages change their license to GPL3. You want your build to fail if any of you packages has a known vulnerability.
I know Cargo is not maven, but I believe this is a feature which is crucial for industry adoption. I think I will just add a feature request for this on GitHub.
> Sorry for pestering again
> From what I see in the cargo source the crates.io URL is hard coded.
TL;DR:
[registry]
index = "URL_GOES_HERE"
> It's not about that either because the commercial repos contain very
> much the same OSS packages as the standard repo but don't present all
> of them to everyone all the time.
- crates.io is down
- crates.io says this cached package doesn't exist.
- etc...
(And I'm not sure what that has to do with namespacing.)
What's the difference between a "username" resulting in nike/package-name and a package called nike-package-name?
Not much in practice. Name-spacing doesn't mean "add another made up name entry somewhere".
That's why people who thought about these issues out-sourced this concern completely: Prove your ownership of nike.com and you can publish as nike.com.
Suddenly 99.9% of the trademark issues are gone, handled by registrars.
I think this approach is pretty obvious and I find the lack of thought coming from rust devs deeply concerning.
I have no dog in the fight over whether or not a global namespace is a good idea, but it's tiresome to see irrelevant arguments being trotted out against global namespacing.
Even Node has overtaken Rust.
http://blog.conan.io/2016/05/03/New-conan-release-0-9.html
Arne Mertz has a Vagrant box for people who want to experiment with a "4C" development environment (Clang, Cmake, Conan, Clion).
EDIT: Also worth noting that a common response I've heard to "I need a better manager for my dependencies" is "you have a system package manager for a reason."
I have been using it, adding some libs and even contributing some changes. You should really give it a try!
For *nix based platforms Meson is now my go to build manager for c++ projects.
Are there any big differences between Cargo and Python/Go/D/Ruby/Javascript/etc?
On the other hand I see that Crates solves the problem of dependency when 2 different version of the same library are used across the project. Maven doesn't solve that and it's painful. However this seems a bigger problem that just a package manager I suppose.
Both Ruby and Python are almost as old, if not older than Java. JavaScript is only slightly younger.
I think linux (*nix?) vs non-linux heritage might have something to do with it.
If the answer is that indirect dependencies are still visible, I'd be interested in knowing if rust-lang/cargo plan to change that, similar to JDK9's module system for re-export.
Longer answer: when running rustc, Cargo passes individual `--extern bar=/path/to/bar.rlib` options for each direct dependency, not just a path to a directory with everything. You can see the exact command with `cargo build --verbose` or `cargo build -v`.
However, crates can re-export symbols they have imported from their dependency, which couples their semver to that dependency's in a way that people have often not realized. This has been the source of some upgrade headaches so far, but its less of a problem than it has been with languages in my experience.
Cargo also refuses to follow the XDG directory specification for its per-unix-user configuration; it should put things in $XDG_CONFIG_HOME and $XDG_CACHE_HOME, but instead dumps both configuration and downloaded/compiled stuff in ~/.cargo.
This isn't true: actual build artifacts are cached in a per-project way (in `./target` by default). Cargo does download all packages to a user-shared directory, but this directory is essentially immutable, just a list of the source code.
Sharing more broadly than per-project is the form that is unreliable, things can change subtly between projects e.g. different targets, different compiler flags. Of course, cargo has essentially full information about everything involved in a build and so can track this---in the limit caching all the different configurations of each crate version---but I don't think it does currently (I recall an issue about it, but I cannot find it at the moment).
I was going to submit an issue but one already exists [1][2]
https://internals.rust-lang.org/t/is-a-shared-build-cache-a-...
TLDR: There's interest, but some wrinkles need to be sorted out.
AFAIK, all it takes is a single `build.rs` to make rustc, and therefore Cargo, an impure 'function'. I'm not sure about compiler plugins, but I expect them to behave the same way.
I've experimented with using this to "vendor" dependencies in a local .cargo and so far that seems to work. An early example with discussion of the python version is here: https://www.reddit.com/r/rust/comments/3ea6je/is_there_a_bet...
I'm not sure concurrently writing to a group-writeable cache-folder is such a hot idea -- even things like apt use a write-lock when doing updates.
I suppose a third option is to mount (or probably symlink) ~/.cargo on a filesystem with deduplication -- but that wouldn't buy you caching for free (might work well coupled with a caching http proxy though).
My unix does not follow XDG[0], neither of these are set and the XDG "fallbacks" are utter garbage, now what?
[0] hell, only a minority of linux distros do at all
1. Like every enterprise build system I've ever gotten to use, I wish Cargo would manage the compiler version as well.
2. The compiler forces me to mark `unsafe` functions/blocks to use unsafe code. Why does Cargo not force me to mark `unsafe` dependencies to use unsafe code?[0]
[0] Only dependencies that explicitly use `unsafe` should need to be marked in my Cargo.toml. eg. Iron doesn't use `unsafe`, but Mio does. Iron depends on Hyper depends on Mio. If I have a dependence on Iron, I really want to be able to mark `#[unsafe_deep_dep] mio`, or else my build fails.
Only crates which use `unsafe` explicitly should need to be whitelisted.
Fun Example: I need to depend on Iron. Iron boasts in its README that it doesn't use `unsafe`. Cool.
However, Iron depends on Hyper. Hyper is complicated, it needs to use `unsafe`, I can understand that. Hyper is also used by a lot of people, I trust it, and whitelist Hyper. My build passes, I know only `std` and Hyper use `unsafe`.
Why is this helpful?
Hasn't happened yet but @reem, being human, adds an `unsafe`[0] and lets Iron's README go out of date. It happens. However, as a user, I don't like to trust documentation, and I don't like to trust other developers[1]. When I update to this version of Iron, I'd rather my builds now failed. Then I can decide if this escalation of responsibility is appropriate.
This is especially true in an enterprise context where teams of new hires do whatever it takes to push features, and "accidentally" push some really unsafe code to a deep dependency[2].
[0] Lets pretend this `unsafe` was actually used in production code: https://github.com/iron/iron/commit/ba4d197030067d4347134c2e...
[1] Hence my other preferences for type checking, core reviews, and tests.
[2] Not everything is manually catchable in code reviews. Sometimes its 3am. Sometimes whoever it was, was really new and the CR is 14 pages long and needs to be deployed Thursday! Sometimes a reviewer is just having an off-day. It happens.
Even slightly discouraging the use of unsafe via Cargo will push crate owners to find already trusted "miro-crates" which abstract their usage of unsafe. Imagine something akin to left-pad existed solely to abstract unsafe. Most crates already use it, so most teams already whitelist it. Using finding and using a micro-crate might mean less friction for users to add your dependency into their application.
Benefits:
1. Smaller the crate, smaller the surface area, better the audit-ability.
2. The more people using the same crate, the less probable there is a bug.
3. If a relatively small crate exists, solely for its abstraction of `unsafe`, and it is relatively popular, then that could be a good indicator for moving that logic into `std`.
With rustup.rs becoming official, it's a much better way to handle compiler versions since it handles cargo versions as well and allows the two to be coupled together. This allows for tighter integration between cargo and rustc as well as making testing of new versions of cargo significantly easier (i.e. there's no permutation matrix of whether cargo A.B.C works flawlessly with rustc X.Y.Z)
They are both binaries I need to install that help me manage my Rust application's dependencies. The only arguable difference is that Rustup manages "build dependencies" where Cargo manages "code dependencies". However, the tools I've used, managed both those concepts, and did so seamlessly (I wish I could share more).
Meanwhile, I don't believe Rustup.toml currently exists (I'd love to be corrected). Since other people like the difference between Cargo and Rustup, I'm happy to change my Wishlist 1.
1. I wish Rustup had a Rustup.toml file, such that my git repo could manage the currently used version of rustc. Primarily for consistency across teams, and seamless transition when I `cd ../<other_application> && cargo build`.
I feel like Rust and Cargo are in a great position to deterministically (per compiler version, per set of build flags) build libraries. That would be an amazing step forward in security if it could be enforced. Does anyone know if this is planned?
#[deny(non_deterministic)]
which would error if `build.rs` or similar is present? Also, for usage of things like the `file!` macro, which might mess up determinism based on the build directory.
Theoretically, syntax extensions could be written to be deterministic as well (for a given version, of course).
Is there a tracking issue on this currently?
Unfortunately, unlike with borrowck, reproducibility is inherently hard to verify, and you'd have to do it manually with a VM or some different build environment.
This is interesting: https://github.com/rust-lang/rust/pull/33296
This wouldn't be the first time Servo has forked a dependency, for example they maintain a fork of glutin (library that handles opening windows) because they want certain features that upstream doesn't.
Curious what features servo/glutin has that are exclusive?
Libraries can be compiled as static libraries or dynamic libraries.
even static libraries might want to do some dynamic linking (for
example, against the system version of openssl).
Similarly, applications are often built for different architectures,
operating systems, or even operating system version...
Compiling winapi v0.2.6
Compiling libc v0.2.10
Linking against the system version of openssl? Are you sure it's the right version of openssl not to break your binding?
The C ecosystem is fundamentally problematic to get repeatable cross platform builds with; and trying to get repeatable cross platform builds with rust when it touches any c / c++ library is quite a pain.
Piston is a good example of how this is done... but people still struggle to compile it; the 'uses the compiler to explicitly build the dependency as a static library and links it' (eg. git binding afaik) is a much better solution, but it massively increases build times.
It's a rough edge point; and I still feel like there's been no real progress towards solving the issues.
Everyone just does something completely different in their native build (build.rs) step, and then your 'repeatable build' is largely, hit 'cargo build' and 'hope nothing goes wrong and all dependencies are installed'.
There are other more important priorities for rust at the moment (like recovering from panics...), but this is certainly an area I'd love to see improvement.
You'll note from the file paths in the blog post ("file:///Users/ykatz/...") that the builds are actually being run on a OSX machine: winapi is careful to not break builds when it is irrelevant.
It is possible and even necessary to write code that depends on operating system details (e.g. to implement the high-level cross-platform code), and indeed that code may not compile on different platforms. The point is that rustc and cargo together provide the tools to make it possible and easy to write code that builds and runs everywhere everywhere, they don't/can't/shouldn't guarantee it.
The only time it doesn't Just Work is if one of those crates re-exports a type from the sub-crate in a public manner, and then you try to call something from the other crate with a value of that type. You'll get a compile-time error about mismatched types.
If you look at the build step from github it looks like it's relying on mach also:
https://github.com/servo/servo#building
Shouldn't it instead be using cargo purely?
The typical `./mach build` command will download known-good versions of Rust Nightly and Cargo (unless that’s already done), call `cargo build` with some flags, then show a system notification when it’s done. That’s it.
Cargo isn't a general build system, it builds Rust code. I use Make to drive Cargo for my OS project, because I use nasm to build the asm and Cargo to build the Rust, for example.
The trend to put meaningless but catchy adjectives, like "safe" and "predictable", as if everything else is opposite, is misleading.
Designers of classic languages were not some arrogant punks (they were brilliant, like David Moon and other guys of his generation), and the most of findamental problems were well understood in times of Standard ML or Common Lisp.
So, in comparison to C or C++ it might be "predictable" and even more "safe", but in comparison with well-researched classic languages, these mere redundant adjectives.
(I have no idea on CL's dependency management situation, maybe it is even better than Rust's, in which case it would be amazing.)
"there is no guarantee that a build will be the same from day to day" is simply not true.
For them different location (to import from) for an incompatible version is good-enough.