Cargo, Rust's Package Manager
crates.io
crates.io
For example, if I have three packages:
- uno depends on json 1.3.6
- dos depends on json 1.4.12
- tres depends on json 2.1.0
Hopefully rust builds a culture that respects semantic versioning better than the Ruby & Node cultures do. That has to start at the top. There were several Rails 2.3.X releases with minor ABI incompatibilities. Truly respecting semver would have required these patch level updates to get a new major number.
We hypothesize that this difference works better for AOT compiled languages. And since it's pre-1.0, it's all good. This is the 'we'll watch this closely and adjust' from the email: it might not be okay.
When you declare an x.y dependency, tools that use SemVer assumes x.(>y). So, a 1.4 dependency says "Anything greater than 1.4.anything, but less than 2.anything.anything".
This is assuming the ~> operator, if it was =, it would ONLY be 1.14.2, which is even more restrictive.
The project _should_ work, which is why Cargo is using this modified version of ~>.
Rails follows a shifted semver version, as documented here: https://github.com/rails/rails/blob/master/guides/source/mai...
Bumps from 4.x to 4.y might contain smaller breaking changes.
For a large project as Rails, I find that reasonable, otherwise, we'd be at Rails 20 by now, which also doesn't quite give a good sense of how the project evolved.
I would much prefer Rails 20 than the current situation. If you want to make a major marketing level move, introduce a codename or something. Separate marketing from semantics.
Ruby as an ecosystem doesn't really care for semver. Some projects follow it anyways, which I can respect, but they aren't the norm.
The Ruby system cares for semver in general and it is propagated there a lot, but I agree: it certainly isn't uniform.
https://www.ruby-lang.org/en/news/2013/12/21/ruby-version-po...
Side note: it already does this by defaut: http://doc.rust-lang.org/search-index.js
I'm not sure if that's enough information to do this analysis, but the basics are there! I've opened an issue: https://github.com/rust-lang/cargo/issues/47
Because, of course, Mozilla is paying you to sit around on your hands.
Now sure why you think the culture there doesn't respect semver.
$ cargo cult
would build a new project/module from a template.
https://mail.mozilla.org/pipermail/rust-dev/2014-June/010569...
The best solution (as taken by npm, virtualenv and others) is to install libraries locally to the project that is building them.
That way, package management becomes the sole concern of the build system.
"Accumulation" is a good thing, it means each project has the exact version of a package that it was tested with, not some later version that a sysadmin decides it "probably should work with".
- that packages follow semver
- that the OS packagers are in a better position to test package combinations.
If the author releases a new version of libfoo, and A, B and C in an OS repo depend on libfoo, then the OS packagers do not release a new version of libfoo until the tests for A, B & C pass.
These are two good assumptions, and the language package world would be in much better shape if they followed those assumptions too.
Either you rely on system librarys OR every binary / library pulls in its own copy of its dependencies (npm model).
The latter lets you have multiple versions of the same library for different dependencies, without confusing your system package manager.
The former might mean 'less build time', but it's pretty much whats wrong with the C/C++ ecosystem; you can't track the system libraries from the package tool, and you get out of sync.
Pillow and freetype specifically pop to mind as an irritatingly broken recent example; when freetype upgrades, perfectly working python projects suddenly stop working because the base system freetype library has been upgraded; and the pinned python libraries that depended on the previous freetype version no longer work, because they rely on the system package manager not to do stupid things.
It would be nice if you could point cargo as a 'local binary cache' to speed things up, and make them work even if the central repository goes down; and that could be package manager friendly, I imagine.
Flipping your question around a bit: will package manager creators and maintainers ever develop better solutions to the use case of development rather than system administration so that we don't have to keep creating these language-specific tools?
You wouldn't even require upstream NixOS packages, just place built cargo packages in the nix store, using it like a cache. Then upstream NixOS channels could start accumulating cargo packages, making cargo dependency "builds" faster.
Hopefully the cargo team can come up with a solution that works a little better here, but I wouldn't hold my breath.
Understand that enterprise OSes are not built on developer Macbooks. Enterprise distros have reproducible builds on automated systems with chroots that contain only what the package needs, no network access and sometimes the build happens inside a virtual machine. Its is almost ridiculous how after maven forgot the topic of being "offline" almost every tool released afterward has done the same mistake.
Understand that Linux distributions sell support and that means being able to rebuild a patched version of a failed package. So whatever dependencies are pinned in Cargo.toml or Gemfile is irrelevant. The OS maker will override them as a result of testing, patching or simply to share one dependency across multiple packages. Distros can't afford having one package per git revision used on a popular distro and then afford to fix the same security issue in all of them.
So having "cargo build" be able to override dependencies and instead look on the installed libraries with a command line switch or env variable helps the packager not having to rewrite the Cargo.toml in the package recipe.
Maven was probably the first tool that made packaging almost impossible and completely ignored the use-case of patching a component of the chain and be able to rebuild it completely.
Semantic versioning is great news, because it allows you to replace the dependencies for a common semantic version (not exactly the same).
For integrating with the C libraries not much needs to be done. If you support pkg-config then you cover 85% of the cases.
While technically the cache directory is a place where files can accumulate outside the view of a well documented and designed administrative tools, this is common problem shared with many tools including your favourite browser.
The quiet nature of the development process of Cargo was actually a response to the previous package management failures. The idea was not to publicise heavily before it was ready for dog fooding. This seems to have paid off.
The next features we're planning on working on are: - Supporting refs other than `master` from git packages
You can keep it on your local filesystem and reference it w/ `file:///path/to/repo.git` -- you can also use SSH and HTTPS URIs from any other repository host, not just github!
So you could just clone the repo to some `vendor` directory and move `master` to whatever version you want!
I'm pretty excited to see Teepee and Rust come together so I can really give it a spin doing what I'm currently doing daily for a job.
I hope it will be more stable and work better than the Haskell package manager, Cabal. I literally never got that to work on any machine. It would typically destroy itself while attempting to update itself...
It doesn't ship with Rust and the docs on GitHub and crates.io are not very enlightening.
1) Install latest version of Rust found here: http://www.rust-lang.org
2) git clone --recursive git@github.com:rust-lang/cargo.git
3) make
4) make install (could be that sudo is needed for you) DESTDIR=$HOME make installaroch:~/staging/|⇒ brew info rust
rust: stable 0.10 (bottled), HEAD
/usr/local/Cellar/rust/0.10 (74 files, 174M) *
Poured from bottle
aroch:~/staging/|⇒ brew info cargo
Error: No available formula for cargo
brew tap caskroom/cask
brew install brew-cask
brew tap caskroom/versions
brew cask install rust-nightlyThe new tutorial will be based around 'real' Rust development, and so will assume Cargo.
That said, http://crates.io/ should have install instructions on the site. I'll open a ticket and get on that.
1. Wycats (Yehuda Katz) is on Rust apparently :)
2. `.toml` -- some crossbreed YAML/INI file format that I like
That said it's not that big of a deal. At least it's not an in-house markup like Haskell's cabal...
About 14 months ago, it caused some of the most serious vulnerabilities in the Ruby on Rails world ever: http://tenderlovemaking.com/2013/02/06/yaml-f7u12.html
> why not just use JSON?
JSON is not really human-editable. Those quotes and commas, ugh! Also, JSON lacks comments.
The vulnerabilities in YAML (which is a superset of JSON, by the way) point at why YAML and JSON both aren't appropriate for configuration: they are _serialization_ formats. Configuration isn't what they're built for.
And you're right, it's really just not a huge deal in any way. Especially once we have `cargo project` to autogenerate the basics.
Live by eval, die by eval. But more seriously, nobody is forcing a Rust YAML library to support arbitrary structure deserialization (or maybe as an optional switch). I don't think you'd want such a switch on in a build system configuration file.
Infact, why not just use npm's package.json?
You cannot write comments in JSON, for instance.
{
'package': {
'name': 'hello-world',
'version': '0.1.0',
'authors': [ 'wycats@example.com' ]
},
'bin': {
'name': 'hello-world',
'comment': 'the name of the executable to generate'
}
}
And you have the advantage, that other tools can use the complete file including the comment. In TOML you need an extra parser to grab the comment.
Your solution works fine for docstrings, but comments and docstrings are not the same thing (although many languages that don't support docstrings in the syntax hack them together using comments, admittedly).
But beyond that, what's the argument for switching to json? Is there some kind of intercompatibility with npm/Node.js to be gained?
Also, having to quote everything makes writing JSON by hand a pain. Why would you want to use it over a nicer format?
What I wouldn't do for a cut-down YAML standard with most of the serialization crap cut out.
From: https://github.com/toml-lang/toml
"Latest tagged version: v0.2.0.
Be warned, this spec is still changing a lot. Until it's marked as 1.0, you should assume that it is unstable and act accordingly."
If the plan is to jettison TOML, then it's simply just an odd choice to use for a first cut. And from a purely perception manner, seemingly reaffirms concerns some have had about the bundler team building cargo (right or wrong).
I can see an argument for INI files, but TOML is basically INI with some extensions. And YAML (which has no Rust parser, and nobody who wnats to write one) and JSON (which does have one in the standard library) are very poor configuration formats.
Which one would you have preferred? It seems like a reasonable choice to me.
> And from a purely perception manner, seemingly reaffirms concerns some have had about the bundler team building cargo (right or wrong).
You could just spell out the personnel issues rather than being all FUDdy about it. (I'm friends with Yehuda, but also was room-mates with the current maintainer of Bundler (until very recently). Bundler isn't perfect, but lots of that had to do with people assuming a pre-1.0 project is stable, and upstream bullshit with RubyGems.)
As for the FUDdiness, there was another extensive comment thread here on HN when Cargo was announced. I didn't feel it necessary to re-enumerate all those concerns. And to the best of my recollection, the problems listed were all technical in nature, not personal. For my part, my list was:
https://news.ycombinator.com/item?id=7421629
But, I'm hardly alone in being concerned. This is something that will drastically influence the ecosystem. So, it should be met with some level of scrutiny. If it comes out passing muster, great. But that's not going to happen by being glib about things.
Absolutely. Let me make an issue about that: https://github.com/rust-lang/cargo/issues/46 If he doesn't respond, I will email him.
> As for the FUDdiness...
Thanks! Since the points are enumerated, I can refute them:
> * You can't override a dependency:
You can, in modern Bundlers. But the transitive dependency issue is inherent to Ruby, not Bundler. Cargo will work like npm in this regard, not like bundler.
> Dependencies can disappear on you:
The central repository isn't built yet, but this is very valid. The reason that `gem yank` is still available to everyone is that you'd be surprised how often people push proprietary stuff up to RubyGems. It was removed for a while and caused a significant burden on the RubyGems team.
Regardless, yes, this is a serious consideration.
> It has weird rules:
Totally fair. Let's learn from that and make Cargo not have weird rules. :)
> * It promotes bad practices:
Yup, that's a Rails thing. Rust should be way better, as it doesn't have this kind of issue.
> * It's slow:
This actually has just as much to do with RubyGems as it does Bundler. See all of the presentations Andre has been doing recently, and the bundler-api project.
> * It was designed for MRI:
Quite fair. We only have one Rust implementation so far, but `cargo exec` won't be necessary.
------------------------
> So, it should be met with some level of scrutiny.
Absolutely! I don't mean to say "just take this without complaints." But without specific, actionable complaints, it won't get fixed. Please make as many specific, actionable complaints as possible, especially now, pre 1.0.
I feel compelled to mention it because this seems to be a clear case where an aesthetic DSL was chosen over performance and it can't really be fixed without a backwards-incompatible change. Rust and Ruby are two different beasts and I get that. I just hope that performance is a core design consideration for Cargo (I have no idea if it is or if it's just a nice-to-have).
Airing dirty laundry is hardly ever pleasant. I'm certainly not immune to my own set of WTFs. I legitimately just want to make sure Cargo comes out awesome. And I much appreciate Mozilla's commitment to having a standard dependency resolution tool when Rust ships.
> Using binstubs rather than "bundle exec" can be a good deal faster.
Absolutely, which is why we switched to them with Rails. It is a tough problem, though, and given Ruby's constraints. In my testing, it's the startup time that's the issue, not Thor, which is because `bundle exec` has to re-start the interpreter multiple times, and binstubs don't. Anyway.
> I just hope that performance is a core design consideration for Cargo
Rust people already feel the pain of very long rustc compiles, so while I'm not sure that it's an overriding concern, given the Rust world's concern with performance in general, I expect it to be way better. Ruby has always kinda thrown performance to the wolves.
> I legitimately just want to make sure Cargo comes out awesome.
We all do. And I'll admit to being a bit sensitive to 'lol bundler,' which I feel is often said without fully understanding the problem space, which is admittedly large. Not that you are doing that, but I have seen similar comments elsewhere. Once you explain the details, it's pretty clear why Bundler does what it does.
Anyway, yes: let's make Cargo 1.0 and Rust 1.0 super awesome! I'm really excited that we're taking this step forward. It's a huge day for Rust.
I hope we can keep the Rust community mega positive and no-bullshit. Please let me know if I'm ever not being so.
For what it's worth, I've just been brought on as a maintainer for TOML proper. Tom is still the guy though. Putting aside my involvement with Rust (<3), I also maintain the TOML library for Go, and I want to see an expedient path to 1.0 with minimal breakage. (There are lots of folks in the Go community already using TOML for things, including Mozilla's Heka project and CoreOS's `etcd` project.)
cargo read-manifest --manifest-path .(I admit I have a bias against toml, but still...)
This complexity means there is already a parser for TOML, and not one for YAML. That's IMO main reason they went with TOML.
As you say, this is a testament to TOML's simplicity.
EDIT: Furthermore, TOML is going to have a 1.0 soon: https://github.com/rust-lang/cargo/issues/46