Release engineering is exhausting so here's cargo-dist
blog.axo.dev
blog.axo.dev
One of the most honest paragraphs ever written.
Seriously though, great tool and great write up. I hope something like this lands as an official cargo feature. Coming from mostly Python land at work with crazy dependencies for TF and PyTorch GPU support (On Windows sometimes!) makes me super jealous.
If it was in the article and I missed it I apologize.
A key feature of cargo-dist is that
cargo dist init --ci=github
should simply set everything up for an arbitrary* Rust workspace and you just check in the results.
* Not sufficiently tested for all the wild workspaces you can build, but a "boring" one with a bunch of libraries supporting one binary should work -- that's what cargo-dist's own workspace is, and it's self-hosting without any configuration. Any time I want to update the bootstrap dist I just install that version of cargo-dist on my machine and run `cargo dist generate-ci github --installer=...` to completely overwrite the ci with the latest impl.
However this isn't fully supported by the actual github CI integration yet[1], as I haven't implemented proper support for detecting that you're only trying to publish a new version of only one of the applications (or none of them!), and it doesn't properly merge the release notes if you're trying to publish multiple ones at once.
I never build workspaces like that so I'm waiting for someone who does to chime in with the behaviour they want (since there's lots of defensible choices and I only have so many waking hours to implement stuff).
[0]: https://github.com/axodotdev/cargo-dist/#concepts [1]: https://github.com/axodotdev/cargo-dist/issues/69
It creates a ~100 line release.yml file
1. create GitHub release
2. upload artifacts
3. upload manifest
4. publish GitHub release
I like it, it's opinionated, but I don't know how much it will catch on. If somebody needs to maintain a 100 line GitHub Actions release YAML file, in my experience you typically want to understand everything in it in case you need to adjust it for future needs/as they grow.
It's well done though. Curious to see how much adoption it picks up.
Yes! Why is this accepted??
Gitlab has a way of running CI locally (for Docker based builds anyway; who knows about Windows or Mac) but a) it doesn't support the same features at the "proper" one (even basic ones like the `default` key) and b) they deprecated it!
Ok in fairness they've stated in a random comment that they won't remove it before providing an alternative.... But still, how is this not a core feature of all CI systems?
Workflow is something like:
- Guess together a random CI or CD workflow, this is just to kick off the process. You can also start with a empty config.
- Something fails, get SSH host+port to connect to
- Enter environment and manually do everything you want to be able to automatically do
- Execute `history` and copy output, trim it into something nicer and put in your config
Also artifacts don't seem to be supported.
Is there a way so I can still do this? I have terabytes of SSD just waiting to be used.
That would allow for a tooling ecosystem of static checkers, offline runners, IDE integrations, etc etc, and would also cut down on the learning barrier every time you switch to a new company that uses a different vendor
Vendor lock-in, presumably.
My jaw hit the floor here.
The check of interest is for a Mark Of The Web[0] flag that Windows includes in file system metadata. The builtin unzipping utility just faithfully propagates this flag to the files it unpacks. Other utilities like 7zip are unlikely to do this propagation (effectively clearing it).
But yeah either way it has nothing to do with code signing!
https://textslashplain.com/2016/04/04/downloads-and-the-mark...
https://textslashplain.com/2022/12/02/mark-of-the-web-additi...
You can remove this flag with the xattr command:
xattr -d com.apple.quarantine the_quarantined_binary
So for example, a single file can actually contain two different "files"(File data).
So, foo.exe, actually will effectively open the file foo.exe:DEFAULT. You could also add a piece of malware to the foo file in place of a datastream. So foo.exe is legit, but if you open foo.exe:MALWARE , it will open up the malware datastream.
So tldr, how Windows does this, it when you get a file from a third party source(Internet, USB Drive, etc), it adds a new datastream in the form of a textfile. And the textfile contains info about the source. Namely, a number for location it came from(3? for web), and then some more info.
Most surprising to me on Mac was that the "flag" (I'm not sure that's the right term here either) was preserved on files extracted from a tarball downloaded from the internet. Although I think this also required extracting it via Finder (GUI) and did not apply when using the tar command - I can't remember exactly.
But the fact remains that end users want to download a binary, and the common approach to this is to simply publish them to GitHub actions. Surely Cargo could offer a better option than this, while retaining the ability to build all packages from source (maybe you can only publish binaries to Cargo _in addition_ to the source files... or maybe Cargo.toml could include a link to GitHub releases, and Cargo could include a command for downloading directly from there.)
In the meantime, I've been considering publishing Rust binaries to npm (in addition to GitHub releases). I got the idea from esbuild, which is written in Go but distributed via npm. What do people think of this approach? Here's a recent blog post [0] describing the setup.
I wrote a bit about that pattern here: https://simonwillison.net/2022/May/23/bundling-binary-tools-...
i have personally packaged and published many rust devtools on npm (cloudflare's wrangler, apollo's rover, wasm-pack) but that was largely because they were targeted at a javascript developer audience.
as a former npm registry engineer i'm curious what you find to be the particular value of publishing to npm? installing node is actually very unpleasant and then getting global installs to work is also... very unpleasant. i think it works well for people already in that ecosystem but i think we can build something better for a more agnostic audience that can give a similar out-of-box exp without requiring a centralized registry. would love to learn more about your perspective!
I have to admit I find this to be an amusing sentence. :D Poor npm (maybe rightfully) does not have the best reputation.
I suppose it appeals to me mostly for reasons of personal bias, and the thought that most people downloading binaries from GitHub probably have node installed. Although I'm probably wrong about that. I also like that optionalDependencies and postinstall steps can use JS code to effectively act as an install script, which at least feels somewhat cleaner than curl install.sh | bash.
You make a good point about this being more appropriate for binaries that have some relation to JS (like esbuild). I think this is probably a compelling enough argument not to distribute binaries to npm if they're not related to JS - or at least, if there is no obvious overlap between the expected userbase of the binary and JS users.
Perhaps the solution friendliest to end users (but least friendly to maintainers) would be to distribute the binary to as many package managers as possible. I mean, why not publish it to npm, and PyPi, and maven, and rubygems...? If your Rust tool has bindings in these languages, then of course it makes sense. Otherwise, it sounds ridiculous... but is it really so different from publishing to multiple OS-level package managers like apt, brew, macports, etc? Almost certainly your users, if they're downloading a tool from GitHub, will have one of these package managers installed. (But then again they'll probably also have one of apt/brew/nix, so maybe no need to use the language specific manager.)
Actually, if this is something you're working on, maybe that is a problem you can solve - you could handle distributing the binary to multiple package registries, and I could just push it to GitHub releases and add an extra line to my workflow.yml. You wouldn't even need to store the binaries - you could just fetch them from the GitHub releases page. (Although now we've strayed quite far from my original motivation of a more secure distribution channel! And perhaps this leads to reinventing a few wheels...)
There are millions, perhaps even billions of people who aren't js developers.
I'm one of them :)
what is an example of a language package manager that does this well, in your view? I think the idea is something similar to how apt-get/deb/rpm work at the os-level, but I'm interested in the details of what the improved version of this is in your mind.
Github's CDN is a fine way to host binaries in a highly available manner which is also not easy to tamper with.
GitHub's project page gives instant access to source code, and to a rich README.
I don't see how Github in this regard is any worse than npm or pypi.
What I would appreciate is a way to sign binaries the same way commits are signed, attesting that it was built from a particular commit, and a particular state of dependencies, by a compiler isolated from the internet. GitHub's CI runner might sign it with some private key, while a public key to check signatures would be available from github.com.
Of course that would require some cooperation from code authors, but for important code that should be manageable.
$ npm install foo # or
$ npx foo
Find repository
Click latest release, downloads to ~/Downloads
$ tar -xvzf ~/Downloads/some.tar.gz
$ cp foo/bin/foo /usr/local/bin cargo install foo
Just puzzled; I think binary distributions make any supply chain issues basically impossible to solve.
Vendoring them into the tool chain instead of distributing source code you can compile yourself seems the opposite of solving the problem you’ve posed.
I realize you’ve probably thought about this a lot since I know you as the guy who wrote “mazzle” and posted about it here a few months ago. I wish more CI systems worked closer to the thing you designed.
I would like things to run locally by default and then deployed to the cloud where they run.
Should be easier to debug problems if I can get the code to my machine and investigate issues with tools that my computer has such as "strace", "perf" and debug logging that I liberally apply to the build script.
In production we would have log aggregation and log search (such as ELK stack) and it is a good habit to get into the perspective of debugging production via tooling.
But CICD feels before that tooling in the pipeline. You could wire up your CICD to log to ELK but I would prefer local deployable software.
I think my focus on automating things means I want to be capable of seeing how the thing works without relying on a deployed black box in the cloud and using assumptions of how it works rather than direct investigation.
One of my journal entries is almost a lamentation of all the things that need to be done to release and use software.
This is that entry:
https://github.com/samsquire/ideas4#5-permanent-softwareplat...
I wonder if software could be deployed more like a URL that has all the information to configure a virtual machine. Docker over URL or something.
In general, please don't unnecessarily gender (verb) people whose gender (noun) you don't know. Using "they" has been fine since the 13th century.
Rust does a good job of generating Windows binaries cross-platform. But the tools for generating Windows installers are not yet cross-plaform. Does this project improve that situation?
It looks like the closest thing you can get to an installer is an 'executable zip'.
Not everyone uses nix, but I'd rather push its adoption than trying to build these kind of solutions for all possible language ecosystems when a general solution already exists and works great.
I really wish people wouldn't get so zealous all the time about their favorite technology.
Nix doesn't replace cargo. It would use cargo under the hood. If Cargo supports Windows builds, those builds will work with nix. However, nix (probably) doesn't give you anything additional in this case.
Nix flakes can still be useful on the tool user side, if your application build requires Unix anyway, but on the tool producer side it's a bit less so.
It'll fetch the binary release from the repo so you don't have to compile it yourself.
Tiny feedback:
- Can you consider changing "git add ." to be explicit e.g. "git add Cargo.toml .github/workflows/release.yml"?
- How about modifying Cargo.toml to add cargo-dist as a dev-dependency? I know it's not strictly necessary; it's just very helpful for typical collaboration.
That's an interesting thought, I'm not sure I've ever seen someone employ that as a pattern. Actually no wait, I thought cargo bin-deps specifically gave the developer no way to manually invoke it (i.e. there's no equivalent functionality to npm's npx)? Without that, what use would the dependency be?
I use the dev-dependencies section often, and the build-dependencies rarely. And anyone here, please correct my understanding if there's a better way to do what I'm describing.
For me, these sections are an easy way to be explicit with collaborators that the project needs the deps installed in order to work on the project, and does not want/need the deps to be compiled into the release.
My #1 use case is to have a collaborator start with a git clone, then run "cargo build", and have cargo download and cache everything that's needed to work on the project and release it. My #2 use case is to deal with semver, so a project can be explicit that it needs a greater cargo-dist version in the future.
To your point about cargo bin deps akin to npx, yes, your understanding matches mine i.e. not available yet. I do advocate for that feature to be added because it's helpful for local environments. Cargo does offer "default-run" which shows there an awareness of a developer preferring specific local exectuables-- maybe Cargo can/will add more like that for deps?
Also, seeing the initial attempt fail because of a missing C dependency is a common problem with a lot of language-specific package managers.