NixCon Live
2020.nixcon.org
2020.nixcon.org
I want to install binaries with cargo instead of doing the same build again and again. Similar to https://cachix.org
I don't honestly know why that isn't the case yet, perhaps storing binaries takes too much storage that Mozilla doesn't have money for. Or perhaps it can't do guarantees like the ones Nix gives to allow binaries being shared of builds.
This is the main reason that makes interfacing with Rust as a plain user a pain.
Is it expensive to maintain all the binaries for all of Hackage?
In my naive view, there should be a way to at least cache the latest successful build from a package version. Then when users `cargo install ..`, it will check if a build was already made and retrieve it.
Tools like guix and nix aim to describe packages in such a way that for the same input (package dependencies) you get the same output.
If the output is a function of the input, and you know a cached build of a package was built with the same input, then using the cached build of a package would be the same as if you'd built it yourself.
It does depend on whether the packages are written in a "pure" way. (Or "impure", such that building with the same inputs might result in different outputs). https://github.com/NixOS/nix/issues/2270
The problem with caching just on package name and version is that there are many different possible flavours. For example, multiple different versions of the sourcecode may have the same given version number. Things such as target architecture, optimisation level, and optional features confuse things further. Nix therefore uses a hash of all the inputs as the cache key.
There are 15,000 packages on Hackage, but 48,000 on crates.io. According to http://www.modulecounts.com/ hackage is getting about four new packages a day right now, and crates.io is getting 54. This means that expense is more of an issue.
I don't know about Haskell, but in Rust, many projects tweak various compiler flags, for performance, or whatever. You'd need to get the exact flags the same too. It's possible there's a long tail effect here.
https://doc.rust-lang.org/nightly/rustc/platform-support.htm... is way longer than https://gitlab.haskell.org/ghc/ghc/-/wikis/platforms (I am not 100% sure the latter is canonical). This isn't a hard stopper, and there's probably a long tail effect here, but it's still a factor in you getting a precompiled binary or not.
Rust relies more heavily on inlining than Haskell does, and relies heavily on monomorphization This means more work to do during compilation; it also means that even for a "precompiled binary", you still end up needing to do a lot of work on the final compilation.
There are existing tools like sccache that you can use to implement a binary cache, so people that really want this feature also have the ability to get it, and some do. It's just not a globally available thing.
If you look at Rust's compile times, generally, even once your dependencies are compiled, it's still slower than we'd like. You only compile dependencies once, and so, while binary libraries may help your initial build, they don't really affect later builds at all, and that's most builds. So more effort is going into improving that, rather than focusing on pre-compiled dependencies.
That's off the top of my head...
You can actually do this today. The buildRustCrate function in nixpkgs creates a derivation for a single Rust crate. You can combine this with crate2nix [1] to make derivations for a crate and all its dependencies in a Cargo.lock file. Every compiled crate will be a separate output in the Nix store and you can upload them to a binary cache such as Cachix.
This is what I have been using for CI in many of my Rust projects in combination with Cachix to get very fast builds.
What is missing is cargo support for native libraries.
(import <nixpkgs> {}).your-favorite-projectIt's one of those tools that are technically miles ahead of their competitors but struggle to gain traction because of beginners facing little cuts and bruises in using them.
That's why I'm interested in the Nix Modules proposal to be presented in the conference that aims to bring better usability to the language with types and docs.
Also the updated website is a lot easier to navigate now and explains the use cases of nix quite well
It's been the steepest learning curve, but yeilded the biggest dividends so far.
The other day I needed to install some software from a specific 3 year old branch. Copying the default.nix for the up to date version and changing a few of the options got it up and running with minimal fuss.
There are certainly some rough edges around usability, but the fundamental value prop is immense.
Also just using nix as a secondary package manager on mac or some other Linux distro is great, I'd expect shell.nix to start showing up in an increasing amount of OSS projects. It showed up in hashicorp's latest project
A more expansive list of my issues with nix (with some curse words :p): https://gist.github.com/est31/005f17628d55fda27b401d3c994233...
> No concept of source packages. There is only a binary cache (hydra) and the derivations in nixpkgs.
NixOS _Only_ has a concept of source packages. The cache is a (sound) optimisation om top of it. just disable the binary cache and you can build all packages from source.
The package definition describes how to fetch the source code from a source (like a Git repo or a hosted archive) and build it. The built result only contains what is necessary at runtime.
A sizeable amount of packages don't even fetch source code but a prebuilt binary which is then fixed up to work with Nix.
There is a source cache, but it is optional.
As an example, check out ripgrep [1]. It uses `fetchFromGithub` to retrieve the code.
[1] https://github.com/NixOS/nixpkgs/blob/3bb54189b0c8132752fff3...
I'm currently doing some work for ML and data science companies where full reproducibility and introspection is very much desired.
So you need to run your own source cache to provide that guarantee, because you can't count on cache.nixos.org still providing the source code from a package built 4 years ago.
But that's why I love the IPFS cache efforts. [1] Running your own node to pin all required sources should then be relatively easy.
It also contains source code.
Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download the sources from the various places they are hosted.
As for the hash, it's good that the source code is hashed, but my main concern was that it was downloading from external sources in the first place. This is bad for privacy, as those hosts know I'm downloading from them, as well as for reliability, because the hosts might not have as good uptime as a debian package mirror.
Fixed-output derivations are used for sources (they are content-addressed in the store), so the latter.
>Also, they are only available when hydra builds the package anyways, right? So if some package is not built by hydra (like how it used to be for the texlive packages), it'll still download the sources from the various places they are hosted.
Yes.
>As for the hash, it's good that the source code is hashed, but my main concern was that it was downloading from external sources in the first place. This is bad for privacy, as those hosts know I'm downloading from them, as well as for reliability, because the hosts might not have as good uptime as a debian package mirror.
That's a true and valid concern, but note that it's the same situation as with Debian: If the package is built upstream by Debian/the NixOS Hydra instance, then you have reliable, private access to its source code so you can rebuild it. If it's not built/packaged upstream, then you need to get the source from somewhere else.
The discrepancy is just that there's packages in Nixpkgs which are not built upstream, and which get built only locally on your machine or your own Hydra instance. There are not many of these, but yeah, it would be nice to fully get rid of them.
Or, an interesting option would be to build the source for more packages on Hydra, without actually building the binary for the package. That wouldn't be too hard, if someone adds an expression for doing it.
Good point!
> an interesting option would be to build the source for more packages on Hydra, without actually building the binary for the package. That wouldn't be too hard, if someone adds an expression for doing it.
Yes, that would be awesome!
It starts to become a bit of a grey area in some cases though. For instance - java packages. Is a .jar a binary? Probably. But so many java applications rely on pulling loads of .jars down from maven. Are we going to sit down and figure out how to build all those jars from source? It's not uncommon for there to be literally hundreds.
I agree that it would be nice to tag (with meta) FOSS packages that aren't built from source, though. Every instance of that is a bug, IMO...
Unfortunately this isn’t the case with some languages that have their own package managers, the prime example being Java as the parent commenter mentioned. It’s near impossible to build Java applications without fetching tons of binary jars from maven that Debian just gives up on providing their own package in many cases[1]. While Nix does build Java applications from source, the dependencies are fetched from maven in binary form.
I'll be speaking how about how to get Nix to become mainstream.
Anyways, the widget at https://2020.nixcon.org/#program localizes to your timezone (but also shows weird dates sometimes).
Times from the cfp subdomain are all in UTC.
YouTube link at https://www.youtube.com/watch?v=7sQa04olUA0
> nix-shell --packages vlc --run "vlc https://dash.nixcon.net/dash/master.m3u8"
I thought this was pretty cool, but unfortunately, this doesn't work on macOS...
nix-shell --packages mpv --run "mpv https://dash.nixcon.net/dash/master.m3u8"I was just sad that I couldn't use the cool Nix magic to view the stream.
A functional language, a package manager and an operating system.
I switched all my desktops and laptops to NixOS at the beginning of the year, and I'm starting to use introduce it at multiple companies for data science / ML work environments with full reproducibility and Kubernetes bare metal deployments.
Main benefits:
* pure build system
* extremely easy way to set up and switch between custom environments with their own sets of dependencies (nix-env, nix-shell)
* nix (the package manager) is cross-platform and works on MacOS and other Linux distros (although that's not always smooth sailing...)
* Ephemeral environments that are quickly set up and torn down, like the `nix-shell --packages vlc --run "vlc https://dash.nixcon.net/dash/master.m3u8"`
* (mostly) fully reproducible OS and user environments
* patching and/or locally building dependencies without ending up with a messy system
I can now finally get a (almost) 100% reproducible user environment including hardware configuration (kernel modules, disks, ...), installed packages, configurations, ...
home-manager [1] even enables declarative configuration of things like VS Code/Vim/Firefox plugins, and provides statically typed configuration options for many applications. The package repo is also very comprehensive.
For manually managed servers that aren't just bootstrapped from VM/cloud images the benefits are also wonderful.
It's definitely not all smooth sailing though:
* There is documentation, but it's spread across a lot of pages, disorganized and sometimes quite outdated or incomplete. I often have to read the source code - which is also mostly undocumented.
* Running unpackaged software: you can't just download binary releases and run them, unless they are fully statically linked (like most Go binaries). Instead you either have to whip up a package definition, or manually try to get the binary working (eg with patchelf)
* The language is somewhat quirky. As often the case with such niche languages, documentation and tooling are very subpar and debugging is not exactly a joy. I wish Nix was built on a more common language with good tooling around it. I bet it would be quite a bit more popular, due to the lower barrier of entry. You get used to it though if you are willing to put in some effort.
* Package ecosystem: the package repo is really comprehensive, but maintainership for some less popular ones is not up to Arch + AUR levels. Also the auditing for a lot of the packages is very limited. For any security critical environments or server deployments I would definitely limit myself to a hand full of core packages and/or be prepared for manual auditing. I wish the repo was split up more like Arch into core and community, with different guarantees for auditing and maintenance. (the Flakes effort will hopefully help here quite a bit)
All in all Nix is great though, and it's impressive how stable and featureful the ecosystem has become, despite being somewhat of a niche "product".