Why did Nix adopt Flakes?
jetpack.io
jetpack.io
It insists on copying the entire repository into the Nix store (which makes all of its content world-readable!). Even if your flake.nix is in a subdirectory of a monorepo, the entire monorepo will be copied into the store every time!
You might have two dozen flakes in your system. You'll typically only have a single channel, if you use them.
https://github.com/jetpack-io/devbox/blob/main/internal/impl...
I tested this branch, and a flake-based shell launched in 3 seconds inside a large monorepo (vs 30 seconds before). I don't think it is possible to copy this repo in 3 seconds.
Remember the debian SSH packaging snafu? The application authors weren't involved. (edit: see below)
Nixpkgs is like the state department, a central unifying hub, great to bootstrap the package ecosystem (ten years old now), but it needs to spread its wings.
It's a package spec, just like debian/* and rpm/*.spec.
>Discussion about building the application are kept with the application, nuances and patches can be discussed with the actual authors.
Every distro package ecosystem has the maintainers discuss patches with upstream.
>Remember the debian SSH packaging snafu? The application authors weren't involved.
Assuming you're referring to the 2006-2008 OpenSSL vulnerability (which affected openssh and other things), yes the openssl devs were involved. The Debian maintainer who added the bad patch asked about it on the openssl mailing list, and only added it after he got ACKs from the devs.
https://lists.debian.org/debian-security-announce/2008/msg00...
> he got ACKs from the devs
You're totally right: https://news.ycombinator.com/item?id=6343782
> Every distro package ecosystem has the maintainers discuss patches with upstream.
I don't think that's entirely fair to say, it's certainly best practice.
No. It's not. The interface that Nixpkgs' callPackage expects to find in default.nix within the Nixpkgs source tree is (kinda) like an RPM spec or Debian control file and friends, but that's not what flakes are.
Flakes is mostly about providing mechanisms for distributing and referring to code written in the Nix language, and in that way facilitates the maintenance of 'out-of-tree' (read: not in Nixpkgs) Nix packages. But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs), and the conventions for describing the build come from there. They're not part of the flakes schema.
> Every distro package ecosystem has the maintainers discuss patches with upstream.
Again, not what flakes are about. The situation the GP is describing is where a Nix package's 'recipe' lives inside the upstream repository* and is maintained there directly by contributors to/maintainers of the upstream project. This is more like a project including a Dockerfile for the convenience of new contributors, or application developers directly publishing their own Flatpak images.
Yes, aka a Nix package spec.
>But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
Yes, just like DEB / RPM packages.
>A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs)
Yes, what DEB / RPM call meta-packages.
>The situation the GP is describing is where a Nix package's 'recipe' lives inside the upstream repository
Yeah, and just like you convinced upstream to hold your Nix package spec, many upstreams hold Debian / RPM package specs too.
> Yes, aka a Nix package spec.
Right... one of things that flakes can be used to distribute is roughly comparable to an RPM spec. That is not the same thing as being an RPM spec. A BitTorrent magnet link used to distribute an MP3 file is not itself an MP3 file, either.
> > But flakes can also be (and are) used to ship Nixlang libraries, end user configurations, IaC for clusters or fleets of servers, overlays of packages to be composed with an end user's copy of Nixpkgs, etc.
> Yes, just like DEB / RPM packages.
There is a sense in which flakes and RPMs can (sort of) be used to achieve the same things here, but not remotely in a way 'just like' each other. A flake.nix file is not an archive, and an RPM is not a configuration file format for something like Puppet. The latter point is so obvious that I have to assume you're being deliberately obtuse here.
> > A flake doesn't have to contain any packages at all. And when it does contain a package, the package is always defined with reference to some other package collection that it pulls its dependencies from (almost always Nixpkgs)
> Yes, what DEB / RPM call meta-packages.
No. RPM and DEB metapackages are 'abuses' of an archive format for either (a) just distributing install-time hooks and activating triggers or (b) forcing the installation of actual packages through the normal mechanisms of declaring dependencies on them.
Nix has no equivalent to (a) because 'installation' doesn't mean the same thing with Nix and hooks and triggers are owned by Nix profile managers rather than packages.
If Nix has anything equivalent to (b) in the RPM and DEB worlds, it's this¹ which has nothing to do with flakes and is certainly not equivalent to a flake with no packages declared. In fact, any flake used to distribute such a buildEnv-based metapackage would have to declare it as a package.
A flake without any packages declared is not an archive of a Nix package whose data is empty but whose metadata is present.
Idk what to tell you. Clearly you are trying to somehow 'demystify' Nix terms by equating their referents to things that are familiar to you and others, but your desire for succinctness here pushes you to elide basic differences— not just with respect to Nix but with respect to source-based package management in general. Just like with two natural languages, it's simply not the case that every term here has a 1:1 translation.
--
1: https://nixos.wiki/wiki/FAQ#How_can_I_manage_software_with_n...
I think this is a great strength of flakes. Packaging an application can sometimes have nuances that external folks may not be fully aware of and so having the packaging instructions (i.e. flake.nix) defined in the core repository can narrow that knowledge gap.
Flakes, home manager, half the packages I need, etc
It's not for everyone, but when you work with infrastructure and can guarantee that the software the developers build is the exact same version you're running for them it feels good to have that conversation out of the way already.
1. The purist: Is on an actual crusade to promote Nix and stomp out any competition, because, "reproducible."
2. The enthusiast: Likes using Nix to speed up development but recognizes it has a ton of shortcomings, among others being not friendly to beginners, and tends to avoid using it in production critical stuff.
3. The ex-Nix: matches the enthusiasm of the purist in stomping out Nix wherever possible and actively promoting its demise.
I'm personally in the second category. I use Nixos for my main development machine and really like devshells. Beyond that, I try to avoid it as much as possible for production.
I have no experience running it in production, but I imagine if you don’t want to use containers it’d be a pretty good option.
But also, my "personal config" has two dozen imports and overrides nixpkgs on most of those. I have a list of complaints regarding flakes that is only dwarfed by 6+ year old general nix issues, but I could never go back.
you can smell from a mile away that they were engineered to solve widely-experienced problems... as experienced by a single company, with an existing, idiosyncratic set of methodologies. and they just happened to get the blessing because eelco was at that company. unlike nix proper, however, where eelco had the entire internet for feedback, the core design of flakes ossified before the world at large had reason to care about them.
it doesn't help that nix's command-line ux is currently super splintered as a result, and while that will be ironed out in the long run, the thing in the name of which those tools got splintered is rather insulting.
i love nix but god damn this is the stage at which i'd take someone to couples counseling
I would like to see composability of graphs (and other set operations on binary package repos) integrated into more dependency management systems.
FreeBSD is my server of choice and I’d love to say: create a package repo with my config package and it’s dependencies and nothing more and deploy from that knowing a million other dependencies can’t be pulled in (e.g. give me a new jail that pulls from subset).
It’s probably something I should prototype one day.
The system you describe building is exactly what nix does, as well as debian, El, and Arch. Their spec files describe both build and runtime dependices, and installing the package does not install things like Make.
Nix goes a little bit further by only including runtime dependices that it can find it the build output. It does this by scanning the output files.
But I don't understand how you could separate build graph and runtime graph. If I declare something needs foo and bar, that is useless unless I can get built foo and bar. _Something_ has to know how to build the things this hypothetical system is installing.
This also allows build processes to evolve without affecting the public dependency graph. This makes it easier to show dependencies are modeled and exposed correctly and makes build logic private to consumers.
Nixer Domen Kozar called this property 'static metadata' in one of his talks¹ on Python packaging from a Nix perspective years ago.
The thing he was interested in was the ability to evaluate the dependencies of an upstream software package without having to actually 'install' it or evaluate bespoke upstream code to do so. The reason for interest in this within the Nix community is that by default Nix performs builds in a restricted sandbox, and one of its restrictions is that no network access is allowed.
To use upstream build tools (e.g., Maven or Cargo or NPM, etc.) inside the Nix sandbox, then, fetching dependencies and verifying their contents is deferred from the upstream build tool to Nix, which does so in a controlled, deterministic way that just fetches and doesn't have hooks to let those deps run custom code.
In order to make that happen, Nix has to 'know' ahead of time where to fetch those dependencies and what their contents will be, and what you describe wishing for here— a 'dumb dependency graph'— is more or less exactly what Nix wants to consume (although sometimes just a list of pairs of URIs and content hashes will do). For well-behaved upstream package managers, i.e., those which can emit comprehensive static metadata, that's exactly what Nix does: it just translates a Cargo or NPM lock file into its usual conventions for describing source archives, and then that can be used to download those dependencies in the usual safe/restricted way.
For package managers that don't emit adequate 'dumb' package metadata, Nix has to proceed by either emulating and replacing those upstream dependency resolvers (very error-prone) or by a hack implemented as a Maven plugin or similar that inspects dependencies as Maven resolves them and gets the metadata Nix needs.
As for flakes, flake inputs are certainly a kind of dependency that's distinct from package dependencies in the Nix world, and also 'dumber' in the sense that flakes don't have to be 'built' like packages to be consumed by other flakes. And yeah, in the case of flakes that provide packages, you can definitely swap one flake for another and allow that new flake to provide customized build instructions if those are needed for that version of the package it provides. But there are complexities and entanglements that flakes don't/can't eliminate, since downstream dependencies can still have implicit expectations of build outputs, and you can't really know in advance if the package in the new flake will meet all of those. You kinda still have to be able to peek into those and examine them and fix them up in a pinch. Plus you can write a flake in a way that depends on the build process of a package in a flake it consumes, e.g. by using package overrides on something provided in the upstream flake.
--
It feels good to have your understanding of a problem and where different technologies might fit into better addressing it click into place like that. :D
I'm glad to see the first point (flake.nix provides a consistent interface) mentioned. The consistent interface allows for the cli to be much nicer; `nix flake show` can list the outputs.
The title is simply literally the question that I was thinking about a few months ago when working on Devbox, and wrote a version of this post internally to answer the question for myself!
done :)
"What Nix Flakes Solve" or "Benefits of Nix Flakes" would avoid that.
But, I didn't read the title as provocative.
[1]: https://nixos.wiki/wiki/Flakes section "Enable Flakes".
AFAIR, the only irreversible impact is switching from `nix-env --install` to `nix profile install`.
With repositories, you can add a flake.nix, and still use nix-build or nix-shell however you did before. e.g. You can have the flake.nix import the default.nix or shell.nix code.
example:
nix run .#cowsay -- flakes are neat
It's convenient for passing options and args to a subprocess, but it customarily means "don't interpret anything following this as a short or long option, just as a positional argument". For example:
touch -f # error
touch -- -f # creates a file named "-f"
rm * # oh crap we just passed "rm -f" a bunch of files
rm -- * # don't treat that "-f" as an option, just delete the file named "-f"The first -- argument that is not an option-argument should be accepted as a delimiter indicating the end of options. Any following arguments should be treated as operands, even if they begin with the '-' character.
To explain what's happening:
- The example above is running `nix run <flake output attribute> -- <arguments to pass to the flake output's binary>`.
- The `<flake output attribute>` here is `.#cowsay`, which is to be read as: `<flake reference>#<attribute path>`. The # is a separator here.
- The `<flake reference>` being `.` implies its a local flake at the current directory. The `<attribute path>` in this case is the output from the flake i.e. the `cowsay` program.
further fun to be found at: https://nixos.org/manual/nix/stable/command-ref/new-cli/nix....
As a meta-comment, yeah, this is kinda non-trivial for most of us. This is why we build devbox which provides a more familiar UX like `devbox add cowsay && devbox run -- cowsay "flakes are neat"`
So it's run the cowsay attribute from the flake in the current directory, passing flakes are neat as input to cowsay.
But with the myriad of fetch* functions maybe it only applies to some.