About using Nix in my development workflow
ejpcmac.net
ejpcmac.net
The small things are:
- nix-direnv[1] is much more powerful and stable, with or without flakes
- devshell[2] exists and is reasonably mature for those who want to give this to a team without teaching them (most) nix
Still, apart from those, this is still a good intro.
* What they add from vanilla Nix.
* What they take away.
* How they work.
* Why channels are unnecessary with flakes (once they're complete) as TFA implies.
* How much they replace home-manager.
* Why they add even more reproducibility.
* Why else I should care about them.
Other questions that might help me: Are they just adding the equivalent of lockfiles to vanilla Nix? If so, why is that such a hard transition? My initial impression on reading about Nix, and I've read a lot, is that Nix already had lockfiles.
Context: not a Nix user, but I've read the Nix thesis, the NixOS thesis, a lot of the Tweag blog posts about flakes, a lot of the xeiaso.net posts about flakes, and a lot of the other posts listed in the Reddit link of a child of a sibling comment. Despite all of that reading, I don't get them. I have a hard time following regular Nix as it is, and if I start using Nix, I want to start with flakes.
Flakes add a couple of things:
1. They ensure the version of nixpkgs used is 'locked': In pre-flake Nix, working with nixpkgs involved using channels. The channel more/less was a clone of some git repository. But, what revision is checked out may vary from machine to machine, since the channels may be updated at different times.
2. Flakes provide a consistent entry point to Nix code. In pre-flake Nix, there were different conventions, but you'd have to know the contents of a file named "default.nix" to know how to work with it. With "flake.nix", the file follows a specific schema, and has a standard way of working with it.
> What they take away.
There's added restrictions on purity. IIRC, stuff like fetchUrl needs to specify the checksum when using flakes.
> How they work.
In pre-flake Nix, you'd need to have certain channels installed to use some expressions (or to use a tool like Niv to avoid using system channels).
With flakes, pretty much once you've declared your inputs and declared the outputs of your nix code (what its packages, overlays, apps, etc. are), the Nix code used by the flake is otherwise the same.
> Why channels are unnecessary with flakes (once they're complete) as TFA implies.
Channels are an implicit per-user way that inputs are used in Nix code. Flakes need inputs to be declared.
> Why else I should care about them.
The CLI is much nicer. e.g. you can have "nix run github:<owner>/<repo>" and it fetches/builds/runs the repo, if it specifies a default app or default package. - Stuff like this is why I'd compare Nix to "like Docker, without containers".
> Are they just adding the equivalent of lockfiles to vanilla Nix? If so, why is that such a hard transition? My initial impression on reading about Nix, and I've read a lot, is that Nix already had lockfiles.
I think the tool Niv offered similar functionality as the input locking from Nix flakes. But, well, that's a community tool, and still doesn't imply a standard way of invoking a Nix file.
There was a lot of fuss over the big Nix 2.4 release, which had many changes since Nix 2.3. Some of the frustration was that the Nix 2.4 release made breaking changes to non-flake stuff which worked in Nix 2.3.
I used non-flake nix for some time before trying out flakes. I needn't have waited. It's initially a bit of boilerplate to copy-paste. But, I didn't have pain moving stuff to flakes, and flakes are neat.
I think what hindered my understanding was the claim that Nix was purely functional, so it never made sense why flakes were needed. Knowing that Nix is not purely functional, i.e., that channels were still an implicit input, might be the missing piece for me.
So, for example, I could set my NIX_PATH to a private clone of the nix store and have my personal derivations there and builds I did would not be the same as on your machine where you point your NIX_PATH at the canonical upstream.
Flakes require all of these dynamic parts of the build (including the build architecture etc) to be injected as inputs to the system. Similar to how Bazel workspace files work from what I understand.
As part of that standardization of the inputs into a `flake.nix` / `flake.lock` file there was also work done to define a standard interface for a flake as a unit. Before flakes, if you imported another "unit" there was no telling what was being exported by the target (is it a derivation? an overlay?). With flakes there are well-known, enforced patterns on what and how you would expose to consumers.
> How much they replace home-manager.
Not at all really. But home-manager is now provided as a flake and so you no longer need to install it separately, and can instead import their flake to get a fully-working system.
> Context: not a Nix user, but [...]
To be honest, nix is a system you have to actually use to begin to understand and synthesize. As noted, flakes make this hill less steep, but you won't really internalize the value and the system as a whole until you actually start using it some.
Then it is fairly bad as a tool, you don't wan't to spend disproportionate amount of time learning to use the tool when the goal is to just use the tool and forget about it (ie. using Nix for more deterministic builds).
It took me a long time to learn git. When starting out, I could only follow some rote commands, and wasn't able to work my way out of mistakes.
Or, tools like Vim or Emacs are tools with steep learning curves, but which provide a powerful user experience if you take the time to learn them. -- Although, you can certainly get by using other editors like VSCode without having to put time into them; but I wouldn't consider vim or emacs bad tools.
Developers commonly work with packages. (e.g. when installing toolchains, or dependencies for building some package). Having a tool that can so expressively and powerfully deal with packages like Nix can, I think if it's got some rough corners or needs some effort to use, it doesn't discount it as a tool.
However, I just want to address one thing.
> To be honest, nix is a system you have to actually use to begin to understand and synthesize. As noted, flakes make this hill less steep, but you won't really internalize the value and the system as a whole until you actually start using it some.
This is discouraging me from trying Nix for a couple of reasons.
Having read the thesis, understanding the necessity of reproducible builds and how Nix turns that up to 1100, and looking forward to being able to tell future contributors of my software to just run `nix-shell <whatever>` to set up my exact development environment, I think I understand the value of Nix pretty well. Being told that I don't understand it's value just because I don't use it feels a bit like condescension or elitism. You probably intended none of that, but that's what it felt like.
Second, flakes have not made the hill less steep for me; they've made it more steep, simply because understanding wasn't there. Some things are harder for some people than others, including things that seem like they should be easy to anybody. Math is easy for Terence Tao in a way that it's not for me. But I bet you there is something I'm a natural at that he finds hard or impossible.
Third, I agree, after all my study, that Nix is easier to understand once you use it. My trouble is that a package manager is a critical part of my system, and I don't want to run it until I do understand it. Standard package managers may be bad compared to Nix, no doubt about it, but they are understandable, which means I've been able to fix them when things go wrong. I have no such confidence that I could do that with Nix, so I haven't used it yet.
It is unfortunate, but there are a myriad of reasons why people who know about Nix, and want to use it, haven't yet used it. Sometimes, the reason isn't because they don't understand the value of it. I don't like that I am not yet confident enough to use it, but that's my situation.
I am glad that you acknowledge that it is only how it makes you feel as the original comment that are the cause of such obviously weren't intended for you interpret them that way.
It's a bit like if I told you that you need to use vim to actually be able to use it efficiently and you told that you had read the manual and understood really well the concept of modal editing.
I used to be like you, very confident in that reading, even regularly reading blog posts/articles on a subject was a great way of absorbing knowledge but at some point I have had to admit that that book/article I had read I did not remember much about and when faced with actually using/implementing something and I needed to actually use/implement multiple times to be able to do it by myself.
Yes, but the unfortunate thing about Nix is that so much of the documentation and interactions with the community come off that way, to me at least.
> It's a bit like if I told you that you need to use vim to actually be able to use it efficiently and you told that you had read the manual and understood really well the concept of modal editing.
I think you need to read my comment more carefully. I fully acknowledged that using Nix made it easier to understand:
> Third, I agree, after all my study, that Nix is easier to understand once you use it.
What you replied to was me saying that I felt like I knew what its value is. To take the example of Vim, it's like I have watched many users of IDE's and gotten some idea of how fast they can be with it, then saw an expert Vim user and realized he was 10 times faster than any IDE user. In that case, I would obviously see the value of Vim, even if I don't understand Vim itself.
Reading Nix's manual, the pills, and many blog posts probably won't help me understand Nix fully, I agree, but I disagree that I won't see its value without them.
In fact, if the documentation cannot help potential users see the value of Nix, then they won't take the time to switch. It's a high cost to switch and not something that should be done on a whim. (I don't consider trying it out in a VM as switching, but doing so adds to the time cost of switching.)
> I used to be like you, very confident in that reading, even regularly reading blog posts/articles on a subject was a great way of absorbing knowledge but at some point I have had to admit that that book/article I had read I did not remember much about and when faced with actually using/implementing something and I needed to actually use/implement multiple times to be able to do it by myself.
I do not deny this. I have never denied it. What I claim is that I understand Nix's value.
However, that brings up three points.
First, to be told repeatedly that I don't understand the value of Nix also feels elitist despite that not being your intention.
Second, if I actually do not understand Nix's value, that is a massive weakness in the documentation. Helping people understand its value is crucial to marketing Nix to a wider audience. It's been almost 16 years since Dolstra's thesis was made public. Nix needs that marketing. The industry needs Nix to be in wider use.
Third, the best technical documentation I've ever seen has made me actually understand the thing it was documenting before I ever touched it, and when I touched it for the first time, I felt like nothing, or very little, of my understanding was wrong, and that my understanding was sufficient.
(Funnily enough, one of the things that did this was the RFC for AES. I had to implement AES for an assignment in college. My code is not published, of course.)
I know that the Nix community acknowledges the need to improve the documentation. I believe that the aspect I mentioned above is one thing that could help a lot of people make the jump from sitting on the fence about it. If not, Nix's lunch will probably be eaten by something that comes along, does the same things or provides the same value, and yet has accessible documentation.
To be frank, I've been tempted to do that myself at times when I've only been stopped by knowing that I do not understand what it's doing.
It is easy to feel overwhelmed by the scope of everything within nix. So much of it is great. Many things aren't great at all. I do much better with it when I stop trying to take in the depth of it all and focus on the one small thing I want to do with it at any time. If there's something small and focused that you want to do with nix that elicits joy within you, I think it is good to dive in and learn enough to do that thing. I really like development environments as a focus, but that aligns with my personal and professional interest, and perhaps you want orthogonal systems or containers or application builds instead.
There is something to the understanding that comes through experience, but it's not because it helps me realize the thing I can see in my head, but rather because it will sometimes reach out through the machine and slap me very hard in the face. I have been dealing with this issue[1] since it began. It is pervasive, and it is painful. But it helps me realize how fragile this ecosystem we work with is, and it gives me satisfaction knowing that I belong to a group that might happen to surface issues present in safety-critical systems through nothing more than ceaseless rigor.
I hope you will join us at some point. I am confident that you will find it enriching.
You are trying to read the dictionary to learn how to speak a language. Theres just no substitute for actually using the thing.
You'll also need flakes and the nix command enabled first [3] (add that line to `/etc/nix/nix.conf` if you aren't using nixos).
Why use flakes? Mostly because it has a lockfile: there's a really good chance that "works on my machine" is "works on my team's machines." Flakes are also much cleaner than vanilla Nix.
[0]: https://gitlab.com/jcdickinson/nix/-/blob/main/home/general.... [1]: https://gitlab.com/jcdickinson/nix/-/blob/main/.envrc#L3 [2]: https://gitlab.com/jcdickinson/nix/-/blob/main/flake.nix#L65 [3]: https://gitlab.com/jcdickinson/nix/-/blob/70844981d5cd63c839...
If you are new to nix I 100% recommend starting with flakes. It simplifies so much for a newbie, e.g. no management of NIX_PATH and simple upgrades (`sudo nix upgrade-nix`). It also enables essentially single-line machine bootstrap on a new box even if you use home-manager.
Why learn this...
nix-env -iA nixpkgs.hello
nix profile install nixpkgs#helloAlso note that the "modernized" `nix` CLI interface is experimental/opt-in as well vs the out-of-the-box "legacy" commands (e.g. nix-env). So yeah, its a bit of an odd time in nix land as they transition but I don't see any indication that you would be poorly served or burned by moving to both nix command and flakes now.
https://www.reddit.com/r/NixOS/comments/v2xpjm/big_list_of_f...
* How does declarative management of user nix-profiles work now with flakes?
A Flake is a git repository or another Flake compatible URL. If you want to run a flake, it's simply:
nix run github:juser/hello
nix shell github:juser/hello
nix profile install github:juser/hello
nix profile update '.*'
nix run github:juser/hello?rev=53cd89289e44b269e28d2340f022e5f63525bcd3
nix-channel is no longer necessary when you go fully with Flakes.
For declarative management of your $HOME directory you need home-manager, which is still a separate community project, not part of the core NixOS.
EDIT: you also might be able to inject your local channels "pkgs" definition into the flake so that they reference the same versions. e.g. for home-manager with flakes you would do it like this
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
home-manager = {
inputs.nixpkgs.follows = "nixpkgs"; # <- deduped version of packages
url = "github:nix-community/home-manager";
};
};I have high hopes for Nickel[1]
[1]: https://www.tweag.io/blog/2022-03-11-nickel-first-release/
[1] https://discourse.nixos.org/t/tweag-nix-dev-update-33/20048
Considering the tight integration between Nix language v. Nixpkgs v. Nix program I mostly see CUE usable to generate data files but would eventually need to be consumed in a .nix config at some point
There is already some experimental setup for Nix to call Nickel for producing derivations using a function call importFromNcl. A more language-agnostic FDI (foreign derivation interface?) would be really exciting.
When writing Nix code it can often feel quite opaque what's going wrong, so writing actual package descriptions in a more constrained language like CUE, at least to me, feels like it might help. Better tooling in Nix and the kind of validation that Nickel proposes also look like good bets. I'm glad there is more than one horse in the race to improve the ecosystem.
I have been wondering for some time what would be required to define a package set just like nixpkgs in CUE (since it's not lazy and not touring complete). I only have very vague thoughts on that, so I would love reading about it from someone who knows more about how nixpkgs works or someone actually trying it.
Main challenge is the value of the nixpkgs repo is enormous, and that looks tightly tied to Nix the language and its implicit constructs. I think instead of an FDI one would have to provide a true competitor that is more attractive on multiple dimensions like:
- reworking Nix as more a modular architecture (see Tvix: https://tvl.fyi/blog/rewriting-nix)
- Tackling a package repo and Nixos equivalent with day 1 reproducibility (not just repeatability) like https://reproducible-builds.org/
- Better UX experience on the "entrypoints" of Nix like home-manager and dev-shell flakes - I think CUE has some nicer language features for this and does not require figuring out derivation generation, just referencing the existing nixpkgs set
What do you mean by day 1 reproducibility instead of repeatability? Isn't tracking and eliminating nondeterminism a gradual process anyways?
I don't see how nixpkgs or NixOS is a faulty basis for that kind of work, since I have not run into any flaws with regards to reproducibility that I thought would be easier to address on a clean slate basis.
I think the `fun` keyword isn't fun. The code clearly states `port =>` yet we still have to type `fun` everywhere.
Even JavaScript arrow functions (ES6; 2015) don't require this, although admittedly it took 20 years for this to become a popular enough idea.
So yes.
The documentation you can find is out of date with what people are recommending right now. Even the command to install a package in the official docs is now apparently the wrong thing to do. (This is not related to flakes)
In the end I didn't have time to separate what's current and what's not in the documentation.
People seem to have carved out bits of nix that sort of works for what they need and called it a day. Which is of course fine as long as it works. I will give it another look in 2 years.
If you're running a CI/CD server your definition of "installing a package" is vastly different from someone who just wants to install Chrome on their machine.
So far it's working like a charm, the only problem I have besides the Nix documentation which is spread all over the place is that if you want to write Nix expressions yourself to build packages which are not on Nixpkgs[2] binary cache gather all the patience you can muster.
[1] https://bob.build/docs/getting-started/package-management
I will go against the grain and suggest not using it, especially for native projects of any kind (C, C++, Go, etc). In my case I was doing Go work, but the binaries that are built are usually alright, unless you require Cgo and libc gets linked dynamically (Go will do this without telling you). Suddenly that binary won't work on production! It broke my server and I had to force static compiling so Nix's libc wouldn't be linked dynamically.
This can happen in Go if you do something like look up the user's home directory or cache directory, etc. Suddenly your Go binary is dynamically linked...
Nix's primary benefit is declarative and reproducible builds. But you can only benefit from it if you actually use it to build and deploy your projects. So instead of copying manually built binaries to production servers, you should create a Nix package and install that to your servers. Alternatively, you could use dockerTools.buildImage [1] to create a Docker image with all the runtime dependencies.
So my recommendation would be to actually use Nix, not avoid it.
[1]: https://github.com/NixOS/nixpkgs/blob/master/doc/builders/im...
That only cements the idea of avoiding Nix if you're using it for your own development workflow and the entire team/production deploy isn't on board with it.
> That only cements the idea of avoiding Nix if you're using it for your own development workflow and the entire team/production deploy isn't on board with it.
I disagree. Using it for your own development is fine as long as you keep it local. Either that or ship the complete dependency by building a Docker image as I mentioned earlier. Nix has various tools to make life easier even if it's only for personal development purposes.
Last time I checked it did pretty much everything you might need and was fairly pleasant to use.
Still feels like I don't know what's going on under Nix's hood, though. That syntax...
But i use Flakes through my entire setup, so i imagine that's why.
[1] https://github.com/nix-community/nix-direnv#flakes-support
If the Nix story was more together a few years ago I might have been swayed. Currently though containers is good enough.
Who knows maybe it is worth using Nix when it comes to ARM devices. We'll see I guess.
Nix also works on other WSL distros, provided they're using WSL2.
Nix supports cross-compiling Windows binaries as well. I know some people use it for that.
There is no 'native' support— you can't use Nix as an alternative to Winget or Chocolatey on Windows. Right now a lot of important stuff in Nixpkgs depends on a POSIX shell and Unix coreutils implementation for the basic build environment, and that's shared between many operating systems. Trying to fit Windows into that doesn't really make sense, and there's not really any momentum behind the idea of using any particular other runtime environment (could be a scripting language instead of a shell + coreutils) for those basic builders.
But it's conceivable that some day, one or more companies using Nix on WSL might see vaiue in taking that extra step and put together a Nix-based package collection for Windows and help get the Nix Windows port out the door. It also seems possible that a motivated Nix developer could take it somewhere on their own, since at least one has a workable PoC IIRC (but they didn't want to create/maintain a separate package collection).
For now, it works very well with WSL2 and that's the way to go for Windows users.
But IMHO I would try to take a flake centric approach because there's a _ton_ of cruft in the docs about stuff like channels that all gets completely jettisoned and dropped once flakes are the norm. It's just a lot of concepts and CLI commands, flags, rituals, etc. in the docs that will give you nothing of value in a flakes world. Unfortunately there are no real docs that make this clear that the nix world is deep into a major transition and flakes are the clear future.
This 1 page intro to the nix language was all I needed to get comfortable with it too: https://github.com/tazjin/nix-1p It's really a pretty simple data-oriented language when you get down to it.
What were you trying to do such that the way the docs said to do it was wrong?
"package manager" is literally in the first sentence.
Nix does have equivalents to "apt install <package>", etc... but, it's prone to "I forgot I did that / I didn't know I did that", and problems arising from that. (Other distribution package managers like apt-get or pacman require installing packages as root; but with nix packages should be installed as non-root. And the nix command to update installed packages isn't intuitive).
What's typically encouraged is to instead install packages as part of the NixOS configuration, or as part of a home-manager configuration.
Or, just using commands like "nix shell nixpkgs#<package>", which just adds the program to the path in the current shell, without installing it. (Which is a neat feature of nix).
Yes, if you think of nix-the-language as a bunch of helpers that creates a big key-value JSON object then you will start to get a better mental model.