Fast, Declarative, Reproduble and Composable Developer Environments Using Nix
devenv.sh
devenv.sh
If the ideas seem pointless and confusing to you, that's fine, don't use it. There are bunch of great package managers and linux distros that will probably suit you better. But, if you see the need for declarative package management, then Nix is for you.
I'm a phd student in computational science. I use a bunch of different packages on my computer and I need to know exactly what I've installed and I do not have time to debug problems from dependency hell. Nix and NixOS have been two of the most interesting and useful tools I have every used. Not only have a gained huge productivity boost from having reproducible environments and declarative package management, but I've thoroughly enjoyed the process of learning these unique tools that, for me, are leagues better than any package management system I've used in the past.
If it didn't work for you, I hope you find a tool that suits you better. But that doesn't mean its bad. It might just mean it wasn't intended for you.
In my case, I have a few different computers, and I declare all of their environments with NixOS in the same directory. Then I use syncthing so that all the builds are automatically updated and in sync. This means that I can develop something on my desktop, and open up my laptop and run it without configuring anything. If it works on my desktop, it works on my laptop and vise-versa.
For example, I make a development environment on my desktop using a flake.nix file and a .envrc file in directory. I type direnv allow in that directory (actually all this is done with a script I wrote so I can just type "dev-env" in the terminal in the corresponding directory". Then open my laptop, syncthing makes it so the entire environment and all the files are already on my laptop. I type "direnv allow" in the directory in my laptop, and then "rebuild" if the NixOS configuration was changed. Then I can run the software that I wrote on the desktop.
That's a terrible copout. Nix ought to be a great deal more accessible than it is. The premise ideas are sound, but the UI/UX is not.
Of course this is a whole env thing and not just a package thing but I dunno I'd almost rather build a container and work within that than write nix to get this functionality, the ergonomics of teh language are that galling for me.
Started using nix this year, language is easy to use. Has very few concepts and syntax, it’s a standard functional language with all those goodies and a clean syntax in my opinion.
The language is fine, especially compared with its alternatives (e.g. templated yaml, yuck).
How are you meant to install a piece of software and keep it up to date?
Is it a channel or flake? Are flakes stable or only for testing? So what's homemanager for?
Good luck getting answers on those questions other than "read the source code" and then followed by "no, not that source code, this branch here".
I really like the idea of Nix, but the communities priorities leave a lot to be desired of.
I usually recommend this guide for newcomers to Nix: https://zero-to-nix.com/
I don’t think there’s a beginner-friendly guide to NixOS. I personally learned it (and continue to!) through a number of blog posts and examples on Github. The NixOS manual is a key reference to rely on, but not really a tutorial: https://nixos.org/manual/nixos/stable/
Channel = group of named deps. 'Traditional' nix tracks a channel so when you update it, anything pointed to it also updates.
Home Manager = NixOS lite. Good option to define your preferred environment deterministically if you like another OS.
I experienced a similar situation last week with git-hooks.nix[1], a pre-commit integration for Nix.
I wanted to run biome[2] checks on my repository during pre-push so I wrote a custom hook because git-hooks.nix has pre-defined integrations with prettier and rome, but not biome.
Or that's what I thought. I eventually found out that the rome hook is actually referred as "rome" everywhere but calls biome instead[3]. This wasn't documented anywhere, so I opened an issue[4] suggesting to rename the hook to "biome" and keep the former for backwards compatibility reasons.
As of today, this has been acknowledged by one of the maintainers, whose sole feedback has been to "thumb down" the issue.
TL;DR: It's not just the documentation, but also the code not doing what you would expect. It also seems there's no means to improve the situation other than just forking the project since there's also clearly some kind of communication problem.
[1] https://github.com/cachix/git-hooks.nix [2] https://biomejs.dev/ [3] https://github.com/cachix/git-hooks.nix/blob/40e6053ecb65fcb... [4] https://github.com/cachix/git-hooks.nix/issues/428
I've not seen such hilariously bad communication since trying to talk to my thesis advisor.
Guix might use a scheme, but at least it has good documentation and seems more open to accepting contributions.
Looks like the OP, domenkozar, is that maintainer. Perhaps they'll reply to this thread :)
Trying to use a container to accomplish the same thing is not a substitute. They are fundamentally different.
The Nix language is designed by people that think everyone is familiar with lambda calculus and function programming. That's fine. But it's crazy to use a language like that for something that everyone has to use.
I bet Nix would be way more popular if it used something like Starlark instead.
I'll keep trying to give Nix and the language more work on my end.
I wholeheartedly disagree. Nix's `a -> b` is just a boolean function, like `&&`, `||`, `==`, etc. It's meant for assertions, e.g.
assert format == "csv" -> depth data == 2;
assert(depth(data) == 2 if format == "csv" else True)
> The Nix language is designed by people that think everyone is familiar with lambda calculus and function programming. That's fine. But it's crazy to use a language like that for something that everyone has to use.
The Nix language is basically just "JSON with functions" (although the syntax is unfortunately different, since JSON wasn't so popular back around 2003; it does support XML though!). There's only an emphasis placed on functions since they're the USP of the language; otherwise we could just use plain JSON. Lambda calculus is just a simple model of functions; you don't need to know or care about it in order to write functions in the Nix language. In the same way that you can write integer arithmetic in Nix without knowing Peano's axioms, or von Neumann's set encoding of ordinals, or whatever.
I've not used Starlark, but it seems to be waaaay more complicated than "JSON with functions"; e.g. it has loops, which makes the semantics dependent on some notion of time/temporal-ordering, which is sounds like a massive headache.
It's fairly straightforward to use. Running `devenv init` and then looking at the devenv.nix file should be all you need to know what to do.
Package names can be found by `devenv search $name`.
If you're happy writing shell commands/scripts, then just stick them in the following template:
with import <nixpkgs> {};
runCommand "name-of-your-thing"
{
someEnvVar = "some value";
someOtherEnvVar = "some other value";
buildInputs = [ programs you want to run ]; # e.g. see search.nixos.org/packages
}
''
run any bash code you like here
just make sure the file/folder you want to output
uses the env var "$out" as its path
''I don't think it's a good idea to defunctionalise the Nix language itself, since the result would either be too inflexibile or incredibly complex. Keep in mind that the Nix language has no idea what a shell is, or any concept of "package", etc. yet it's flexible enough to write functions like `runCommand`, which is used in the example above.
Also, it's reasonable to argue that Nix is already declarative, since that's what the `.drv` files are for! However, they're so tedious to work with that it's preferable to generate them using a "proper" language like the Nix expression language (or Guile Scheme if you're using Guix)!
(See http://www.chriswarbo.net/projects/nixos/bottom_up.html for a more in-depth explanation of .drv files, and how they relate to the Nix expression language)
Some of these 200+ issues are unsolved for a fairly long time.
It's a very ambitious project, so a lot of issues are related to either Nix or general support for each language/service.
Though something that annoyed me with the recent v1 release is that it changed the default repository where it pulls the package definitions from Nix's official to a fork made by the author.
That is dangerous and also lags behind an incredibly active and large upstream.
If you want to patch things, use proper Nix overrides or apply the patches using devenv code, don't fork a +80,000-big rolling release package repository.
I can unzip a directory, cd into it and cargo build it and for the most part it just works.
The fact that we have added shared libraries and environmental variables and so many other magic incantations that we need massively complicated second systems is an indictment of our culture of complexity.
I believe Go also embraces that level of simplicity.
And when it comes to deployment, instead of needing a simulation of a whole operating system (containers), I can just remote copy a directory and just have it work.
For example, anything using bindgen needs libclang, and to make matters worse, doesn't look in the conventional LIBRARY_PATH but requires a special environment variable LIBCLANG_PATH.
I like static binaries a lot. Whole load of failure modes just gone. Building a clang that targets musl by default on glibc systems is loosely practical these days. That builds self contained binaries that only depend on syscall.
(It took me a few days, a lot of cursing at cmakes ideas about cross compilation and patching trunk slightly, but it can be done and enough of the patches stuck.)
In general, you can configure systemd yourself through `systemd.services`, where you can write systemd service files ‘directly’ in Nix syntax. Services for which multiple instances make sense often (unfortunately, not always) provide configuration that allows you to specify multiple instances (i.e. their top-level configuration object will be a list or attrset of instance configurations). I've written a little bit about patterns to do this here: https://twey.io/nix-patterns/inputs-and-outputs/
If you needed, say, multiple instances of Postgres, it's not too challenging to copy-paste the nixpkgs implementation, change it a bit to parameterize the config on (e.g.) a service name for namespacing, then import that module into your NixOS configuration to allow you to define multiple instances. For example, I did this here for the Rainloop email client: https://github.com/Twey/dotfiles/blob/main/nixos/modules/ser...
I still think it’s easier to just spin up a NixOS container - should work for any service out of the box.
I often want to have some packages/tools installed when I'm working on a project and docker isn't always the right fit, which is where devenv comes it.
I basically use it as a generalized version of venv.
Some recent examples:
- Downloaded a .7z file, nix-shell -p p7zip, extract, exit shell.
- Was working on a paper, set up texstudio w/ devenv.
- Needed to use vagrant and virtualbox, however apt installing them was impossible due to a dependency issue (incompatible on pop os at the time). Trivially fixed by just: devenv init -> add to devenv.nix file -> devenv shell
* Declarative -- as a developer, I don't care... Why is this important? Also, declarative means there's a bunch of imperative code hiding behind it. It just makes it harder to debug when things break.
* Composable -- in over 25 years of being a developer not even once did I want my dev. environment to compose with another...
And the article opens up with:
> Simple JSON-like language
Why on earth?.. And things down the road that announce "features" like "run processes"... Is this some kind of April 1st joke that is meant to be funny because its two weeks too late?
If you are interested in my job history: it includes two of the top-ten largest US s/w companies. My area of expertise is storage (more in the direction of SDS as this was a fashion when I got into this field) and more recently HPC. I'm less familiar with the aspects of modern programming s.a. making Web sites for example, and somehow, I have a feeling that programmers who make Web sites really like talking about microservices. So, maybe this will explain my lack of understanding of how you set up your development environment using microservies.
If you are working on something that involves microservices then you might have several different projects with different development environment needs, all checked out locally. It doesn't mean you're "setting up your development environment" using microservices, it just means that's literally what you're developing.
In my practice, it's very uncommon to be able to run even a small portion of the project I'm working on on my personal computer. There's usually no way for me to even attempt to set up anything resembling the system I'm building on my computer. Which is unfortunate, and I'd love to be able to, but it's just physically impossible.
I don't know how typical my case is. I'd assume that programmers working on desktop applications or phone / Web stuff are more likely to be able to fit their entire product or at least a significant portion of it on their own computer. In which case they could think about their testing environment as being part of their development environment. Still, I'd not call it that... I mean, conceptually, it's for testing, so it's a testing environment. You don't need it to do development, it's just coincidental that you have both in close proximity.
I've worked on stuff that's not at all web or desktop related (e.g. Materialize, hosted cloud data warehouse software) and still, we had ways of running it locally; otherwise most people would have found it impossible to develop.
How fast it installs, how fast it's ready when needed. This can be relevant depending on how you use them. For example, if I let every script run in a separate environment, it would matter if it's ready in 10 seconds or 10 milliseconds.
> Declarative -- as a developer, I don't care... Why is this important? Also, declarative means there's a bunch of imperative code hiding behind it. It just makes it harder to debug when things break.
As a developer, I do care how much work something will offload on me. And declarative is more about the state of the world, than the execution of a process. If there is a trustable system which can manage that world for me, then I will have less work, and fewer problems.