Automate Project Environments with Devbox and Direnv (2023)
jetify.com
jetify.com
I'm a bit confused. Direnv is rather simple, is it not.. ?
1. put env variables into .envrc 2. direnv allow
I don't use nix and I don't think I'm over its learning curve. I don't remember investing any time into it..
Edit: this is the only thing I use direnv for - I haven't used direnv without touching Nix.
Last time I tried to use Nix to build a dev environment, it did not go well. I wound up in a Turing tarpit of dependencies, as my Python program needed natively compiled libraries and other things to even get off the ground. I kept having to pull in more of the Nix environment/OS to achieve functional isolation from the parent Ubuntu OS. I also ran into problems with online docs and advice skewing wildly from what the CLI options required of me (probably a skill issue).
And that's when it struck me. Docker has pretty much raised the bar for the developer experience. It can achieve the same amount of isolation for far fewer keystrokes, in less time, and (with docker-compose) has a sharable configuration for others. At the end of the day, a container is just a well isolated process whether it's a webserver or a user shell. And really, that's what you need if you're trying to solve repeatability and isolation. As a bonus, if your software is going to deploy as a container, you're that much closer to emulating the production environment (read: fewer bugs).
I like the Direnv concept and have nothing against Nix or folks trying to share the NixOS experience with others. But from a DevEx perspective, my expectations are set at "as easy or easier than Docker."
But if you are comfortable with Nix you don't need Devbox per se, as you can get by with using Flakes. Here are some examples -- just `git clone` and run `direnv allow` (once), then you are put in the corresponding devShell
Haskell: https://github.com/srid/haskell-template
Rust: https://github.com/srid/rust-nix-template
home-manager: https://github.com/juspay/nix-dev-home
https://github.com/nammayatri/nammayatri
There's dedicated direnv configuration for backend and frontend development: checkout `.envrc.backend` and `.envrc.frontend` files. They both use `use flake`, specifying the appropriate devShell argument.
I add .envrc to not only avoid committing it to projects that don't use it, but also because it turns out that it's still pretty environment dependent. There's 100 different ways to skin the dependency problem: Nix, pipenv, nvm, rustup, etc. There's no telling which a contributor prefers to use, so .envrc doesn't belong in git IMHO. What could be committed is something like .envrc-flake (which can then be sourced into .envrc).
As for needing a flake in the repo, I have found a workaround. `use flake` accepts parameters, including the path to the flake. I have a bunch of shells[2] that I can import with e.g. `use flake ~/Nix#rust`.
[1]: https://codeberg.org/jcdickinson/nix/src/branch/main/home/co... [2]: https://codeberg.org/jcdickinson/nix/src/branch/main/flake.n...
Regardless of the distro or even OS. An easy thing to say, but hard to execute on actual projects with patched versions of cPython and some obscure in-house build tool.
I say this as someone that was using Gentoo before lxc even existed. So this isn’t code for “I don’t get it”.
I think it should be standard practice that you can pull and run a project without configuring environment variables. I agree with that. The defaults should work.
This is not always possible (when you have something that absolutely relies on a third party thing, which you need an environment variable for) - in which case you should be greeted with a helpful error telling you what environment variables you need to set up and why.
Obviously don't put secrets in there. For secrets and overrides you can use separate a `.env` file which is not checked in, and source it in `.envrc` with `dotenv_if_exists`
.envrc is a whole bash script, so it can invoke command line tools that fetch secrets over the network or decrypt them from the disk.
It's very hard to define in general what it means to build a project, what are the expected inputs and outputs. Not every build aims to produce the same artifacts, not using the same inputs etc.
What you want could be based on some typical practice in a particular field with a particular project size or structure, but this doesn't transfer well to other fields.
Also, as a benefit, by tracking the JSON and lockfile, the CI can work with the exact same environment. I highly recommend giving it a try.
This benefit is also possible under both Nix flakes (inherently) and vanilla Nix. The latter takes mild deliberate effort to not use channels-based import syntax but instead a fetcher with a pinned URL argument.
They mention Nix but I guess they are referring to the use_nix integration built-in to direnv? There is a fairly detailed (but old) comparison in https://github.com/direnv/direnv/wiki/Nix#some-factors-to-co... and in the meantime lorri and nix-direnv are the only ones still maintained. But you aren't limited to Nix when using direnv, for example there is nvm. https://github.com/direnv/direnv/wiki/Node#using-nvm
Actually, the post's solution, besides being built on top of direnv, uses Devbox, which itself is built on top of Nix... https://www.jetify.com/devbox/docs/faq/#how-does-devbox-work So they are saying they are outside the Nix ecosystem when in fact they are in it. It is yet another set of nice scripts on top of Nix.
In case you hadn't realised, the very concept of having two sets of binary distributions on one machine, vying for superiority and the correct version of glibc... is fraught.
Most of my use was with rails projects, and I can't recommend it.
Coupled with an abstraction that tries to save you from Nix, but almost entirely fails, you end up with a bloated hellscape where every time you load your project it will unnecessarily reinstall your packages and several times an hour it will have forgotten curl exists and so you have to manually reinstall curl (not-so-slowly increasing your /nix folder's size), every week or two a new version of devbox completely changes the workarounds you need to do, and don't try to garbage collect nix or it will delete vital files, and you end up scrubbing it all and starting again.
In python, it overrode the path so I couldn't get it to reliably use the binaries in the venv. Pip and Python were using packages in different places and I couldn't get them to converge for love nor money.
The devbox team were great and really tried to get things working, but in the end I couldn't get it to work with enough stability to properly recommend it to my team, and if I wanted it to half-work for any substantial length of time I had to lock to a version of devbox.
Obviously, ymmv, please do give it a try, it's an impressive project. But my view is that it's trying to do something that is very very hard, and for that you need a very clever solution. And this is a very clever solution, with very clever bugs, and so it's not something I'd recommend jumping into with both feet.
One difference that may have been decisive for our success is that when I selected devenv, my goal was not to avoid Nix at all, but rather to choose a nice convention for defining our project-specific Nix environments. I chose devenv because I trust the author's technical leadership (and it also has a decent community of contributors), it is very Nix-forward for this type of tool (most of defining your environment can be done in Nix, and you can even use it just as a Nix library rather than an executable tool (which we do, for one project)), it supports flakes in a first-class way, and it's built on my favorite library for defining flakes (flake-parts) which works via a NixOS-like module system. It also takes care of the direnv/caching optimizations we'd otherwise want to roll ourselves. I also love some of the small conveniences it offers, even when they're pretty easy to do with Nix alonw, like the `scripts` functionality. I also find it pleasantly easy to extend— my team has a couple partially completed upstream contributions on the backlog right now, and they were delightfully easy to get working.
Devenv does admittedly have a lot to it. Each of our projects generally only needs a subset of its functionality, but that breadth of functionality seems justified by the fact that it's not always the same subset that we need. Devenv does currently do some things I don't love, like use a custom Nix build for evaluation and recommend a custom Nixpkgs fork for best compatibility, but why it does those things is clear once you dive in and the future direction in those areas seems sound to me.
All of this is from my perspective as my small team's 'Nix guy'. My manager has some casual Nix experience outside of work, and has had success creating his own devenv environments for some projects I'm not involved in as well. Our other guy is totally Nix naive, but has a solid Unix background. He's never initiated any new work with devenv, but he's used our existing devenv integrations without issue.
I think for a team where everyone is still getting their feet wet with Nix, the approach you guys settled on is quite a sound option. But for teams with more mixed Nix experience levels like mine, maybe devenv can work better. For us, it does a nice job of providing a well-documented, idiomatic, highly compatible Nix library to experienced Nix users on the one hand, while presenting a nice porcelain for the other members of the team on the other.
Apart from glibc, I have never had issues with two sets of binaries vying for superiority. Nix binaries take preference with my "vanilla" setup. I think that might be Devbox doing something strange with $PATH?
There were a couple of other libraries, can't remember which ones. I remember once having a fun chain of a library that depended on a library that depended on two libraries that in turn depended on glibc, and for some reason the last link of the chain, only one of the libraries was hitting the system libc incorrectly - that was a fun one to debug. I think I ditched that dependency in the end, it was the only solution (and was clearly badly written).
One of my projects used an older version of ruby. In that case, there was a gem to connect to the database, and that gem links to the db client library, but the db is new and the ruby is old and guess what? Two different versions of glibc, both being used within the nix ecosystem.
I worked around a lot of it with LD_LIBRARY_PATH (I think? from memory) which I had to unset for everything in devbox, and used aliases to set it to a backup of that env whenever I found a binary that needed it - and then they tried to fix that, but it just seemed to stop my workarounds from working, so I had to come up with new ones.
But yeah, it was a wild ride. Most of it came back to glibc or environment variables or both, and probably me doing something I really ought not to do (like support old projects). Alas, for me, it wasn't worth the effort - but I sure learned a lot.
Can't manage lower-level deps like dynamic libraries and C compilers, though. I just containerize those.
Also, any solution of this kind will inevitably introduce a number of new problems proportionate to the internal complexity of such solution. I.e. this is just asking for a lot of problems, while offering very little in return.