Devenv.sh: Fast and reproducible developer environments using Nix
devenv.sh
devenv.sh
I've been part of the Nix community for more than 10 years and in the last 4 years focused on making it documented, simple and accessible for any developer.
After building Cachix (where you can store any software binaries with a few steps) we realized that there needs to be an intuitive interface for crafting developer environments.
I'm really looking forward what you build on top of devenv. We're only beginning to explore the area of what's possible, so please give as much feedback as you have.
I've used it as my daily driver for the past year so maybe I can help elaborate too.
For desktops there is that value as well. However, I have usually set up my macOS environment in two hours or so, the pay-off is a bit less for me. However, setting up a dev environment for development projects is awesome, especially if a project has dependencies that are normally considered system dependencies (e.g. native libraries).
NixOS is more similar, culturally (rather than technically), to Debian than it is to Ubuntu or Fedora, both in the sense that it's not backed by a large corporation and also in the sense that it doesn't promote a particular vision of the Linux desktop.
On a technical level, its package management approach will feel like a larger paradigmatic leap, coming from a Fedora or an Ubuntu, than it would from a Gentoo. NixOS doesn't have quite the same notion of a repository as either of the former. The concept of 'overlays', used to extend Nixpkgs, would be familiar to a Gentoo user but not a Fedora user.
That said, an Ubuntu user won't notice a lot of compilation from source, and in fact they'll likely find Nix pretty fast as a package manager. And likewise, NixOS is more different from any of those than it is similar to Gentoo.
An Ubuntu user switching to NixOS will likely be
- impressed with the wide selection of packages
- happy with how easy it is to iterate on a config
- delighted by the rollbacks
- pleased with how easy it is to add most software to Nixpkgs
- eventually frustrated by the inflexibility of having to package things for Nix to get them to work well on NixOS
- appreciative that their system doesn't accrue cruft over time
- perplexed with having to learn slightly different processes for packaging things in different language ecosystems in Nixpkgs
- blown away by the knowledgeable and helpful community
Jumping right in with NixOS is a good idea if
- your needs are simple, or
- you have a bit of packaging experience (for any distro), or
- you have an interest in functional programming, or
- you like learning new things, or
- the reproducibility and uniform management of your system's configuration is highly appealing to you.
On the whole, I think any experienced Linux user will at least find it interesting. But the more common response from those who really get a handle of it is 'I could never go back'.
The two things that stand out that maybe look slightly easier are the processes section, and the pre-commit hooks sections - but from what I can tell these are generally all solveable via native flakes.
Just curious - if this is a valuable tool to add to the arsenal, I'm all for it, but I'm curious what the strong selling points are here over what is already available.
The main benefit is abstraction using modules: this allows you to extend, compose and abstract your environment. Processes and pre-commit hooks are only the beginning of what's possible.
There's also higher-level CLI interface for building images, sharing environments, etc.
Note that this is only scratching the design surface of what's possible.
I've been mentioning to a work colleague that someone needs to do for the somewhat baroque Nix tools and concepts what Docker did for containers.
It just needs a nice porcelain over the top that provides some consistent CLI UX, but lets you dive down deeper as you get more comfortable with the system.
As someone who just two weeks ago got into Nix/NixOS via this repo[1] from mitchellh, one thing I'd love:
A nice, seamless way to remote connect JetBrains IDEs or VS Code into the environment.
I tried the VS Code server, but had some problems getting it going on ARM Linux (my machine is an M1 Mac, and I'm running nixOS inside an arm64 VM).
I have been surviving on NeoVIM, but really would like to use the editors/IDEs I am used to.
Something like starting up the JetBrains Fleet/Gateway kind of thing, or VS Code server, when entering an env, so that I can connect the IDE to the environment, and it has access to everything in that environment.
Since I don't see me convincing the 20+ people in my team to switch to VIM :)
This shouldn't be more than a day of work.
Also, it's called devenv.sh, but.. is it written in shell script?
image: nixos/nix:latest
before_script:
- nix-env -f shell.nix -i -A buildInputs
Of course I'm not sure how that would work - one of the things that interests me with devenv over 'vanilla' nix is your approach to processes which doesn't map so well in the scenario I've described.
It calls `devenv ci` (which checks that the devenv file is good), then runs a 'script' in the devenv file... https://github.com/cachix/devenv/blob/8bea91cd1073b0d51cf96e...
The 'script' in the devenv file: https://github.com/cachix/devenv/blob/8bea91cd1073b0d51cf96e...
the adjacent area i’m struggling with is the “i want to write a tiny program to verify some conjecture, and i’ll probably throw it away an hour from now”. think codegolf competitions, if you can’t relate.
the environments i create for these follow similar patterns. it’s either python with numpy, pandas and plotly, or it’s Rust with clap and serde, say. i’d love a tool where i can just `cd /tmp/my-hourlong-project` and then `devenv python` to get a python shell that’ll have everything i often use (and probably more).
hearing from people who use these tools, nobody has told me that any of these can do this — except that since they crawl up the fs tree looking for their env definition maybe i could just stash some definitions at the fs root and get globally-invokable environments that way. seems hacky though: i’d hope for a method that’s more officially supported.
mkdir /tmp/my-hourlong-project cp ~/my-env-definitions/python.nix /tmp/my-hourlong-project
Is that something like what you mean?
Or you could write an alias to do:
mkdir tmp/project && cd tmp/project
cp ~/stuff/scratchbox.nix .
<env-manager-tool> scratchbox.nix
There's a dozen projects that do this. But it would honestly take you as much time to write a shell script to do it as use some other project to do it.
Like, seriously:
#!/usr/bin/env sh
set -eu
[ "${DEBUG:-0}" = "1" ] && set -x
MY_TEMPLATE="${MY_TEMPLATE:-$HOME/.my-template.d}"
_cleanup () { cd ; [ -n "${tmpdir:-}" ] && rm -rf "$tmpdir" ; }
trap _cleanup EXIT
tmpdir="$(mktemp -d)"
cd "$tmpdir"
cp -a "$MY_TEMPLATE"/* "$MY_TEMPLATE"/.??* . || true
[ -e ".init.sh" ] && . ./.init.sh
python* I think I remember that `set -e` means to exit early on error, but I don't know enough of the sharp edges to ever feel comfortable using it given how the few times I've tried I inevitably end up having it set for my entire shell session instead of only within the script and not noticed until my shell exited some time later when I made a typo * I recognize `${DEBUG:-0}` as the syntax to use the value of `$DEBUG` if it's set and `0` otherwise, but I can never remember it without googling * I'm guessing `trap _cleanup EXIT` means that `_cleanup` should be run when the script exits, but it's not clear to me whether `EXIT` is some special keyword, and if so what other values would be valid there * I don't think I've ever seen anything close to the left hand side of the `&&` in the cleanup function. Presumably it means to make sure the directory exists before trying to remove it, but I have absolutely no idea why `tmpdir` could possibly be unset or why it would even be a problem if it was even independent of the fact that I would have thought that the `-f` flag to `rm` would make it ignore the directory not existing * is the final `;` in cleanup necessary? If I got a syntax error without it, I don't think it would occur to me that it would be needed * I've never seen `??` in a path before. I'm guessing it means to try to expand `.*` but then not fail if there's nothing? I don't think I would have even thought of the edge case of the expansion not finding anything being a failure
Could I write a script that does maybe 80% of this fairly quickly? Yes. Would the remaining 20% mostly be not thinking of edge cases and then getting frustrated when using it? Probably. Would I give up at this point and just try to find some other tool to handle it for me rather than doing it myself? Almost definitely.
don't mistake me for a lazy user in the sense of "doesn't want to write my own code". i'm actually the type of lazy user who prefers to find a project that gets me 80% of the way there, patch it where it falls short, and then upstream my changes so that i don't have to maintain an ever-growing pile of custom patches and one-off shell scripts on my lonesome ;-)
I'm really excited for nix as true sandboxing would be great. Nix however I couldn't get working properly.. and I've been using Linux for almost 17 years.
Once the UX gets sorted, this is a game changer.
Someone has told me that this can be solved with asdf-direnv so I am wondering if this is true and I should actually consider using asdf.
There are kind of three levels of language support in devenv:
1) Basic tooling + system dependencies, this allows you to use native language package manager and provide system deps to make sure they build.
2) Compiler/interpreter versioning: pick any version of the tooling as you need
3) Full automatic translation of the language tooling into a Nix expressions for the project. This allows you to get all the benefits of what Nix has to offer. This is the most work for each language but the benefits are enormous.
devenv.sh plans to achieve that step by step :)
Sometimes I've own image with basic dependencies that I like.
Honest question - do these Nix based tools offer more than that?
Where can I view that work? That's a feature the whole Nix ecosystem could benefit from. Is there an RFC for Nixpkgs to change the policy on versioning, or is this an effort to rig up some kind of frontend for an index that lets you pull package recipes from different versions of Nixpkgs?
For a lot of the most important ones, though, nixpkgs already maintains multiple recent versions, though— fifteen boosts in there right now: https://search.nixos.org/packages?channel=unstable&from=0&si...
Binary package managers need a first-class notion of versioning for install-time dependency resolution, Nix doesn't actually perform install-time dependency resolution.
Nixpkgs is a monorepo, and thanks to the way it supports side-by-side installations of conflicting versions, instead of using version numbers to check/assert compatibility, Nix just ships each package with exactly the dependencies it was built with, all the way down the dependency closure.
While it doesn't retain that fixity all the way down the dependency chain in the case of 'partial upgrades', this is also how Homebrew works. Gentoo Portage does include some functionality for reasoning about version constraints. (I'm not sure about other source-based package management systems like Ravenports or FreeBSD Ports or MacPorts.)
If Nixpkgs (or a counterpart, like an archive repo) included not just the 'current' set of combinations but a collection of old build recipes along with a record of version numbers for which builds had been successful with that recipe, or a range of versions that were known/expected to work... that could be really useful for projects downstream in the Nix ecosystem. It could power tools that import or generate Nix code given some packages and version constraints. But it's not actually needed in order for Nix to match the practical functionality of best-in-class binary package management systems like dnf.
There actually are already some tools in the Nix ecosystem that use (indices of?) old copies of Nixpkgs as this kind of archive: mach-nix comes to mind! Idr if it uses them at a source level for creating new build definitions or if it actually uses the whole dependency tree, embedded in those old versions of Nixpkgs. Either way, a better version of that is what I think Nixpkgs could soundly facilitate, while still only maintaining a single combination of packages as the official, coherent, cached collection for use by end users.
You could do the same basic versioned dependency solving all the way down your dependency tree using the same tool, whenever one of the recipes expected a version not in your current environment.
That's what it would mean to incorporate real dependency solving into Nix/Nixpkgs. I don't think you'd want packages in Nixpkgs' top-level/all-packages to rely on that feature, but you could have it be there for devs who want to build out bespoke environments but rely on the collective wisdom about how to build this and that at such-and-such version currently encoded in the history of Nixpkgs.
Sometimes you couldn't be offered a solution that you're confident would work— this kind of thing would be a starting point for maintainers of complex environments. But it would make porting old versions forward to a newer env easier just by helping automate that process of digging for examples in the history of Nixpkgs.
Maybe this is basically what you have in mind with the idea of a 'patching layer'. In that case, I agree that a really useful implementatiom would involve tackling some substantial problems. But I do think it is possible to make something useful along those lines, and it would be an exciting tool even if it was only 60% of the way there. Hence my curiosity a couple of posts up. :)
All this lowers the fragility of the package definition and I think helps increase the likelihood of that software being forward- and backward-compatible with different versions of its underlying dependency pool.
Nix is basically a whole load of compiled dependencies pathed to /nix/hash/dependency
So you can have things that would ordinarily be dependency hell running side by side because foo that requires bar6 is compiled against that, and baz that requires bar7 is linked against that.
Both versions of bar are present in the nix structure, on a specific path that the software is compiled against.
Nix 'isolates' packages by ensuring that they do not know where to look for each other, rather than ensuring that they cannot possibly see each other.
In addition to the example given by the parent poster, here are some other steps taken towards that end in the Nix ecosystem:
Nix-built programs look for their libs, they don't see libs other than what they were built with because each package has its own little FHS-shaped tree, which it thinks of in the way a 'normal' package might think of as 'the system'— that thing which has a `/usr/lib` in which to find libraries and a `/etc` in which to find config files and a `/usr/share` in which to find assets, etc.
In addition to linking against hardcoded full paths to dependencies, outlined above, maintainers also take steps to ensure that, e.g., external programs referenced in shell scripts in a Nix package also refer to full paths into the Nix store.
https://nixos.org/manual/nix/stable/command-ref/conf-file.ht...
Installing and using Nix packages doesn't generally involve any sandboxing or containerization features. But on Linux, there are some exceptions. A few proprietary packages use something called an FHSUserEnv, which leverages user namespaces to simulate an FHS-compatible environment. Additionally, Nix (through one of the new, experimental commands as well as an older third-party tool that inspired it) can also bundle any Nix package into a containerized package which can be run without Nix. I think those bundles, if you choose to create them, also use some container-y Linux features.
Anyway devenv.sh isn't built on anything container-y in Nix.
Also you can have multiple versions of the package cached.
Also all your environments benefit from the cache, since each "layer" is independent.
Docker's layer based caching is very limiting for larger images. With Nix you spend basically no time on incremental builds outside of the time for the one package you changed.
Build once on one machine and all developers can just download the build.
Nix (not to be confused with NixOS) is a package manager. Think of it like apt.
Containers on the other hand are (usually) utilizing kernel level isolation to run a whole user space starting with PID 1. These isolation techniques have overhead.
Since Nix is a user space application, you can run it in a container and Nix provides one `nixos/nix`.
Overhead: Windows and macOS can't run Linux-based containers natively. Instead, there's always a full Linux virtual machine running in the background acting as an intermediary and host for your containers. Nix can conjure arbitrary native development environments on a per-command or per-terminal basis, giving you all the performance of directly running tools without the risk of clashing with systemwide software.
Reproducibility: Nix provides much stronger guarantees about the exact versions of software you're running. It effectively gives you a lockfile for your entire dependency chain, all the way down to libc. Containers tend to be more stateful: everyone on your team may be using the same Dockerfile, but if you build an image from it two weeks apart, you're probably going to get very different outputs due to things like your apt-get update step returning new versions of packages. This doesn't happen with Nix.
The beauty is that this isn't either/or; you can actually use Nix to generate OCI container images which are thus fully specified and repeatable.
The build recipes pull from the original source and use a hash to ensure that the source artifacts don't change. Nix caching is usually done at the build output layer, e.g. the resulting binaries.
This is indeed very hard, but Nix's deep locking makes this a tractable problem: if a version of a package builds once, it will keep building for as long as the source code is available.
So in some ways, this Nix has an easier job compared to traditional distribution mechanisms. On the other hand, Nix can be sufficiently weird and unwieldy as to limit the number of potential contributors.
This works going backwards, too. If you need an older version of some software, Nix can happily install it for you in a way that won't conflict with other software on your system. The only requirement is that you can still get the original sources and you can get a copy of the nixpkgs repo at the time the older version of that software was initially committed.
> If apt could compile all its packages from source with an option to do so and a cache, it would be as reproducible.
This is simply wrong. There are many source-based package managers besides Nix: Portage, Sorcery, Pkgsrc, Ravenports, MacPorts, FreeBSD Ports, etc. Some of them, like Homebrew for example, even support binary caching. None of them have the same kind of reproducibility guarantees as Nix offers, except for Guix.
The counterfactual about apt doesn't even really make sense, as using its own special source archive format, having binary repos, and performing install time dependency resolution based on version constraints are all pretty essential to what makes it apt. But even so, the mere fact that it installs to global, shared locations, e.g., under `/usr`, means that there are circumstances where you can't guarantee that a package will always load the same libraries it was built against, namely in the case of transitive dependencies which had different versions at build time.
Guix, a similar project to Nix, has been tackling this head on by automatically falling back to the Software Heritage archive when the original upstream is unavailable: https://guix.gnu.org/en/blog/2019/connecting-reproducible-de...
i.e. if a package depends on the systemd package https://search.nixos.org/packages?channel=unstable&show=syst... , Nix will not automatically find a replacement to run the package on Mac. But it may be possible to manually work around this with https://github.com/LnL7/nix-darwin
More on building Docker images with Nix: https://nix.dev/tutorials/building-and-running-docker-images
Nix runs on Windows exactly like Docker runs on Windows— only inside a Linux VM. If you ship a Linux VM on Mac or Windows like people usually do for Docker, you're free to run the Linux version of a Nix package on those platforms.
> i.e. if a package depends on the systemd package https://search.nixos.org/packages?channel=unstable&show=syst... , Nix will not automatically find a replacement to run the package on Mac. But it may be possible to manually work around this with https://github.com/LnL7/nix-darwin
Nix-Darwin doesn't do anything for packaging issues, it just offers an alternative module system for declaratively managing services and configurations in a NixOS-like way.
Docker containers also don't normally do any kind of service management, they're single-process images. They're not really comparable to orchestrated services managed by NixOS or Nix-Darwin modules. But if your concern is just shipping the same thing and you don't care about what's managing the services, a whole NixOS VM isn't any less efficient than some other VM in which you run `docker-compose` or Kubernetes.
That said, there are Nix-y ways to ship one or more supervised processes in a way that's portable across the platforms that Nix supports. One way is with a manually tailored supervisord config with a nix-shell or something like devenv.sh. Another strategy would be to leverage something an abstraction layer like this one: https://github.com/svanderburg/nix-processmgmt
The advantage of Nix is similar thought: The first green build of any revision is cached and kept as "that revisions build" forever. Similar like a container.
- Build caching in containers treats the build process as a list of steps: first you do step 1, then step 2, etc. (these correspond to the *layers) of a container). If step 1 changes, you have to re-do step 2, even if it's unrelated; Containers don't have enough context about the commands they are running to know if step 2 depends on step 1.
- Build caching in nix is more like a directed acyclic graph: nix understands the steps that depend on other steps, and when something is changed, nix only has to re-run the steps that depended on that step, and the things that depend on them, etc.
In what way are they slow?
Also reproducibility; it can be achieved with containers if you save the artifact (the container image), but that's not what people do in practice, they save only the recipe (the Dockerfile), and if you execute it tomorrow, it will produce a different result than today, and it will likely not even run one year from now (due to e.g. third-party apt repos changing their url, signing keys expiring, curl|bash installers that are no longer hosted, etc.). With Nix, every run will produce the same results, and sources for everything packaged in Nixpkgs are saved by nixos.org.
Can't you just mount a volume from your host machine? Then you can use your regular editor, and just run commands from inside the container.
Any way to hack this? Just pretend it's a different date, the pointers to the latest package will changed to reflect that.
Or somehow record the exact versions when you first build the container, and freeze that.
For just the problem of "I want to be able to distribute the same application everywhere", container images solve this well. This can also be solved using Nix instead of container images; or Nix can even be used to build the container images.
The Nix expression language is much more expressive/powerful/elegant than Dockerfile's syntax is. -- Which can be useful for stuff like declaring "I want <program> built with different build flags / patches".
Nix ends up being useful for scratching the itch of "put in effort now, to save effort later".
- Nix is very good at cross platform support. A single entry-point creates environments on both MacOS and Linux.
- Docker containers run slower on MacOS because the virtualization overhead.
- Docker images can provide a reproducible environment but the images themselves aren’t reproducible.
Case in point for my current employer's Python shop - everyone runs PyCharm. Well, JetBrains doesn't really support Nix-based environments. See e.g. https://youtrack.jetbrains.com/issue/PY-42461 . So basically something like this would be DOA. Is this something that someone like @domenkozar can fix? @grhmc ? I don't know.
Setup was easier than Arch and I had more trust in the stability of the system because there's no state/config that I wasn't aware of (albeit after a learning curve).
The biggest issue though is how rigid it is. The "nix way" spreads like a plague into everything and I often felt like I couldn't do basic tasks such as install some node modules without having to relearn how to do it in nix.
I told myself that FHS was a good enough fallback, but that just wasn't the case. Despite using FHS for node development, I still encountered nix-specific issues that needed a nix-specific fix (i.e. using prisma binaries).
I still believe that the Nix way is the right approach, it's just going to take more time to mature out the Nix ecosystem's integrations with various language and package ecosystems to make it more dependable for developers.
I use Python with a few key native packages that are a pain to build myself (NumPy, GDAL primarily). Should I be considering Nix instead of MacPorts?
I use Homebrew for casks and the few CLI tools I need for all my projects (e.g. the Github CLI tool) and I use Nix for CLI tools I need to work on a specific repository using specific versions (e.g. node).
Massive doubt.jpeg, as all my previous attempts at understanding and using Nix have all been "trust me bro, THIS random github repo source code has the Current Best Practice (already outdated)" and various other random two article blogs on how great nix is.
About reproducibility, unless nix promises to fix all upstreams (apt, pypi ??), I don't see how it can fix reproducibility on the client side only.
Some differences are incidental (e.g. developers using different operating systems) and Docker can help reduce those, for sure. By some differences are essential and desirable (e.g. you don't deploy your compiler).
I'm starting to think it's best to first reduce the number of differences that matter. E.g. if you're using an interpreteded language, reduce or isolate native dependencies so that most code can just depend on the right runtime version.
Concrete example, dev environment with nix but production is Dockerfile with apt getting packages?
Thoughts?
[1] https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerToo...
> Thoughts?
Canonical answer is to use Nix to generate the image that goes out to prod as well, based on the same versions used in your devenv.
> Tangram takes a lot of inspiration from nix, and could be described as a mix between nix and bazel where you write JS/TS
but I don't know what project among these GP was specifically referring to, perhaps it's not public at the moment
I like to idea of having this in every repository in our codebase. Would make bootstrapping easier for new developers. However, you often want a specific golang or python version. Once you update some tool or language everybody gets the new env. That would be neat. Is that possible somehow?
I guess initially we would roll this out just for dev and keep Dockerfiles how they are but eventually we could then use it as builder in docker.
Bonus question: Can I use devenv/nix together with Bazel? We use that in quite a few newer projects and it also suffers from the local dev env issue.
Versions are locked using devenv.lock and everyone gets the same version.
I'm staying away from these Nix wrappers and learning how to do some basic nix-shell work, myself. Until it's proven the wrappers are adding real value, not just trying to hide scary scary Nix from devs who only feel safe in YAML files.
For the curious, this post[2] describes mixing Nix and Docker. And this[3] is a pretty decent tutorial on using Nix itself for this use case (consistent developer environments). Without going full NixOS, setting up a shell.nix with some OS-level environmental dependencies is a great way to dabble.
It seems there may be a sensible reaction away from "Dockerise all the things" for development at least, e.g. this blog post[4]. There are real downsides to containers for development, even though using things like `rvm`, `nvm` and `venv` were also a pain.
We've run into issues ourselves with developers starting to use the new M1 Macs - we've still not figured out how to make filesystem mounting fast yet. There seem to be other advantages to running more things locally, like simpler configuration of debuggers etc. (EDIT for clarity: the issues were with Docker, not Nix, hence the reference to filesystem mounting. Nix == "running more things locally".)
But taking a broader view, while containerising apps with complicated dependencies is a win, we really don't need to pretend that production and dev are actually identical environments. For example, only one of these environments needs a compiler. Once you acknowledge that the developer environment shouldn't be fully identical to prod, then you can start to think about what things do need to be the same, what can differ, and what your actual requirements are.
[1]: https://www.jetpack.io/devbox/
[2]: https://ghedam.at/15502/speedy-development-environments-with...
[3]: https://nix.dev/tutorials/declarative-and-reproducible-devel...
[4]: https://blog.testdouble.com/posts/2020-02-11-the-slippery-sl...
Oh and I managed to brick nix on my mac, for some reason I can’t even reinstall it. Need to spend more time debugging that.
Didn’t play nice with our Rust dependencies as well I found. At least it seemed unnecessarily complicated to set it up with an already existing Rust project.
It'll really be wonderful once Nix gains some popularity and some of these kinks get ironed out.
Highly recommended.
Also when trying to adopt nix for the enterprise it is still a tough barrier. At least for me struggling how to package something like the synology active backup client nix or how to setup your cups printers in nixos.
Besides that I hope for nix to get secure boot support.
Unfortunately, Nix suffers the fate of Haskell: so powerful that the masses can't and don't use it. By contrast, Homebrew spreads like cancer. "Worse is better".
The current state of the software supply chain makes me very uncomfortable to install any packages.
Putting "developer environment" in the name of this tool perpetuates bad practices.
Any tool that constructs environments for applications should be general enough to handle both production and development.
Does this suggest a bit of unknowns in Nix and Nix OS?
The other way around that we build a dev environment that is different from prod just makes no sense.
I, at the very least, completely disagree with you on this specific point as this tool is using Nix.