Switching from pyenv, rbenv, goenv and nvm to asdf
jinyuz.dev
jinyuz.dev
For me, it has come so far that I don't have any interpreters or compilers in my main OS environment and even for one off REPL sessions, I'll use `nix shell nixpkgs#python3 -c python3`.
[0]: https://nixos.org/
Mostly I think Nix would be just as hard if packages were expressed in any language.
Stuff like "this program compiled by nix is using a different glibc than this library" is going to be confusing regardless of whether Nix expressions can be written with JavaScript; or stuff like some program assumes it can write to $HOME in its build script not playing nicely with Nix.
A lot of why Nix is hard is because it's weird, and that even as an end user when things go wrong it may require understanding more about what's going on than what other solutions require.
(Tongue-in-cheek, but I'd actually like to see this happen: Nix is a fantastic concept obscured by space-alien syntax.)
https://lists.gnu.org/archive/html/guix-devel/2022-03/msg001...
As for Windows support: It seems to be possible to use nix in WSL [0] but I've never tried that. I have used nix alongside apt and pacman on Ubuntu/Arch before I decided to go all the way and install the NixOS distribution. Using it alongside other package managers worked really well.
I've used Nix on Linux (i686, x86_64 and ARM), macOS and Windows (WSL2). I believe it also works (although not officially supported) on BSDs, etc. too, although I've not tried that yet.
> Lorri sounds amazing though! ...which handles the "this directory needs these specific packages" installed locally, not globally problem that Debian gives you).
You're describing Nix, not Lorri. Specifically:
- nix-shell provides a directory-local environments (and more, e.g. #!nix-shell to fetch script dependencies automatically)
- direnv can enter/exit this nix-shell environment when cd-ing in/out of a directory
- lorri makes direnv+nix-shell faster, by building the environment asynchronously in the background (rather than causing our terminal to freeze)
The biggest drawback is installing Nix on OS X is a huge pain. Docker has lots of magical UX tooling to make it rather seamless in comparison.
Did it on an aarch64 mac the other day. One command (two?), 15 minutes, done.
I don't understand the concept/motivation behind it, everything that nix does should be able to be done in userland, like podman does, which I love.
The daemon is not powerful at all, almost all the logic happens in the client.
The client does the parsing, job ordering, and tells the daemon exactly what to do.
The daemon, only required in multiuser environments, does only two things really.
- ensure that the /nix/store is protected as the daemon should be the only process able to _write_ to it. Everything can read from the nix store; no nix required at all.
- execute build commands in a sandbox. Providing isolation between multiple builds.
I'm not sure if there are other reasons.
Be prepared to sink dozens and dozens of hours learning the language, it’s features, and switching everything over. Be prepared to get stuck when you are required to use an old version of a language and things don’t install correctly — and sometimes blocked by nix itself from installing it. Be prepared to get stuck if you are not in the happy path. Be prepared to get lost in the sometimes scant documentation.
I love the idea of nix, but it did not work for me except for the simplest cases.
I've found it to be quite flexible though. For example, here's a commit in which I apply a patch to a tool to solve a problem that the derivation hadn't taken into account (and absent a home-manager solution): https://github.com/samhh/dotfiles/commit/867dd3b4d4b3942a0aa...
I understand everything in that sentence except the word "so".
I really like Nix but in this particular way it feels like taking a big step backwards from the other languages I frequently use, particularly for a language in which you'll necessarily be constantly interacting with bespoke interfaces.
I guess knowing allowed types for each parameter is informative (more if very specific types are used rather than string or integer), but documentation usually specifies that anyway. And even knowing all the types isn't usually enough without documentation.
My point is just that I've used plenty of libraries in dynamically typed languages without needing to read the source. And conversely, I've occasionally needed to look at the source of a function in a staticly typed language, to answer some question not answered by the types and documentation.
Haskell is the ideal here, since it separates the type signature into its own separate line at the top of the function declaration. That confers readability and reduces cognitive overhead more than any other language I've used.
It also changes the programming thought process. You can pseudo-code an entire program just with function type definitions. Then test that the type defs compile without error, and go back and implement the function definitions. As long as the function definitions adhere to their type signature, the program almost always works (barring some I/O errors).
Not all statically typed languages are created equal. The ones fundamentally oriented around function type signatures, rather than variable types, are the ones I think parent had in mind in his comment.
Like you, I find Nix hard to use and the docs confusing (beyond trivial examples). The amount of time the Nix mafia spends proselyting could be better spent making better docs
I've never heard of asdf and popped in here to read about it, people's experiences with it, etc. If I want to read about Nix/OS I'll find a thread on that, or go to /r/nixos, etc.
And what are you doing :-)
Oh I think you're using HN wrong. To learn about the subject you can read the actual article that's linked to. No guarantees about the comments section.
But, since you're downloading & running code written by someone else, it seems weird to talk about that but ignore "what if this program I'm downloading does something malicious".
[1] https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...
The reality is that the docs are fine. The reference material is comprehensive, and there are some very good tutorials and introductory essays to get people started. The problem is that it's difficult. Nix is very, very different from what people already know, and it takes quite a bit of effort to learn how it works. Most people just aren't prepared to do that much work just to install some software.
Most pilot text books are absolutely horribly written, finding a good one takes significant effort. Reading just an ok book shines a light on how absolutely trash the majority of pilot text books are and that it is absolutely bad "documentation" that is partially to blame for it being difficult to learn.
Just like inherent complexity in some software or business problems.
Maybe airplanes wasn't the best thing to compare with -- the comment below compares with C instead, maybe that's better. I never flew a plane.
It helps that people who are into Nix most often use NixOS. NixOS is the only distro I've written packages for. It's not even particularly hard since there's a lot of inspiration to find in the official and community repos + people sharing personal configurations.
The UX wasn't the best but it's improving.
It remains extremely reliable which is not something that you could say of other systems.
Docker is more pragmatic unless you spend hundreds of hours on Nix, in my opinion.
then show me where `nix-env`'s `--arg` and `--argstr` is documented.
--arg name value
This option is accepted by nix-env, nix-instantiate and nix-build. When
evaluating Nix expressions, the expression evaluator will automatically try to
call functions that it encounters. [...]https://nixos.org/manual/nix/stable/command-ref/opt-common.h...
But it's no wonder that people keep commenting that Nix is an awesome idea that solves a lot of problems. That's because it's an awesome idea that solves a lot of problems (yeah, like Rust).
It is also completely undiscoverable (not completely unlike Rust), and that's why almost nobody uses it. But that's a matter of improving the documentation or maybe fixing one or two superficial problems on the language.
Based on a quick check of their Wikipedia page, Nix has been around for 15 years, since 2007.
So those for sure look like structural problems. I don't know which ones exactly, but paraphrasing Tolstoi, popular software products are popular in much the same way, while unpopular ones are unpopular in their own unique ways.
In my experience almost no open source project that hasn't entered the mainstream in its first 10 years manages to turn the ship around, unless it lucks into a major change of environment. Programming languages are sometimes exempt if they have a "killer" library or framework pop up.
HN tends to see success in terms of adoption rate & ranking. Nix is on another plane. It's beyond critical mass. And for people like me, it is a huge force multiplier. But I had to grind my way to the point where that is true.
I tend to not bother selling people on it. People who want what Nix does will find it at some point. So I just focus on using Nix myself in a world of worse tools. Feels great.
That said, I haven't found Nix hard to use, quite the opposite. The new 'nix' command is pretty self explanatory, it's just a little incomplete in spots. And the Nix language is quite simple and easy to understand if you have ever touched a functional language. Compared to all the other Linux stuff I have played with over the years, it was a very pleasant experience so far.
Evidence of that is that you can just talk about it by name here on HN without explaining what it is.
Is Haskell mainstream? What about committing murder?
And yes, this is 100% Nix's fault
I would say that HN is far from mainstream.
Nix provides a level of flexibility that other solutions simply do not offer. If a package in Nix doesn't fit your requirements for whatever reasons, you can create a modified version of a package with ease. For example, say that you need a version of Nginx built against a custom version of OpenSSL. You can do just that with a few lines of code:
let
mypatch = pkgs.fetchpatch {
url = "https://example.com/bugfix-for-openssl.patch";
sha256 = "...";
};
openssl = pkgs.openssl.overrideAttrs (old: {
# add build flags
configureFlags = old.configureFlags ++ [ "--enable-foo" ];
# add dependencies
buildInputs = old.buildInputs ++ [ pkgs.foo ];
# add patches
patches = old.patches ++ [ mypatch ];
});
in
nginx.override { inherit openssl; }
You can't do that with other common package managers. You're stuck with whatever the package manager provides you. So if you're not in the happy path, you're out of luck. You'd either have to give up or build from scratch. If you need a bugfix for a particular package, you'd have wait until the fix reaches the package repository. When I was using Ubuntu, that was often until the next major release. None of this stuff is a problem with Nix, which allows for customization with very little effort.
It's easy to learn about Nix if you know where to look for. Nix Pills [1] would be a good start. The core language is the easiest part. It's JSON, but with functions and variables for proper abstraction. The documentation, while imperfect, is quite extensive compared to a majority of other high profile open source projects. If the documentation fails you, the Nixpkgs repository [2] is an even more rich source of information. The code is well organized, and I was able to get familiar with writing Nix packages fairly quickly by grepping the codebase. And finally, Nix has a sizable community so you could always ask if you're stuck.
I've been using nix as my daily OS for six months and I'm able to be productive. At the same time, I'm often confused as to whether I should use nix-shell, nix-env or modify my configuration files.
Thanks for the great comment.
I understand you are saying it with the best intentions but it might sound like not the most correct way of defending a tool being criticized for its lack of documentation or easy of use.
Ultimately documentation doesn’t solve usability problems it just makes note of them.
Also rollbacks in nixos are amazing.
On the other hand newly packaging something can be really daunting.
Nix provides better abstractions and composability to make that experience far less painful. Note that in my previous example, I didn't need to specify how to build OpenSSL or Nginx. I was able to reuse the definition from Nixpkgs and apply my own customizations to it.
That's less of a guarantee with a collection of shell scripts.
Here's the simplest example of an utter failure when I tried nix: installing ruby 2.2. It's very possible I did something totally wrong, but when I have to use ruby 2.2 and nix cannot do it, it sort of kills your awesome example of being able to override openssl.
Another extremely frustrating experience was the bifurcation between nix and nixos. Clearly one experience is preferred over the other. So many blogs and docs talk about things like HomeManager and other configurations that are nixos specific.
On top of nixos/nix, there's also shell.nix/flakes. There's just too much development in nix right now to make sense of what's going on.
Considering that ruby 2.2 was released in 2014, I can't say I'm surprised. It's too much effort to continue maintaining something so old. But if you really want to use it, you could try using the package definition from an older commit of Nixpkgs:
let
nixpkgs1709 = pkgs.fetchFromGitHub {
owner = "NixOS";
repo = "nixpkgs";
rev = "e09c0adc63d10249dac8f90313f91e1050861d3c";
sha256 = "sha256-Di9D0gvaESV3JmX/kW2uEJ68QDlAke23t19bImTXsJ8=";
};
pinnedPkgs = import nixpkgs1709 { };
in
pinnedPkgs.ruby_2_2
I'm not sure what you mean. Nix is the package manager while NixOS is an entire distro based on Nix the package manager and its configuration language. They have clear separation of concerns. Keeping it that way has actual benefits too, namely portability. It allows you to use Nix on a wide range of platforms including non-NixOS Linux distros, macOS, and even BSDs to some extent.
> On top of nixos/nix, there's also shell.nix/flakes.
Flakes are currently an experimental feature and not meant for wide use yet. So if you're just getting started with Nix, I'd recommend looking into it later until you're sufficiently comfortable with Nix.
They're not fundamentally different, though. Flakes formalize the conventions for writing Nix expressions to make them more reusable.
I also feel like nix is immediately tangible and beneficial in a way that makes us want to get to the power user level, but I don't feel like wanting to learn, say, kubernetes means one automatically wants to learn to be a cluster operator. The scope and complexity are great, and I have experienced all of the things you have described as well. Now that I am fairly confident in my abilities, I think it's important to make sure we find ways to help everyone get the most out of it, even if it doesn't involve scaling the wall that is the learning curve.
Can you clarity what you mean?
What do you mean by "trap"? Which (if any) of the following concepts factor into your understanding? Broken expectations? Premature optimization? Over-engineering? Lock-in?
Are you suggesting that many users of the Nix package manager "fall into a trap" and begin using NixOS?
What do surveys and/or data show about usage of just the Nix package manager? Combined with NixOS?
The Nix ecosystem is attractive to me at a high level because it looks like it has the possibility to revolutionize a lot of the problems that come up with an increased focus on devops culture. The problem with the Nix ecosystem itself is that it is larger and more varied than it might appear.
There are really three large entities in the ecosystem: nix the language, nix the package manager (nixpkgs), and nix the OS (NixOS). Despite the fact that there is a lot of overlap in some of the details of these three things, they are easily understood as separate entities. There are a bunch of smaller secondary entities such as nix-darwin, home-manager, devshell, and digga (formerly devos). The aim of these entities is usually to supplement, enhance, replace, or reproduce features of nixpkgs and/or NixOS.
Each of these entities has its own learning curve. It takes surprisingly little nix knowledge to use it as a dev environment generator, but nixpkgs knowledge will help a bit more. Even within nixpkgs itself, each major language has a large ecosystem built up around facilitating package building and deployment, and these are extremely heterogeneous. Some of them require that you regenerate the entire matrix of package dependencies for the package manager (javascript, haskell). Some of them have tooling that allow you to get started with almost no effort (mach-nix for python). These all have their own peculiarities. NixOS has a ton of modules, all rather heterogeneous, on top of requiring more careful knowledge of how to put together a system than most other distros. And nix the language without nixpkgs is very small, and some of what it does have built in is necessary so it can bootstrap an actual stdlib from nixpkgs.
The trap is sprung once you've been sold on the idea, and even seen it shine through some things, but feel like you have an entire wall to climb to get to a point of general competency. All of these things I've described, and even some of their subgroups, require new knowledge for a more complete understanding. I do not understand the details of many of the things I have described, and I would consider myself a power user at this point. What looks like one gigantic, insurmountable cliff is really dozens of much smaller cliffs, many of which are not necessary, at least not to most people.
At one point I wanted to test my skills in a really generic way, so I helped fix broken packages on macOS during one of the two yearly regression periods where the community tries to get the CI builds passing. I picked the packages completely at random. Most of what I ended up fixing was C/C++ based, and I don't normally work in it, or I haven't for many years, but I feel like I learned a good bit about cmake. I don't really write Haskell, but I was able to fix a bug in a solving library having to do with a missing portability shim. I didn't have help for any of this, and in some cases I was either too stuck to fix the build or unable to fix it due to problems beyond my control.
All of this is to say that I think getting to this point is a lot less painful if one takes a use case, preferably a narrow one, and focuses on learning whatever is necessary in the service of that use case. Some good topics are dev environments, using nix as an app/project builder, using nix to build containers, using nix to manage remote machines (the difference between these and personal machines is that the scope of these machines tends to be much smaller). I think people try to take on too many of these things simultaneously and end up overwhelmed. That's not to say that NixOS isn't worth learning, but I think many people could realize most or all of the real-world benefit if they use nix on an OS they're comfortable with. Coming back to my comparison of being dropped in the Romanian countryside, if you want to learn Romanian, start there. If you want to experience the countryside, maybe a guided tour is the way to go. If you want to learn about the life of Ligeti, you might not even need to learn Romanian! But total immersion is difficult and draining, and a failed immersion experience doesn't mean Romania is a bad place. So it is with Nix.
package manager (whether on Darwin or any linux distro), to have some tools globally available:
nix-env -iA nixpkgs.${some-package}
{
pkgs ? import <nixpkgs> {},
}:
let
some_var = some_value;
in pkgs.mkShell {
buildInputs = [
pkgs.some_package
...
];
shellHook = ''
# this is bash, so, whatever floats your boat
'''
{
pkgs ? import <nixpkgs> {},
}:
let
some_package = pkgs.some_package_1_2;
in pkgs.mkShell {
buildInputs = [
some_package
pkgs.some_other_package
...
];
shellHook = ''
export SOME_VAR="some_value"
...
'';
}
{
pkgs ? import <nixpkgs> {},
}:
let
ruby = pkgs.ruby_2_7;
python = pkgs.python27;
node = pkgs.nodejs-14_x;
in pkgs.mkShell {
buildInputs = [
ruby
pkgs.sqlite
python
node
pkgs.nodePackages.yarn
];
shellHook = ''
export RUBY_VERSION="$(ruby -e 'puts RUBY_VERSION.gsub(/\d+$/, "0")')"
export GEM_HOME="$(pwd)/vendor/bundle/ruby/$RUBY_VERSION"
export BUNDLE_PATH="$(pwd)/vendor/bundle"
export PATH="$GEM_HOME/bin:$PATH"
'';
}
{
pkgs ? import <nixpkgs> {},
}:
let
python_packages = python-packages: [
python-packages.pip
];
python = pkgs.python38.withPackages python_packages;
in pkgs.mkShell {
buildInputs = [
python
];
shellHook = ''
export PYTHON_VERSION="$(python -c 'import platform; import re; print(re.sub(r"\.\d+$", "", platform.python_version()))')"
export PIP_PREFIX="$PWD/vendor/python/$PYTHON_VERSION/packages"
export PYTHONPATH="$PIP_PREFIX/lib/python$PYTHON_VERSION/site-packages:$PYTHONPATH"
unset SOURCE_DATE_EPOCH
export PATH="$PIP_PREFIX/bin:$PATH"
'';
}
{
x86_64 ? import <nixpkgs> { localSystem = "aarch64-darwin"; },
aarch64 ? import <nixpkgs> { localSystem = "x86_64-darwin"; }
}:
let
foo = aarch64.foo;
in aarch64.mkShell { # this makes nix-shell drop to an arm64 shell, change it to x86_64 to be intel/Rosetta2
buildInputs = [
foo
x86_64.bar
aarch64.baz
];
}
{
stable ? import <nixpkgs> {},
unstable ? import (fetchTarball http://nixos.org/channels/nixos-unstable/nixexprs.tar.xz) {},
pinned ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/ca2ba44cab47767c8127d1c8633e2b581644eb8f.tar.gz) {},
git ? import (fetchGit { url = "https://github.com/nixos/nixpkgs/"; ref = "refs/heads/nixos-unstable"; rev = "ca2ba44cab47767c8127d1c8633e2b581644eb8f"; }) {},
}:
let
foo = stable.foo;
in stable.mkShell {
buildInputs = [
foo
unstable.bar
pinned.baz
git.qux
];
}
{
pkgs ? import <nixpkgs> {},
}:
let
clang = pkgs.clang_12
in pkgs.llvmPackages_12.stdenv.mkDerivation {
buildInputs = [
clang
];
}
Thanks to a chatty Romanian taxi driver I had earlier today, I understand this.
Really, I like the idea and I actually like the fact that they use Guile (scheme) instead of a home-built DSL, but guix is a horrible, poorly documented, doesn't work even on the happy path mess.
But maybe I'm just a "GNU zealot".
I didn't mean to imply everyone who uses it is a GNU zealot, but there's definitely a subset who are.
"possible future efficiency gains".
With this in mind, the question shifts from rather vague talk about possible futures to various kinds of scenarios.
What happens when software developers have a deterministic build system? On the whole, the benefits are significant. Sure, there are costs to get there. In my view, the benefits often outweigh the costs.
I keep seeing nix mentioned alongside complaints about it's maturity but that seems par for the course for something new, doesn't it? Until a tool reaches critical mass sharp edges are to be expected.
I'll look into nix again when I have an appropriate use case.
More broadly, even if it isn't as popular, Nix is a viable tool and also encourages us (developers) to seek out repeatability in our dependencies.
I disagree with the whole "highly likely" bit. Just because something new has a large positive value, that does not mean that a large time-investment or effort-investment in it is likely to pay off.
After all, it may never get traction to make the cost of learning it worthwhile - some other, newer, better, easier system could come along and be adopted instead.
We've seen this play out again and again in the technology world[1]. In 2000, a large time and effort investment into UTF-16 would be wasted as the world adopted UTF-8 instead. A large time and effort invested into new languages with powerful features (Haskell, circa 2005) would be wasted as the industry largely ignored it in favour of (presumably) inferior alternatives.
Is nix highly likely to provide solutions to a whole class of problems?
Sure.
Is it highly likely to be the dominant, or even a common tool, in the future that solves those problems?
Doubtful; "easier-to-use" has always won against "can better solve problems". I see no evidence that this will change anytime soon.
[1] I've found that by procrastinating about learning a new technology, I can sometimes manage to avoid having to learn it altogether, because the tech dies off before my procrastination does. By putting off learning VRML in 2001, it went away before I ever got to it.
Likewise, I'll put off learning new things until I actually have a need for the solution they offer. At least then, even if they die out in a few years, I'd have at least solved my immediate problems.
You are disagreeing with something I didn't write.
The way you've framed the question is notable. The argument structure ignores a common pattern in software: tools steal ideas from each other often.
For example, MySQL is very different than its 2002 version. There are more database back ends with various guarantees and performance tradeoffs.
Your choice of language: saying 'need' suggests a black and white decision. See what I mean?
Costs and benefits often cannot be estimated accurately 'in one go'. It is common to spend some time and money to increase understanding and thus reduce uncertainty.
What deterministic build systems have you used? If you have not, how confident should you be that you can accurately assess their costs and benefits?
PS. Your comments suggest to me that, like most people with experience, you have become overconfident. To disabuse yourself of this, I suggest you find a prediction market and make some bets. You'll find you have to make your claims testable. Make a specific prediction/bet on Nix and let's see what happens. Let us know how it goes.
I recently gave nix a try because of a hacker news comment and agree with most of the points in this comment. The fact that new users are encouraged to read not 1, but 3 separate manuals to grok the tool is extremely discouraging for new users. Also the fact that nix is currently transitioning to a new feature called flakes and deprecating channels signals to me that the ecosystem is not stable enough for me to recommend that my team at work use it right now.
That being said, I intend to read at least 2 of the manuals and use nix for dependency management in my side projects and to manage my system dependencies. I think nix takes all the great things that asdf can do, to a whole new level. The dream of having a nix section in my projects 'Getting Started' that only has 2 steps (1. Install nix 2. Run `nix-shell`) is something I desperately want.
I think nix is much like erlang in that it's incredibly powerful and solves a lot of common problems, but it's used and maintained by an old guard that went through the trouble of learning how to use it and maintains the "I did it, why can't you" mindset. The erlang ecosystem has been given new life and been made easily available to new users thanks to the awesome work done by Jose Valim and the rest of the elixir team and community. However, I really hope that it doesn't take 30 years and a new language built on top of nix for it to become accessible to everyone.
For those working on nix and the new flake functionality please look to things like elixir and asdf for inspiration on documentation, ease of use and pragmatism. Help make the tool you love ubiquitous. It will ultimately make your life easier when working with new developers because they will already have nix installed.
I intended to do everything I can to help push nix in this direction (once I understand how the hell it works) if anyone reading this, has some time on their hands and wants to improve the state of software development for everyone please go down the nix rabbit hole and try to find ways to contribute to making it more accessible.
Look at Python with Nix: https://nixos.wiki/wiki/Python And then consider the simplicity of `pyenv install x.x.x` and a requirements.txt.
- you want to use lang-x.y.z, so you'll just reference an old commit
- turns out that version was never packaged, so you copy x.y.z-1 to your flake/overlay
- you end up patching the build because it doesn't work by default with new environments
- on the other hand a module you're using needs the new environment so you do your best to make everything happy - more overlaying
- because you changed something crucial in a few steps, you're compiling llvm 20 times through this process and now it's 1am
Compare that to putting "rust 1.59" in your .tool-version and running "asdf install". Guess who tried to use 3 specific versions of rust at the same time recently and got very familiar with the process...
(That's also ignoring the issue like Ruby being packaged without the platform identifier in nix which makes it unusable for some development purposes)
It should never be Nix's business (or any other OS) to provide elements of a project's toolchain. Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
Why? I used to think the same way, but that's because of the scar tissue of all the tooling being terrible at accepting that build concerns and packaging concerns are one and the same.
Nix proves it and makes this ancient pain a non-issue: "My package depends on x, y and z, additionally, building it requires a, b, and c, please also make build dependencies available in my shell while I write code too". That's it.
> Running a project should never require more than "git clone $project && $build-cmd".
That's precisely what Nix allows you to achieve. Meanwhile doing this:
> Reciprocally, projects should go out of their way to not depend on OS installed toolchain components.
Means that you can never depend on anything you're not vendoring (read: maintaining a copy of, on your own). Or it means sacrificing productivity in the altar of shitty tools. Nix allows you to have your cake and eat it too.
It is also not a package manager like "apt", "dnf" or "pacman" as it does not fill the same niche. It does not even have to fill the same niche on every project.
As an example: at my job we're using it exclusively for optionally managing shared developer dependencies. Everyone is free to manage their own dependencies if they don't want or can't use Nix. But a flake.nix is there so you can get everything configured for you automatically.
In a personal shared project of mine I use it for tracking developer and build dependencies, meaning that I can build the project through Nix without having any of the dependencies specific to this project installed globally on my machine, meaning I can work on other projects that use incompatible versions of the same library, and keep the project reproducible.
On my desktop and laptop I use it as part of NixOS (which is an OS), to configure both from a single, mostly-shared configuration I keep in Git, replacing a mountain of terrible Ansible and shell scripts that did not work even nearly as well.
You can also use it to generate Docker images, operating system ISO images, distribute your app as an AppImage, Snaps,... and many other things. And you get an actual programming language to do all that with all the tooling the community maintains for each usecase.
If you're just using it as an asdf replacement, as the comment you originally replied to compared it to, your project will not be any more locked in to Nix than it would be to asdf.
"Nix shouldn't provide compiler toolchain" is not.
Nix's whole design is oriented around explicit description of inputs in programmatic descriptions of package.
asdf does not allow you to keep three different versions of the same language, so I'm not sure how that compares? It's not super-trivial to do in Nix, but at least you can do it.
asdf is also no different than Nix when it comes to minor/major versions. You're at the mercy of what the plugin does, other than that you have to create your own plugin from scratch or make a fork. Nix has the option to patch things up more easily at least.
Sure it does. You can have any number installed and either switch by changing the tools version file, setting it for your shell, or referencing the right path directly.
> You're at the mercy of what the plugin does
Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
Oh, that's cheating :) Nix can do the same thing! Derivations are just kept in the store until you garbage collect them. You can also configure multiple versions at the same time to present you with multiple binaries, which is what I was referring to.
> Again theory vs practice. In theory, sure, it depends on the plugin. In practice, almost every plugin lists every single released version to choose from. Which is much different from nix where the official choices are limited and you have to work to enable the rest.
In practice, nixpkgs supports every maintained version of most tools. Certainly far more tools than asdf can ever hope to maintain. If not, there's an overlay or flake out there to support it. Can you come out with specific examples and a usecase?
As others pointed, the idea behind nix is cool, but it sometimes fall short, hope it get ironed all the issues in macOS so I could give a try in the future.
nix-shell -p bundlerAre there any other advantages to using nix?
The advantages are more pronounced for DevOps than for developers.
e.g. for DevOps, Nix allows copying a package of software (and all its dependencies) from one computer to another, with confidence that the programs will work on both machines.
For developers, Nix is more appealing if you ever work on more than one computer. e.g. with Nix, it's easy to ensure you've got the same version of packages installed regardless of whether you're on macOS or Linux.
It's pretty neat for side projects, in order to get "I was able to build it now, I'll be able to build it in 6 months". Whereas, if you're otherwise relying on Ubuntu's system packages, maybe the package versions got updated, and so you'll need to spend time to get your program to work with the latest compiler/libraries.
For $REASONS I want python 3.7.10, and the full scientific stack (numpy, scipy, pandas, numba, matplotlib, ...) at the latest compatible versions along with pytorch. I don't mind waiting for things to compile. How, if at all, can Nix help me get into an environment that meets those constraints? Can it help without requiring me to find and copy hashes of things?
Put an incorrect hash in. Run the build. The error will tell you what hash it actually got. Copy that. Run build again.
This makes me wonder: does Nix use libsolv, or a similar idea?
Roughly, Nix's approach is to describe packages as a function of it dependencies as inputs. (And all the inputs have all their dependencies described in Nix, too). Nix doesn't do things like "install latest version of X". (One of Nix's principles is build reproducibility; the package should have the same behaviour if built with the same input).
Tools like mach-nix or poetry2nix wrap around nix.. as I understand, they adapt the requirements.txt or whatever and pass it into Nix in a way that Nix can understand.
Both for parent's quandary, and for when you're working with a group and don't want to force everyone else to use Nix.
There's a reason why, despite being almost 20 years old, Nix has at best a small cult following. It just doesn't solve the problems that most people are looking to solve, most other solutions are simply easier and in many ways better
I spent time understanding it, and getting it working, but it really doesn't work just universally. Once the install process becomes more like, say, miniconda -- where an unprivileged user can install it locally with ease -- nix will get a massive boost in my books. Until then, I can only yearn.
It's astounding because the local version is doing a naive, case-sensitive regex search, which is pretty bad when you're looking for, say, the Perl package MIME::Lite which can have various junctions and capitalizations. In fact, that package has a listed name of perl5.34.0-MIME-Lite but is installable as nixpkgs.perl534Packages.MIMELite!? I can't even
I found the code a few months back (but can't be bothered to look it up just for a comment). I didn't fix it then because I had other priorities than learning how to modify my local copy of nix to test it.
Could it solve the issues we're seeing lately with NPM?
But, I would encourage people to try out NixOS. Its much better than the package manager, yet it follows the same basic principles.
- Uses a simple DSL language
- 1 config file describes your entire environment, you basically use nix the language to tell your computer if you want wifi enabled or not, or if you want sound, or if you want i3 etc.
- Its easy to roll back, since the config file is versioned (like git)
I personally use a mac machine, running nixos in vmware fusion and I can't believe how long I had been using a separate machine for development and another for personal use.
I get the best of both worlds with this setup; my messages and notes are here, I get world class hardware (come at me bro), and I get to control every single aspect of my development machine.
I still use docker inside nixos, and yes I know its almost supposed to replace docker and all but I will get there one day, right now is not it.
Areas for improvement: - They should have come up with a better name: nix, Nix and NixOS are confusing (language, package manager and operating system - in that order. Yes, they went with an upper case letter for the pkg manager) - Docs NEED to be better, this is not an easy task. The language is shared between every component. - Give playgrounds to beginners, I shouldn't have to download the minimal iso and boot it on my machine to get a sense of how powerful it is. There has to be an easier way.
End of my rant. I started using it a month ago, I have run into issues with dependency pinning but I remain optimistic.
like "nixpkgs" is almost always the default, from what I can tell, so it could be omitted and simply be assumed to be the default, and not sure what the -c param does but perhaps it could also assume that the specified package(s) are also the ones to install/choose, in which case the whole thing could be like `nix shell -c python3`?
> Hermit is probably most similar to asdf, but their goals differ. Hermit's goal is to make isolated cross-platform tooling consistent, self-bootstrapping, and reproducible at the project level. asdf's primary goal is to allow developers to install and switch between multiple versions of languages and tooling.
The downside for the author of this post, though, is that Hermit doesn't yet support Ruby because it's not static/relocatable.
[1]: https://cashapp.github.io/hermit/ [2]: https://cashapp.github.io/hermit/faq/#asdf
Be careful if you’re also interested in Hermit. These kinds of things bit you up way down the road when you least expect them to.
I'm quite happy with direnv to avoid this problem.
Funny enough, it was submitted to HN most recently just yesterday:
https://news.ycombinator.com/item?id=30908321
direnv is complementary to the various language-version tools, so it obviates needing asdf:
https://github.com/direnv/direnv/wiki#project-layouts
But it seems some folks combine the two:
https://blog.mikecordell.com/2021/12/18/better-project-envir...
You are giving people a _ton_ of trust when cloning a repo or pull request that has a .envrc which will be automatically executed by direnv. This file could do anything from steal your SSH keys to delete your entire home directory or worse. And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops. It's just begging for a supply chain disaster on a similar scale as npm's infamous issues.
VS code and some vim extensions have similar issues with automatic project-specific configurations that allow arbitrary code execution too, and neither have a good solution beyond 'just read all the code and understand it to make sure it's not going to hurt you'. It's not really a unique thing to direnv.
We really need a better model for collaboration and consistent dev environments. I think containers and docker/chroot/etc. environments with explicit opt in of your local resources, configs, files, etc. are a much more sane and security conscious path towards it. A script executing to setup my python environment for a project should never, ever have access or even know about my personal password store database for example (unless I explicitly decide it should).
> And the only thing preventing that from happening is the suggestion of "oh be sure to eyeball the .envrc file and make sure it looks ok, then run this command to enable it". People will just google for the first "how to I unblock .envrc file" and blindly run the command, then oops. Or a determined attacker will just obfuscate what they're doing so the .envrc looks fine but you failed to realize some esoteric bash-isms were actually invoking an obfuscated script and... oops.
You need to be absolutely sure you've read and understand every single line in the .envrc file, and every single line in any script it may be invoking, before you run direnv allow. Otherwise you are giving code from the internet a blank check to do anything on your machine with your user account.
I personally do not think bash/shell is a sensible language to trust that you can quickly or easily determine from a visual check alone if a script might be doing nefarious things.
Sure, you can trick idiots into running malicious code, but I don't think you need direnv for that - any pip/npm install can run anything and so does curl|bash.
Seems safe to me
If I wanted to include it, I use something like git-crypt and encrypt the contents
Personally that's not something I can live with as these binaries are sometimes launched many times in subprocesses.
They just released a major new version (asdf-direnv), so might be worth giving that a shot and see if it works for you with asdf?
To be clear since I think it might not be obvious: lapser is talking about asdf-direnv, not direnv by itself. See https://github.com/asdf-community/asdf-direnv
IIRC the main reason our team started tracking go versions was we were already using `asdf` for tracking other tools and it’s a nice way to ensure that everyone/everything was using the same to version (e.g. for unit tests which usually aren’t run in a container) but it is another spot to keep in sync with the Dockerfile.
- https://github.com/asdf-vm/asdf/issues/383 - https://github.com/asdf-vm/asdf/issues/547
They did do something to ameliate this though: now they refer to the proect as asdf-vm as much as possible, to distinguish it from Common Lisp asdf.
I've even tried doing that with all the devtools we have. Like normally a developer here has to install gcloud, terraform, kubectl, apt-get lots of stuff, and then 3-4 internal tools. Then configure all of it, and of course some stuff will just not work on someone's computer. But creating a docker image including all that means that I can grab a new computer and be up and running in minutes.
"Setting up a Python environment" takes about 10 minutes and it's dead nuts reliable.
It is a very effective package manager.
I did some googling, but I’m unfortunately not experienced enough to be able to describe the problem succinctly to google.
Using pylance, btw.
TIA!
This works in VSCode over SSH perfectly fine - the locally running VSCode is able to detect and select Python environments in the remote server.
Although, now you’ve got me thinking what would happen if I mounted the interpreter inside the container to something local that I can point to? Might try this later, thanks for the help!
Conversely, the experience of using Ruby within Docker for development is often clunky and awkward by comparison.
I don’t see why this is so hard to avoid with per-project version pinning. I’ve never had this problem with Java.
This seems like a lot of effort and complexity to work around global package installation. Just stop assuming that packages are global and this would all go away. How do we know that? Because other languages don’t require this kind of workaround!
I'll agree that how to do things properly with Ruby isn't obvious when you're new to the language - just as it isn't obvious how to do things in Java (or other JVM languages) when you're coming from Ruby or another language.[0]
But Bundler doesn't work all that differently from Maven, in the end. Instead of running your commands through `./mwnw`, you run them through `bundle exec`[1]. Dependency management is straightforward and mostly hassle-free (the problems arise "only" when you use C extensions which, to be fair, is not that rare, but I don't really see a way around this in a language where you need C for certain fast things). In particular, Bundler by default[2] also stores dependencies centrally like maven to avoid duplication and then selects the appropriate version of a given dependency (unlike e.g. NPM which stores everything locally).
Rbenv (or alternative tools such as chruby or asdf) solves a problem completely distinct from Bundler or Maven, it's about selecting the right Ruby version for your project, a problem that also needs to be solved with Java. IME, many developers just let the IDE deal with that, but Ruby developers are more used to running things from CLI, so that's why rbenv exists. In any case, the most popular analogue for Java would be sdkman which works, mostly, the same way (and which, IME, is even a bit hackier than rbenv).
JS and Python are different. Node is a mess IMHO, and Python isn't able to agree on which package manager to use (I would hope people would migrate to Poetry, which seems to be doing things right).
[0]: For example, one thing that I find very annoying over in Java-land is that it's a real hassle to make sure things behave the same way when called from the CLI as opposed to from the IDE, such as setting the active spring profile. This is because, mostly, the IDE circumvents maven/gradle (you can run things through the build system, but typically the output, e.g. test output, is better when using the IDE-internal system, at least with IntelliJ).
[1]: There are "hackeries" to get rid of having to run everything through "bundle exec", such as binstubs and direnv, although I would mostly call them "shell script tricks", and anyway, if you instead just alias "bundle exec" to something shorter or create a wrapper script, it's not really worse than `./mwnw`.
[2]: This can be overriden and is actually a good idea for production builds, especially if you use a multi-stage docker build since then it's easy to copy over all your dependencies.
If that's the discussion you want to have, well, I personally also prefer statically typed languages, but that's beyond the point.
That said, I've had issues, e.g. on CI, that didn't manifest locally because of a mismatch in Java versions (something with JAXB, IIRC), and Java is maybe sometimes a bit too cavalier about the issue.
Regarding Python, I don't have much experience, other than the general feeling that it's fragile and that specific projects work with specific dependency managegers / virtual environments, and they are not really interchangeable or anything. I either could replicate the env or I couldn't. I definitely don't have much of a Python experience.
Now regarding your Java IDE annoyance, I share the same experience. Also the case with Tomcat. IDEs do so much hackery so that user experience is convenient, that you would be hard pressed to replicate the results, and often need to test both because it just doesn't work the same way, due to the different dependencies used, deployment method etc.
Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
Keeping the "right context" when you do things in different directories with the same shell config is, unfortunately, a problem that I don't think is easy to solve. My preferred approach is to use direnv (https://direnv.net/), but that's not perfect either.
As for rvm, I think that a lot of people at some point discovered that rvm was maybe doing a bit too much (in particular, it would also manage gem sets, something that seems superfluous with Bundler). I think that's a reason why rbenv and chruby emerged as lighter-weight alternatives which are a bit less magic. I haven't seen many people use rvm anymore in recent years. By contrast, Bundler doesn't really do anything to your shell.
> Now that I think about it, every single one of the programming ecosystems I used were quite annoying.
That's something I agree with. As a more recent convert to the JVM world (and mostly just for lack of better alternatives), I'm still sometimes annoyed at the IDE-centrism (and UI-centrism, see also sonarqube) in this space. Luckily, things are already better than they were when I learned Java in university and "it runs in my IDE" was the default state of affairs...
If people don't use rvm, do you know what they use to install a specific Ruby version? We basically use it so that we can use a Ruby that's not bundled with the distros anymore (large legacy project).
1. For selecting the correct JDK to use with standard plugins, including the compiler plugin, there's support for toolchains:
https://maven.apache.org/guides/mini/guide-using-toolchains....
2. For ensuring Maven itself is run with the correct JDK, it is a commom recommendation to use the Enforcer plugin to specify the required JDK version and/or vendor, as well as the Maven version:
https://maven.apache.org/enforcer/enforcer-rules/index.html
3. It is increasingly common to use Maven Wrapper to supply Maven, which you seem to be aware of.
4. Maven will run with the JDK pointed to by JAVA_HOME, of course.
I'd respond to production disasters with hacky bash scripts and then we'd spend the next sprint incorporating that functionality into the server so the hacks weren't necessary. We kept catching ourselves working really hard to optimize what we thought at first needed to be custom code but what ended up being easily achievable by leaning on well-known non-java tools like `diff`. There's something about the way Javs encapsulates everything into a JAR that discourages Java devs from using anything on the outside.
If you can live in a world where everything you need is in the JAR, life is good, but that hasn't frequently been my world.
It's only relevant because the high level of encapsulation that made deps easy to manage also represented a barrier. Using diff and awk was fast enough, the reimplementation in java wasn't (happened just a few times). Language ecosystems with a more permeable encapsulation style make it easier to reference tools outside of that ecosystem at the cost of making it harder to get your deps straight.
I was amazed to find such a mess. Each project has full copies of all sources of all modules you use? They’re not shared in a global cache? There’s just hundreds of thousands of individual files?
Yarn does a better job on some of this, but it all feels like stepping into an alternate reality where no one has ever heard of Maven.
Maven predates Node by 5 years. Even if Node was developed without knowledge of Maven surely someone has brought this up in the last 13 years, right?
It’s all very surprising to me, and I guess an interesting lesson in “parallel” evolution.
It’s also possible to install multiple versions of Python on a system. The dependency management is the real problem — having more than one version of a runtime is not the hard part.
> The dependency management is the real problem
I'm not sure how this works in Java, but in many ecosystems like JavaScript and Rust dependency management isn't an issue at all. Installing a package defaults to a local install and even things like dependencies transitively depending on different versions of the same package are supported out the box and just work.
Not to say that I like the Python package management story.
Previously: https://news.ycombinator.com/item?id=26018684 (Feb 2021: 98 comments)
FYI, the link to asdf there is a link to google.com
1. You need a tool for X to solve an issue so you write it. (AIM)
2. Guy over there needs a tool for Y to solve the same issue, so he writes it. (Yahoo Chat)
3. So many tools exist, we need to merge them all into one. (Trillian, Adium)
4. Merging all the tools created a monster, we need a common protocol they all understand.
5. No one remembers how tool X works anymore, updates become more sparse.
6. Both X and Y are no longer popular, big company took over. (Slack)
When I set `ruby: 2.6.8` in my `dev.yml` file, for example, I get this Shadowenv program written out to `.shadowenv.d/550_dev_ruby.lisp`:
(provide "ruby" "2.6.8")
(when-let ((ruby-root (env/get "RUBY_ROOT")))
(env/remove-from-pathlist "PATH" (path-concat ruby-root "bin"))
(when-let ((gem-root (env/get "GEM_ROOT")))
(env/remove-from-pathlist "PATH" (path-concat gem-root "bin")))
(when-let ((gem-home (env/get "GEM_HOME")))
(env/remove-from-pathlist "PATH" (path-concat gem-home "bin"))))
(env/set "BUNDLE_PATH" ())
(env/set "GEM_PATH" ())
(env/set "GEM_HOME" ())
(env/set "RUBYOPT" ())
(env/set "RUBYLIB" ())
(env/set "RUBY_ROOT" "/opt/rubies/2.6.8")
(env/prepend-to-pathlist "PATH" "/opt/rubies/2.6.8/bin")
(env/set "RUBY_ENGINE" "ruby")
(env/set "RUBY_VERSION" "2.6.8")
(env/set "GEM_ROOT" "/opt/rubies/2.6.8/lib/ruby/gems/2.6.0")
(when-let ((gem-root (env/get "GEM_ROOT")))
(env/prepend-to-pathlist "GEM_PATH" gem-root)
(env/prepend-to-pathlist "PATH" (path-concat gem-root "bin")))
(let ((gem-home
(path-concat (env/get "HOME") ".gem" (env/get "RUBY_ENGINE") (env/get "RUBY_VERSION"))))
(do
(env/set "GEM_HOME" gem-home)
(env/prepend-to-pathlist "GEM_PATH" gem-home)
(env/prepend-to-pathlist "PATH" (path-concat gem-home "bin"))))
"Late but latest" -- Rajinikanth
https://github.com/asdf-vm/asdf/blob/master/lib/commands/com...
Thalaiva! :)
[1] https://docs.gradle.org/current/userguide/toolchains.html#se...
[2] https://github.com/halcyon/asdf-java/graphs/contributors
The only plugins under the asdf-vm github org are for Ruby, Node.js, Elixir and Erlang, because those are the only languages I needed.
Not sure the error lies at the feet of asdf, though.
For ruby/node it's a relief to be able to easily "activate" a certain version of ruby and a certain version of nodejs - at the same time, via simply "asdf local ruby ... and asdf local node ...".
Not sure I'd do this, though:
mkdir -p ~/.config/fish/completions
cp ~/.asdf/completions/asdf.fish ~/.config/fish/completions
For the most part, latest version linting is good enough for me, currently.
Docker is better for version managing services like databases, caches (Postgres, Redis, etc), things that have little need for different configuration or exploration and persist through multiple sessions. Once they're up, they're done.
Asdf is the next best thing for local development, where you need that kind of exploration.
There are advantages to the docker approach, and Nix can be painful to get going, but a significant portion of the community (including me) find the pain to be worth it.
It was really impressive how smooth and quick the process was. Great tool!
No crazy settings in your profile. Downsides are: You need to have root, you can switch any time but different versions don't work at the same time, and it does not work in windows.
This isn't true
> different versions don't work at the same time
Within one stow directory: true. But nothing prevents creating multiple directories (I don't think I've ever done that).
It strikes me that not for the two above items we might be able to solve this with $PATH resolution and directories filled with symlinks to our "development versions."
Just today on trying to get rvm to do something I got angry at the number of language-specific package managers I have installed: nvm, jenv, rvm, rbenv(after I couldn't get rvm...)
Of course, I know about Nix but it is too complex for my use cases.
No, asdf is a replacement for them. It's not an abstraction on top of them. It uses its own plugin system, and everything is written purely in bash. Everything gets installed in `~/.asdf/installs`
> asdf version resolution is slow which makes every command execution pay that penalty. asdf reshim is needed for finding new executables, and some tools are not happy with their executables being masked by shims.
> Perform asdf version resolution only once and defer environment loading to direnv.
Nevermind trying to setup a development environment where you have multiple dependencies on different language runtimes.
It can't actually install different versions though (looks like `asdf` can?), which is somewhat inconvenient.
Regarding the suggestion to “do things manually” — what is the value of doing things manually? For me, it is very far away from how I want to spend my energy, which is doing computer things that only I can do. I am happy to outsource environment variable fiddle faddling to someone else.
Package managers often don't have a way to install specific old versions of a package, or multiple versions of a single package. That's the whole point.
if asdf could have a plugin that hooks into their shims to also handle direnv I would be in heaven
edit: just did a quick google search and it exists, i'm elated
I found it works well for Ruby, Elixir/Erlang, and Node.
let's see if the change is as smooth as it says!
pyenv I can understand. dunno the others.
but goenv?? you deserve whatever hell this is. it's a single binary configurable with an envvar for gopher sake. just RTFM.