Comment by ducktective | Hacker News Reader

Comment by ducktective | Hacker News Reader

Parent Full thread

Link to the thread for other people: https://nitter.net/dmitriid/status/1439957729909559298#m

Ju (@Arkham) proposed a solution:

1- Finding the commit hash of of nixpkgs which contains a specific version of a dependency (e.g. ruby 2.6.3) and use that to bring in the dependency

2- If a specific version does not exist in any of the hashes, we should patch the `.nix` file and build the dependency ourselves. This causes all other transitive dependencies to be rebuilt.

Arguments against 1:

Why should we rely on an external thrid-party webservice to find the hashes of a specific package? This should be integrated to the `nix` tooling. Hope the community incorporates history search into the tooling :)

Arguments against 2:

@dimitriid says the solution is cumbersome.

Arguments for 2:

- What if rvm or nvm don't have a specific version or deprecate that version from their repos? At least there is a solution via Nix.

- What are you going to do for other languages which don't have a package/VM manager like Java, C/C++, etc? At least the Nix solution is universal and you don't have to learn the tooling of every technology.

I think @technomancy over at Lobste.rs has a wise take on this issue: https://lobste.rs/s/emyfhx/ditch_your_version_manager#c_5acg...

> I feel like Nix/Guix vs Docker is like … do you want the right idea with not-enough-polish-applied, or do you want the wrong idea with way-too-much-polish-applied?

> Arguments for 2:

> - What if rvm or nvm don't have a specific version

Let's get back to reality for a second. For most development needs this is a hypothetical situation.

Right now, in reality, ruby-lang hosts versions dating to 2002.

Also, right now, in reality nix limits the number of available versions. Quote: "in NixPkgs we choose to restrict the number of maintained versions", https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6... So it's already worse than the hypothetical "what if rvm don't have a specific version".

Also, right now, in reality there's no easy way to pin a version of a tool in nix. Except going to a third-party website (lazamar's site), hope that your specific version is listed there (he only maintains a snapshot of a few years), and dig through nixpckg commit hashes. And this has been a known issue for as long as nix has been around. 18 years, at least. And we're still hoping that someone somewhere will incorporate this into standard tooling. You yourself linked to an issue on this.

I'd like to link to a reply in that issue: https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6...

And to this reply: https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-7...

> What are you going to do for other languages which don't have a package/VM manager like Java, C/C++, etc?

That varies between languages, of course. I have several Java versions installed side-by-side. C/C++ is worse because they are mostly stuck in 1960s, tooling-wise (though on windows you can have multiple msvcrt versions installed side-by-side).

This still can be (and is) handled. And nix doesn't offer great advantages, except providing a complex, complicated, poorly documented tool.

>Right now, in reality, ruby-lang hosts versions dating to 2002.

OK, so you want to pin ruby 1.3.4 in your project and use a ruby gem that is dependent on a specific glibc (or maybe some standard ruby library itself is dependent on a low-level C dependency.). With python these stuff happens all the time. What would you do?

Craft a dockerfile inheriting from an old Debian and hope ruby 1.3.4 installed with rvm would play nicely with this specific version of Debian? How do you find what version of say, Ubuntu, has fixed security updates of the low-level dependencies of your project? And what happens if the apt-managed JRE v.1.6.x in that instance would break other parts of the project that was dependent on JRE v1.6.y?

>That varies between languages, of course.

Well, that's the point. For each technology, you have to handle their specific "VM manager" and put up with their quirks + random state-changes imposed by apt. With Nix, the solution would not vary between languages, setups and teams. I mean, at least theoretically, the Nix/Guix way makes much more sense than shipping the whole state of the computer (with docker).

> OK, so you want to pin ruby 1.3.4 in your project and use a ruby gem that is dependent on a specific glibc (or maybe some standard ruby library itself is dependent on a low-level C dependency.). With python these stuff happens all the time. What would you do?

What amazes me is how you keep asking what would happen in other places instead of asking/answering "how easy/convenient would that be in nix".

Let me give you a different question: I have ruby x.x.x. A serious security issue is discovered, and everyone is encouraged to upgrade to ruby x.x.x+1.

Let's see how easy it is to do in nix which is "a tool that takes a unique approach to package management and system configuration."

Oh. Wait. There's literally no such way. Except manually looking for git commit hashes... somewhere.

Let me quote a reply from the issue you yourself linked:

--- start quote ---

I've read the Nix Package Manager Guide, but everything in there about reproducibility seems to point to installing (and therefore finding) specific versions. I mean the first title in chapter 1 is multiple versions

Any other multiple-simultaneous-version managers: asdf, kiex, gvm, rvm, rbenv, pyenv, jenv, nvm, Renv, luavm, all have a list-available-versions as either their 1st or top-5 most-used commands. But nix has 5.3K stars, so there must be something I'm not understanding if nix is missing something so vital

--- end quote ---

> With Nix, the solution would not vary

What was this solution again? In the entirety of the discussion to this post there was exactly one person (the author of the article) who could show how to do it.

Even nix proponents don't know how to do that, apparently.

Sorry if I may have come up as a Nix professional/proponent. I'm actually a nix beginner and as can be seen in this thread, I've asked the author similar question to yours. Using Ubuntu, I have been bitten by glibc and other dependency mismatch in my projects and I'm looking for something more elegant than docker/VBox.

I've read posts from both sides of this discussion.

> Oh. Wait. There's literally no such way.

I think there is? Check to see if the nixpkgs history have that version (+), if not, patch the derivation (could be as simple as bumping the version and recalculating the hash, or, doing so could cause a breakage in one of the transitive dependencies, at which point, you have to fix that also (meaning a version increment of decrement for the dependency or more. Essentially you have to build the specific ruby version down to all its dependencies)

(+) How can we see what versions are available in nixpkgs history?

It seems we have to use third-party webservices! I agree with you here that this should have been implemented in the nix tooling itself. I know of two such services:

https://4shells.com/nixdb/pkg/ruby/2.6.6

https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable...

I think this also answers that quote from the github issue.

> What was this solution again?

From what I understood, in the project `.nix` file, we inherent from a forked nixpkgs repo which has our patched `default.nix` in the ruby folder, for building ruby. (Nix pros, correct me if I'm wrong here). Now we have to build all deps locally since a matching hash would not be present on Nix cache servers.

Now, with nvm, how could you guarantee that a particular version would not conflict with one the host OSs low-level libraries (Like glibC)? The difficulty and complexity of patching the Nix derivation has to do with this part. In Ubuntu/Debian, you can't simply mess with system level low level libraries.

>> Oh. Wait. There's literally no such way.

> I think there is?

There is no official way to do that. There's no history. There are no tools in nix to do that. As you noted, you have to rely on the kindness of strangers to provide this service. Here's lazamar's own explanation why he built the tool: https://lazamar.github.io/download-specific-package-version-... Look at the beginning of the "automating" section to see what is required to do all that with nix.

On top of that, in the entire discussion here exactly one person could actually show how to pin something to a version, so it's poorly documented, extremely complex and not really understood even for people who propose nix.

> Now, with nvm, how could you guarantee that a particular version would not conflict with one the host OSs low-level libraries

You probably don't, or stick it in a container somewhere. But this is about tradeoffs: does nix provide enough tradeoffs to use this complicated, complex and poorly documented system?

But the Nix way (how alien and complicated) is the theoretical sound way of handling dependencies. The docker solution is a blanket over a mess.

Hopefully, documentation and tooling can be improved for a theoretically good but badly implemented solution. The other way around is not possible.

> But the Nix way (how alien and complicated) is the theoretical sound way of handling dependencies.

The approach may be sound, but the implementation and the user experience is atrocious, and can be quite easily[1] improved.

[1] Easily as in: there have been multiple suggestions on what should be done to improve experience, but nix's codebase may not be conductive to that https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-7... and nix contributors don't even see this as an issue https://github.com/NixOS/nixpkgs/issues/93327#issuecomment-6...

That's the point: without using Nix, you'd have to do LD_LIBRARY_PATH hacking to have two programs use two different versions of glibc. When people say "oh nix looks so complicated I can do this using bash scripts" I reply "well show me these beautifully expressive and simple bash scripts then".

Funny how you're talking about how horrible it is to install glibc, when the entire reason lazamar's tool exists is that he couldn't figure out how to install a specific version of a tool using nix https://lazamar.github.io/download-specific-package-version-...

"Why do you look at the speck of sawdust in your brother’s eye and pay no attention to the plank in your own eye?"

Also, not all of us depend on glibc or changes in glibc. Making this solely about glibc while ignoring the larger issue is... can't find a proper word for it.

(and of course there are issues with glibc in nix as well, https://discourse.nixos.org/t/clarification-on-package-names... because glibc is weird in itself)

I updated my reply. Can you confirm that inheriting from a forked nixpkgs with a patched default.nix in ruby directory is the solution for cases that a particular version was not ever present in history?

There is no need to literally fork the nixpkgs repo. Most language derivations were written with the idea of supporting multiple versions at the same time. In this guide (https://www.breakds.org/post/nix-shell-for-nodejs), the key example is this one:

    let pkgs = import <nixpkgs> {};

        buildNodejs = pkgs.callPackage <nixpkgs/pkgs/development/web/nodejs/nodejs.nix> {};
        
        nodejs-8 = buildNodejs {
          enableNpm = true;  # We need npm, do we?
          version = "8.17.0";
          sha256 = "1zzn7s9wpz1cr4vzrr8n6l1mvg6gdvcfm6f24h1ky9rb93drc3av";
        };
    in pkgs.mkShell rec {
      name = "webdev";

      buildInputs = with pkgs; [
        nodejs-8
        (yarn.override { nodejs = nodejs-8; })
      ];
    }

where we import the normal nixpkgs, then we use `callPackage` to re-use the code that was written by the Nix mantainers, and we specify our own version and SHA.

If the derivation hadn't been written to be reusable what you can do is to copy this whole directory https://github.com/NixOS/nixpkgs/tree/nixos-21.05/pkgs/devel... to your local project and import the nix files locally. In this case, some of the artifact might be missing from the nix cache and you might need to compile some from source. We do this in some cases and upload the artifacts to our private store on https://www.cachix.org so that no engineer has to recompile them again on their own computer.

Reply on news.ycombinator.com