Comment by vishvananda | Hacker News Reader

Comment by vishvananda | Hacker News Reader

Parent Full thread

I've been waiting for this announcement for a few weeks. Since the home page is quite lean on an actual explanation, let me give a rough summary of interesting points as i understand it. I apologize in advance if I have anything wrong.

1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

2. It attempts to move some configuration management primitives into the build pipeline so that the deployment config has enough flexibility for real-world use.

3. It intends to replace single host process supervision with a distributed model based on gossip.

This project is most definitely ambitious, and I'm not sure the whole goal is realizable but some aspects are particularly interesting.

1. The build process seems quite useful for building containers, especially if it gets some traction. The main drawback of nix is that no one knows how to use it. If this gets the backing of the hordes of chef-aware system administrators, we may have a good solution for isolated repeatable builds, especially if they focus on deterministic reproducible builds[1] as well.

2. Distributed systems are notoriously hard to configure in a general way. It seems like every company has their own magic combination of config management, monitoring, and scripting for keeping systems like zookeeper running. Its pretty clear that we don't have the right primitives for sharing this work. I don't think a container management system has the right primiteves for this either, because I don't think you can solve for all distributed systems in a generic way. You really need custom logic for the particular software you are trying to distribute. I'm not convinced that a gossip-based process-supervisor is the best approach for this, but it could give us a place to start collaborating on these primitives.

[1] https://wiki.debian.org/ReproducibleBuilds

>1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

Except the builds aren't at all repeatable in a meaningful way, judging by the example code on the home page. The 'do_build' function runs 'npm install'. This means that 1) the build container has network access (i.e. it's nondeterministic by definition) and 2) the declared dependencies do not fully describe the dependency graph. So, I really don't see how much it could possibly borrow from Nix if it threw out the most important part of it.

They used npm install in their example code? Well that was a poor choice. Having dealt with npm and node apps in my own build system, I have not found a good solution for deterministically building node apps. Does nix have a solution for this? Thankfully the prepackaged software with reasonable source build systems don't do craziness like this. See the redis package for example:

https://app.habitat.sh/#/pkgs/adam/redis/3.0.7/2016061320525...

Nix has a generator that generates a Nix file based on an npm package including SHAs and locked down versions. Unfortunately because typical JS packages use a bajillion dependencies each recursively these Nix files can be tens of thousands of lines, but they work.

Nix also has similar generators for rubygems, python, haskell and some other languages.

Nix's cargo integration is even cooler since cargo is already quite deterministic, you just put in a SHA of the final folder cargo creates with all the dependencies.

The nix solution is npm2nix, which runs npm against a package.json file and generates a nix expression for all the npm modules you need.

This mostly works, but occasionally you need to override the generated derivations to add implicit dependencies that npm doesn't capture, or otherwise tweak packages that rely on quirks of npm.

So basically it "does not work"...

Well, let's be clear, if it doesn't work, you know at compile time, not at runtime. I don't care how broken things are at compile time if I'm guaranteed that everything will work at runtime. That's kind of the whole point of nix. It's not impossible to write code with errors, but you should not be able to build as long as the errors exist.

Also, it does work. It's just not completely automatic.

I think the point is more that it's completely under your control. The application build would be done however best suits your infrastructure and requirements.

Using npm doesn't demonstrate repeatability, but you could just as easily build a given application from fixed, verified source in this step.

Many (bad) build systems will download stuff from the net and you may not even know it. I've discovered plenty of software like this because I used GNU Guix to build them, which does not allow network access to builds for which the checksum is not known in advance.

It's absolutely essential that a package manager constrain the environment in which builds are performed. Without it, the packages are doomed to be built in a nondeterministic manner and potentially tampered with.

You always have the option of running a local npm cache/registry. We do for npm and artifactory.

Not exactly the same as vendoring dependencies, but it's a decent compromise in some situations.

Thanks for trying to explain. So many times when I visit these websites advertising developer-oriented tools I just click around trying to find an actual explanation of what the thing does and how it works, only to close the tab a couple of minutes later with no clear idea.

Reply on news.ycombinator.com