Hacker News Reader

I haven't used Nix, but I would have thought that builds would be fast, due to how cache-able the dependencies should be.

Sounds like they were building everything from scratch, for some reason.

That’s how nix guarantees reproducibility. The compiled artifacts can be cached but that didnt quite work out of the box

Nix guarantees reproducibility, which means anything can be rebuilt from scratch, but that's a very abnormal use case. If it didn't work out of the box, it's a problem with the package scripts (the "derivations"). That said, all of our software tends to bottom out in a bunch of shitty C libraries that are all delicately cobbled together with autotools and cmake, so anything that aspires to reproduce these things is going to have issues. This tends to make Nix difficult to use, because it doesn't have nearly the same investment/manpower (yet) as other package ecosystems that is necessary to wrangle these dependencies into a stable foundation that doesn't leak its underlying havoc to higher levels of the stack.

This comment is really, really confusing. If it builds once Nix, it will build again with Nix... If you can find a derivation that builds on one machine and not on the other there's usually a fundamental difference - either in CPU arch, your nixpkgs config differs (overlays), etc. Or I guess you could write a build script that non-deterministic ally fails, but that has nothing to do with Nix's maturity.

Nix ensures the tooling is called the same. I can understand having a non-autotools project that is more difficult to write a derivation for, but "Nix can't wrangle messy libraries" makes absolutely no sense, either from a "make compiling reliable" or anything at use-time.

Do you have a specific example in mind that is less hand-wavy?

> If it builds once Nix, it will build again with Nix...

That's the aspiration, but it doesn't always pan out. The rate at which you run into problems depends a lot on the packages you use, how much attention is given to them, and how hard it is to reproducibly package them. I've noticed in particular that the Python ecosystem is really fragile.

> Do you have a specific example in mind that is less hand-wavy?

A specific example that comes to mind was the psycopg2 Python package, which would build on some developers' machines but not on others. This sort of thing happens all the time, usually on macOS, and usually with C packages (sadly, so much of the Python ecosystem is built on C and its shoddy build tooling). I've also found quite a few packages in nixpkgs that simply don't build on macOS, but which presumably build on Linux; however, I forget which ones specifically.

We've been using Nix for deploying a Rails app for an enterprise customer for quite a few years now. One area where it shines for us is the ability to build it on relatively recent version of Ubuntu and deploy to a (almost EOL) RHEL6 box. Bundling, assets and various other tasks take just a few minutes. We also have ~20 Go services that are also deployed via Nix, and building takes seconds.

However, it can be quite cumbersome to get a Nix expression to the point where it builds reliably for something that takes multiple steps like a Rails app, especially if you're building on macOS and deploying to Linux. It's come a _long_ way in recent years, but with Enterprise customers now embracing containerization we migrated everything to that and haven't looked back.

I'm having a hard time parsing -- you migrated everything from nix to containers (containers removed the need for nix) or to nix+containers (containers solved the "having to build for multiple platforms" issue)?

Ah my apologies - We use Bazel to build our services, and the output artifacts were then pulled into Nix and deployed as Nix packages. Bazel has _excellent_ support taking the same application code and creating Docker images from them (https://github.com/bazelbuild/rules_docker#language-rules), and the tools available for deploying containers is orders of magnitude more feature-full and higher quality than what you get with Nix today. So we no longer have Nix anywhere in our pipeline, and all of our artifacts are now deployed inside containers.

It is, perhaps they didn't use cache to store previously built packages and build everything from scratch. With no cache Nix will start with building compilers and glibc until it has everything to build the actual application.

While this is true if you are building new versions of dependencies on an existing system, in many cases in high security environments you want to start with a clean build environment every time. Cached dependencies might be considered to be a security risk. Bootstrapping an entire Nix environment can take a lot of time.

Sure, but in most systems you wouldn’t have much of a choice; if you’re using Debian or Ubuntu, you’re probably using binary packages built by someone else. It takes a lot more work to build everything you want from source in a traditional distro.

It seems like a pretty straightforward tradeoff: do you want to rely on prebuilt packages for speed, or do you want to rebuild it all (and accept the extra time that takes)? Nix and Guix at least make the latter pretty straightforward to do.

Sounds like their main concern wasn't Nix but the complexity of the code itself.

The hours long build system appears to be attributed to Nix.

Is Nix really that slow?

Dependencies can take awhile if you can’t used cached versions of the binaries. Building some Haskell tooling from scratch took ~3 hours on my MBP.

There are few compilers that are slower than GHC. I suspect this issue has nothing to do with Nix, and the tools would have taken ages to build from scratch without Nix.

If you're applying idiomatic Nix to modern JS, I wouldn't be surprised - modern JS tends to involve installing thousands of packages by just combining them into a directory, but Nix doesn't want you to edit existing directories. So your dependency graph turns into a build-dependency graph, with each JS package requiring a full build of everything it depends on, and the Nix build system is presumably not optimized for fast turnaround times on five-line JS modules that don't even have a compilation step.

(Personally, I wouldn't try to apply idiomatic Nix to modern JS - I'd apply it to the major components like "my web server" and "my database library" and have "all the JS I depend on" as one big Nix package. That's not really a claim about Nix, I wouldn't try to apply idiomatic, say, Debian packaging to modern JS either. In both cases I'd still get about 90% of the benefit of using Nix/Debian as a delivery mechanism.)

The other thing that could be slow is if you start compiling all your dependencies, including gcc and node.js, from scratch. While there's some security benefit in doing so, the reality is that just about nobody actually does that. You'd want to set things up to use the precompiled packages, or at least set up your own cache server and have compiled binaries you trust but only do it once.

Hi, JS developer here who uses Nix and who liberally makes use of single-responsibility modules.

No, Nix does not need "hours" to build a modern JS project. Something else is going on here. Nix is slow, but not that slow.

(With the limited information provided here, my first guess would be "someone turned off the binary cache because they don't trust prebuilt binaries, and now it recompiles the world from scratch because there are no prebuilt binaries anymore".)

I suspect that someone being the auditors who are suspicious of opaque binary caches and insist on building everything from scratch.

I don't know. If that were the case, I'd imagine they would know why their build was taking hours and that it wasn't a Nix problem.

That was my reading, based on the fact that they glossed over Nix itself and attributed most of the issues to code complexity.

It explicitly says this, though:

> while much of that build time is attributable to Nix's way of building dependencies

And that part just doesn't make sense to me. I can't see any way in which specifically Nix's way of building dependencies would contribute to this. The compiling of the universe, sure, but then why mention Nix?

It does say that, but it's off in its own clause, then shifts focus away from it towards what seems like the focus of the auditors.

The term “modern JS” should be used only as a pejorative.

Would you please not post in the flamewar style to HN? It damages the container here (i.e. the capacity of this place to sustain curious conversation), and you unfortunately did it repeatedly in this thread.

https://news.ycombinator.com/newsguidelines.html

I've built a combined python2/3,c,c++,javascript (VueJS) deployment along with hardware-in-the-loop integration testing and compiling for ARM on an Raspberry Pi with Nix. Should not take hours.

Something else is wrong. Perhaps binary cache is not being used? Something is invalidating the builds like a "src = ./.;" which telling the build system to include every single file in the directory in the build rather than using a .gitignore or git-hash for reproducibility. Or someone was using npm2nix (deprecated)?

No, nix can cache already built binaries with hydra, so on a normal system it simply downloads from there - same speed as apt or the like. During the audit they likely disabled it, so nix had to build everything from scratch (like even system libraries, the build tools, everything).

Ah yes.. cryptocurrency, where bizarre programming practices secure large sums of money

LOL. Reading that audit report made me immediately wonder who in the heck is blowing money on bringing new tech to their critical finance systems. And then I saw your post and, yes, that answers that.

I am aware of a few trading firms that are phasing in nix to handle their dependency clusterfucks.

So the auditors said that the system was hard to audit?

Putting it this way it misses the point of that statement, but yes, that does appear to be what they said.

Considering that “difficult to audit” overlaps with: difficult to understand, difficult to deploy, difficult to onboard, and a bunch of other things that are also very important to non-auditors, that seems like an entirely reasonable and informative complaint.

If you hire a financial auditor and their report was basically an nicer version of “‘books‘ were written in pencil on napkins, many food stained, some illegible. Attempting to make sense of them took forever. Fix this shit to meet minimum standards for business accounting, it’s entirely unsuitable for its purpose” that’s be great info, if you didn’t already know it.

"books written in pencil on napkins" is analogous to how most web application projects are organised today. Nix would be the formalisation of this.

Never used nix, but their stated goal is to make implicit dependecies explicit (and reproducible). Isn't it only natural for this to sometimes bring a dependency hell to light, that otherwise could have been swept under the rug?

Or phrased differently: maybe the problem here wasn't nix, but the way developers chose their dependecies

Woah, I didn't know there is an entire ecosystem of contract job and auditors for big projects. How do companies usually hire contract jobs (outsourced HR, upwork, Accenture)? And how do they hire auditors?

This sounds like a security audit, and auditors found the build system difficult to work with when attempting to audit code.

This used to be called "quality assurance". Web and .com blew that apart. We ain't got time for that.

So I understand correctly from this that nix has no binary packages? If so, why not? To me, compiling from source is not strictly necessary to get reproduceability guarantees. It might actually be harmful if you're not carefully checksumming the build products (e.g. cosmic rays messing with complex builds).

You do not understand correctly.

Nix users use binary caches[0] instead of building everything from source. You can use the binary cache provided by NixOS, or Cachix[1], or something you set up and host yourself, or all of the above.

[0]: https://nixos.wiki/wiki/Binary_Cache

[1]: https://cachix.org/

Then why would a build take so long as GP describes?

I can't imagine why some unknown people encountered difficulty building some unknown project.

We use Nix to build and deploy our big Haskell monolith along with the operating system itself, and all system dependencies, e.g., PostgreSQL, Redis, Grafana, collectd, influxdb, openssh, ejabberd, Elm applications, etc, and it works fabulously.

GP here. My guess is that they intentionally disabled binary caching because this is a financial application and they wanted to make sure no untrusted binaries got into the build pipeline.

Thanks! That makes some sense. Though, I work for a financial service provider as well and have never found compile-from source to be solution to that problem. If you do have dependencies, whether you trust the sources or the binaries shouldn't really matter, as long as you trust the repositories where they come from. And if you don't trust it I wanna see the army of developers actually checking all the sources down to the system level.

I can't entirely speak for Nix, but I know they're fairly similar, so: Guix (and I'm pretty sure Nix) have optional binary packages. Realistically most users will use those, but the option to build from source is always there. So, while you say:

>To me, compiling from source is not strictly necessary to get reproduceability guarantees.

...I somewhat agree, but you do of course need to be able to build from source. And, in a system like Nix or Guix, you have an easy way to verify that a build and all its transitive dependencies are reproducible, if you have someone else's build to compare it against. See `guix challenge` for example: https://guix.gnu.org/manual/en/html_node/Invoking-guix-chall...

But the build time dependencies would still be there. Just implicit and undocumented.

It sounds like "this is the first time I've seen this build system and don't really see how it works, would be better to change it to a more often used one".

Which is fair for a security auditor, but it stems from the newness of the project and nothing inherently bad with it - like I doubt they know what actually happens inside whatever other build system other's use, but since they empirically know it poses no threat they are okay with it. If you ask me, this is absolutely no reason to not use Nix - well maybe not for a bank (though on long term they would definitely win with it)