Hacker News Reader

Comment by hamandcheese | Hacker News Reader

I think the lack of true laziness will be a big performance problem for a large build graph.

On the other hand the monolithic nature of the nixpkgs package set is one of the authors gripes with nix, so performance at that scale may be a non-goal.

View on HN

I'd definitely like to have good performance even for large build graphs! I'm hoping the laziness exists "where it counts". To walk through an example, if you build your backend, and your backend calls the function `postgres()`, and that calls `openssl()`, and THAT calls `gcc()`, etc., etc., each function is basically building an object to represent its chunk of the build graph (each function returns a "recipe"). Nothing gets built until that object gets returned from the top-level function and the runtime does something with it

In other words, the eager part is basically constructing the build graph. Maybe I'm wrong but I don't that this would necessarily be slower than the lazy version. In practice the most complex build graph I've made is basically the full chain of Linux From Scratch builds (that's the basis for my toolchain currently), and I think that takes about 400-500ms to evaluate. It's about 160 build steps, so it's not _simple_ but I know build graphs can also get a lot more complex, so I'll just have to keep an eye on performance as I start to get into more and more complex builds

Maybe I'm missing something but intuitively I'd expect this approach to be fairly efficient-- as long as build scripts only call these functions when they're used as part of the build graph

I think it really depends on your definition of "large". I don't think strict eval + full build graph can scale to something the size of nixpkgs, for example.

I mentioned in another comment that this is why Bazel uses simple strings to form dependencies on other targets. That way Bazel can manage the laziness and only evaluate what is needed without needing to use or invent a language with lazy evaluation.

But that is also the big downside (in my opinion) - the full build graph necessarily can't exist purely in starlark (at least for Google-scale projects) which increases complexity of the tool overall.

Edit: I'd like to add, though, that I think it's perfectly fine to not scale to Google scale or nixpkgs scale! Many many projects could still benefit from a great build tool.

Honestly, I think the "stringly-typed targets" thing isn't too bad, having used Buck2 quite a bit, and being a Nix user for 10+ years. If anything, it's a small price to pay for some of the other smart things you get in return, like the far more fine-grained action graph and the tooling around BUILD files like querying. One weird benefit of that stringly-typed bit is that the BUILD files you have don't even have to meaningfully evaluate or even parse correctly, so you can still build other subsets of the tree even when things are broken; at ridiculous-scale it's nearly impossible to guarantee that, and it's something Nix does worse IMO since evaluation of the full nixpkgs tree is slow as hell in my experience but a requirement because a single eval error in the tree stops you dead in your tracks.

Also, no matter how much I might not like it as a language nerd, I think Starlark is simply far more "familiar" for your-average-bear than the Nix language is, which matters quite a bit? It might be more complex in some dimension, but the problem space is fundamentally complex I think. So other factors like how approachable the language is matters. (And at least in Buck2, you can use MyPy style typing annotations, thank God.)

> One weird benefit of that stringly-typed bit is that the BUILD files you have don't even have to meaningfully evaluate or even parse correctly, so you can still build other subsets of the tree even when things are broken; at ridiculous-scale it's nearly impossible to guarantee that, and it's something Nix does worse IMO since evaluation of the full nixpkgs tree is slow as hell in my experience but a requirement because a single eval error in the tree stops you dead in your tracks.

I think you get more or less the same property with Nix. You can have all kinds of errors, even certain syntax errors in the same file, but if they are unneeded for the current evaluation, they won't cause any problems.

As for language familiarity/approachability - this will always be a matter of opinion, but I personally don't think it makes sense to optimize for the casual contributor. Plenty of people know python, but I never see casuals making anything besides trivial changes to bazel build files. I don't think they gain anything by familiarity with python, they could very well copy paste nix or any other language. And if they get in to trouble, they will call in the experts.

Reply on news.ycombinator.com