Code Staging in GNU Guix | Hacker News Reader

Code Staging in GNU Guix

hal.inria.fr

65 pointsFice20 comments

Another explanation of g-expressions can be found here:

https://www.gnu.org/software/guix/manual/html_node/G_002dExp...

I don't know if it's just me, but a lot of build systems just seem unnecessarily complicated. I don't understand why it's necessary to have any more features than a templating language would provide. So for example, set up a template for each compiler with default values, allow the user to override the default values, and throw an exception if builds fail.

Gexps are not a build system implementation. They are a means to generate code.

Guix continues to use whatever build system a package requires, such as the GNU build system, the CMake build system, the R build system, the Python build system, etc. Gexps are not an attempt to replace these build systems. They are a way to stage code on the host side that is meant to run on the build side; this could be done with mere S-expressions through quoting, but would require more boilerplate. Gexps provide additional context, which allows for simpler code.

I hate to ask, but have you actually checked what the extra features are? There is a solid chance that the folks working on this are hitting real use cases.

Offhand, in this case in particular, it is probably to provide dependency isolation of things that you are building. Some things work as global to the computer dependencies. Some do not.

As for why give the flexibility allowed with a full language? In large, so that users are not constrained by the single vendor of the templates. You could have multiple vendors of the templates, of course, but I suspect you will find that immediately everyone is encouraged to self vend their own. In which case, few would ever actually talk about the templates, but the language the templates are written in.

My question was and is, why is there so much complexity? If there are valid reasons, then fine, but from what I'm looking at a lot of it seems unnecessary.

At its core, what is a build system meant to do? Build packages based on set configurations, right? In other words:

1. Set compiler flags.

2. Define/include/build dependencies.

3. Set build paths.

4. Build and test packages.

All of the required steps can be managed manually via the command line, and at its core a build system is just a way to automate this work. Why are other command-line tasks so much simpler to automate than package compilation? That's what I want to know.

How do you setup the rest of dependencies configuration ? How do you know that there are no hidden dependencies ? How do you handle time ? How do you ensure that it can be reproduce ? How do you define the dependencies ? How do you make sure that nothing change the build paths ? The deps paths ? That your environment is exactly the one you want ? That you dynamicly link to the correct lib ? And let's not talk about cross compilation.

Also how do you clean behind yourself ?

Same thing for testing with even more setup.

> "How do you setup the rest of dependencies configuration ? How do you know that there are no hidden dependencies ?"

Each dependency has its own build configuration file. You build a tree of dependencies and start compilation from the lowest level (i.e. the build objects without any non-compiled dependencies).

With such an approach, why would you have a hidden dependency?

> "How do you handle time ?"

What do you mean by this?

> "How do you ensure that it can be reproduce ?"

Semantic version control on the dependencies.

> "How do you define the dependencies ?"

Is this a hard problem?

> "How do you make sure that nothing change the build paths ? The deps paths ? That your environment is exactly the one you want ?"

By testing during the build process. In the case of the environment, it's just a question of checking against a list of expected environment settings (e.g. OS version, processor architecture, processor features, etc...).

> "That you dynamicly link to the correct lib ?"

This comes down to version control.

> "Also how do you clean behind yourself ?"

Again, is this a hard problem? Even the simplest of makefiles can describe how to handle this.

Perhaps it's best to give a concrete example of a task which is hard for a build system to manage, because I'm still not seeing the complexity here.

There have been a bunch of papers published discussing these topics at great length, and many attempts at producing simpler systems that still work at non-trivial scales. I'd suggest taking a look at https://nixos.org/~eelco/pubs/phd-thesis.pdf and/or https://nixos.org/~eelco/pubs/nspfssd-lisa2004-final.pdf for an overview of the origins of Guix and the reasoning behind its apparent complexity.

Thank you. That's all I was looking for.

Looking in the paper at issue and the Guix web site might have helped.

In my humble opinion, there is a lot of difference in complexity depending on what you're trying to do. If you just want to build software, then, like you I agree that is quite easy. If you want reproducible builds, things start to get tricky. If you want deterministic reproducible builds, then it get really complex. Guix goes beyond being a simple package manager/build system. It has a ton of features, even more if you read about GuixSD which is a complete distro managed by Guix. Really wonderful stuff inside.

This is what build systems do. What additional complexity are you referring to? Is there a particular project that you are thinking of?

Guix is a package management tool, not a build system. It surprises me how many people doing support for HPC systems, in particular, don't appreciate the difference even after they've been bitten by something like lack of dependency management.

Scheme has good abstraction facilities if you wanted to simplify package definitions somehow.

Guix may not manage software compilation, but it is passing parameters to the compilers used to handle compilation. So whilst its not a build system per se, it can be seen as part of the overall build infrastructure.

That said, my complaints about the complexity of build systems extend to package managers also, in that I don't see why they need to be as complex as they are. Part of me thinks it's because the tools lack sufficiently rich metadata to make their jobs easier, but otherwise I don't know what's causing it.

I have no issues with the use of Scheme in Guix (I'm learning Racket at the moment, no major complaints), and perhaps the abstractions will become simpler over time. I don't know enough about Guix to comment further on it. My comments were not aimed at Guix in particular anyway, but were general observations.

> Gexps carry information about the packages or derivations they refer to, and these dependencies are automatically added as inputs to the build processes that use them.

This means that you can just use package resources and they are available. This avoids having a seperate list of dependencies which is constantly becoming out of date.

In the Nix language, which Guix takes heavy inspiration from (to say the least), there are a small handful of primitive types, but no mechanism for building up bigger, more complex types. One of the primitive types is the derivation, which is not a set, not a list, not a number, etc. but a wholly new type which describes a package in the Nix store.

I agree that Guix overcomplicates it, but they didn't have to. It's a natural consequence of using an incredibly powerful host language to embed their build language.

Guix does not overcomplicate things. For example, it uses derivations exactly like those of Nix. (That's the extent to which Guix "takes heavy inspiration from" Nix: it uses the Nix daemon and thus the same derivation format.)

G-expressions are used to generate build-side code without having to resort to ad-hoc string interpolation in a target language like Bash. Ideally, this could be done simply with quoted S-expressions, but the paper explains why S-expressions are a little too primitive to use them conveniently in this case.

The gain here is that there is only one language and one set of rules for everything, namely Scheme. In Nix there is the Nix language and a mix of build-side languages (e.g. shell), where host-side values are embedded in build scripts through string interpolation.

There's work underway to extend the reach of Scheme to the daemon itself.

"Guix...uses the Nix daemon" and "to extend the reach of Scheme to the daemon" are incompatible and I hope you see why. Patches aren't flowing from Guix to Nix, or if they are, nobody's talking about it and the patches aren't showing up on anybody's radar.

The wimpiness of the Nix language is a virtue. There is no call/cc. There's no ports, and indeed networking is tamed. The language is just barely powerful enough to do its job, and community members have found it somewhere between intractable and impossible to build compilers, text processors, etc. in Nix itself.

String interpolation is a fitting punishment for our decision to continue to use Unix-styled systems; our systems think in bytes and communicate in bytes, and I see no reason why we should walk away from our highly-developed long-standing relationship with bytestrings as long as we are still on Unix. (You will say something beautiful about Scheme. I am busy looking ahead to capability-aware languages; Nix is a scaffold and nothing else.)

> "Guix...uses the Nix daemon" and "to extend the reach of Scheme to the daemon" are incompatible and I hope you see why.

These are not incompatible statements. Guix uses an older version of the Nix daemon that has diverged from Nix mainline. As Schemers we prefer to hack on Scheme things, so we're halfway through writing the missing glue to use existing Scheme tools in place of the C++ daemon.

> Patches aren't flowing from Guix to Nix

They cannot because Guix is not a fork of Nix. Guix is an implementation of functional package management as pioneered by Nix.

(As to your claims about Nix and Scheme: I'm not going to bite.)

> The wimpiness of the Nix language is a virtue.

Why does there need to be a language? Why can't there just be something like a manifest in JSON?

There can be, basically. But you often end up with huge repetitive mountains of JSON data, which isn't a ton of fun to maintain by hand, so then you write a tool to generate it from some other input data. Surprise, now you've invented a single-purpose DSL! What if you had a language that could serve as your input DSL, while remaining deterministic and still being flexible enough that you could skip the aforementioned dsl-to-json tool and just write that in the same config language? That's exactly what Nix is, along with some similar languages (Jsonnet, Dhall, Flabbergast, ...). They end up being useful as a way of generating whatever boilerplate or repetitive or obtuse input data you need to drive some system. Sure, you could do it in Python, but there's too many ways of doing non-deterministic non-hermetic things in a general-purpose language like that. Nix and Jsonnet and friends have no facilities for directly writing to disk, or talking to the network, or depending on some weird library being installed locally. They're just static configuration languages, and that can be extremely useful.

Reply on news.ycombinator.com