Guix for Development
dthompson.us
dthompson.us
One thing I couldn’t work out how to do in guix was to go the whole way through. Something like:
1. Install / use some program
2. Find some bug or potential improvement
3. Run the command to be dumped into a dev environment for that
4. Fix or investigate the bug; or implement the improvement
5. (The step I don’t know how to do) Use your fixed/improved version in your operating system
6. (Also don’t know how to leverage guix for this) Share your improvements with others
It feels like 5 is somewhat at odds with how guix is meant to work as step 4 is so mutation-heavy, but it also feels like something that an OS which wants to be as gnu as possible should really want to support. And gui should be able to make it safer too by giving you rollbacks. Maybe there’s an easy way to do it and I just don’t know it?
guix install foo --with-source=foo=yourfixedversion_src.tar.gz
Or:
guix install foo --with-patch=foo=yourpatch.patch
(note: usually packages are installed as regular user into your user profile)
See also section "Package Transformation Options" in "info guix".
For step 6, send a mail with the patch to the package definition (in an existing scm file in https://git.savannah.gnu.org/git/guix.git ) to the guix-devel mailing list. But whatever you did manually to get yourfixedversion_src.tar.gz you now should automate by programming the patching process in Scheme instead--editing a guix checkout (usually just to add: (add-after 'unpack 'patch-problematic-stuff-4711 (lambda _ (substitute* "somefile" (("regex1") "replacement"))))). For more complicated fixes, add a patch to the guix checkout and make Guix use it in Scheme (and contribute it to the actual foo upstream project--which we usually do anyway).
See section "Contributing" in "info guix".
Further automation welcome-- and probably not that difficult to add. Just keep in mind that each package build is in its own container (it's kinda like Docker would be)--so no mixing text editor and building and weird user-defined pauses (for example there is no text editor in the build container).
The reason for the seemingly duplicate "foo" in the "--with-" options is that those "--with-" can apply to dependencies if you specify the right package there (so in the example, something else than "foo"--like something foo depends on). In that case it will recompile everything that needs recompiling.
> But maybe it’s actually hard to do for reasons I haven’t thought about.
It can easily be that we all made our own workarounds and so while it would be easy to add a nice frontend, nobody has a need.
So if you do think of a nice way, please try to do it.
For example we have some bias for emacs: inside emacs the guix development is very comfortable (we have and ship our own extensions for emacs). But that doesn't help you if you are not an emacs user.
The Guix package repo is a monorepo (see below)--all those packages have to work (and be tested) together.
From time to time I do think it would be nice to reduce barrier of entry by automating existing package editing inside a shell script we could ship:
if [ ! -d guix ]
then
git clone --depth=1 https://git.savannah.gnu.org/git/guix.git guix
else
(cd guix && git pull --rebase)
fi \
&& cd guix
&& guix shell -C -D guix -- ./bootstrap \
&& guix shell -C -D guix -- ./configure --localstatedir=/var --disable-daemon \
&& guix shell -C -D guix -- make -j5 \
&& ./pre-inst-env guix edit foo \
&& git add gnu/packages/*.scm
... but so far it never makes it out of my head to an actual script shipped with guix :(
If we did make it--what would the package it would be in be called? Or should it be a subcommand of the "guix" command?
That said, if you want to make your own NEW packages, that's easy and well-supported. You can put a guix.scm whereever you like and lots of guix stuff will pick it up automatically.
This is exactly right, and I think it's one of the coolest things about Guix. Guix System (the full OS based on Guix) seems almost like the GNU that was dreamt of before Linux. It even has an option built on the Hurd kernel, and it uses an init system and initrd images based on Guile Scheme, which is both the language used for Guix itself and as the extension language for GNU projects.
Guix definitely advances the whole
> Learn one simple, high-level language and you can hack on every aspect of your system
idea further than we've seen on any modern/contemporary system.
> Aside: I think this is also great about Emacs; you can easily jump to the source of most things and there’s usually a way to hack or modify or inspect or debug them immediately
I think Emacs is definitely an inspiration, or a model of the virtues intended.
With guile, you get none of that, there's seemingly no debugging tooling at all. The best Emacs integration you get is Geiser which is not a GNU project and doesn't offer the debugging functionality mentioned above.
This is very surprising considering that Emacs and Guix are under one roof, and Guix is arguably GNU's flagship new software. Why do the development tools suck?
The documentation is good
It still has problems here and there, and packages missing. Probably due to the lack of wide adoption for being new and different
But packages that are missing, most are easy to create or find online, to use in your private repository, or submit to main repository
There was less difficulties than expected. Things that didn't work, or I didn't wanted to waste time fixing
There is no way to build the package graph on macos. There is no glibc port for macos, and there's no free toolchain we could legally use.
There is little value in building Guix when we are forced to use a different proprietary toolchain, a different C library, and throw away the from-source bootstrap in exchange for Gigabytes of proprietary blobs. You might as well use Guix System in a virtual machine on macos.
As far as the ideological reasons goes, last time I looked for this, someone on a mailing list asked how you'd go about porting guix to macOS and some core team member answered they had no interest in supporting a non-FOSS system.
This is not about supporting a non-free system; I bet some nuance was left behind on the way from writing that mailing list post to the comment I'm responding to. Building packages for macOS would require an entirely different underpinning --- or cheating but cutting the package graph and grafting it on top of the huge proprietary blob that is the macOS development environment. It would also require that we buy macOS licenses and operate macOS build farm nodes to build binaries.
And for what? The resulting packages would in no way be equivalent to their GNU+Linux counterparts. The "same project definitions" would not describe the same thing at all.
https://guix.gnu.org/en/packages/python-pytorch-1.12.0/
This makes me wonder if Guix is ready for the real world.
ML and GPUs are a smallish subset of the real world, and a subset that's particularly plagued by proprietary software (especially the GPU side); I don't think it's a good litmus test for "is guix useful for real work".
That firmware could literally ship on the card totally invisibly and that would be fine for guix, but because it can be patched in the field, no.
I don't get it.
But there are some spectre and meltdown mitigations that comes with CPU microcode, right? It's probably not a great idea to not apply microcode updates in this case
# du -bs gpu/
2053094 gpu
One needs to jump through a bunch of hoops to get hardware-accelerated PyTorch on Apple’s latest computers. Does that mean they aren’t ready for the real world?
ML is also increasingly relevant for OSes, think OCR and speech recognition, but also painting programs. Image classification and segmentation are used in industrial systems; lots of ML happens on servers. And this is where Linux shines, and thus support from Guix would be great. Categorically excluding this group of GPU users is a mistake.
> One needs to jump through a bunch of hoops to get hardware-accelerated PyTorch on Apple’s latest computers. Does that mean they aren’t ready for the real world?
They are. But their packaging systems are not.
Furthermore, you can install Nix in Guix: https://guix.gnu.org/packages/nix-2.5.1
I think Guix is more than ready for providing development environments, even for proprietary apps on open source technology stacks (but Nix will probably be easier to pitch). As an operating system much less so.
nVidia? Nix? Or Nonguix?
Don't forget about non-GNU.
If you do know the build options to make this work, Guix will gladly consider your patch.
The in-progress diff would be,
diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm
index e702e499fc..73af29487d 100644
--- a/gnu/packages/machine-learning.scm
+++ b/gnu/packages/machine-learning.scm
@@ -101,6 +101,7 @@ (define-module (gnu packages machine-learning)
#:use-module (gnu packages swig)
#:use-module (gnu packages tls)
#:use-module (gnu packages video)
+ #:use-module (gnu packages vulkan)
#:use-module (gnu packages web)
#:use-module (gnu packages xml)
#:use-module (gnu packages xorg)
@@ -2897,6 +2898,10 @@ (define-public python-pytorch
(build-system python-build-system)
(arguments
'(#:phases (modify-phases %standard-phases
+ (add-before 'build 'use-vulkan
+ (lambda _
+ (setenv "USE_VULKAN" "1")
+ (setenv "USE_VULKAN_SHADERC_RUNTIME" "1")))
(add-before 'build 'use-system-libraries
(lambda\* (#:key outputs #:allow-other-keys)
;; Tell 'setup.py' to let 'CMakeLists.txt' know that we
@@ -2973,7 +2978,11 @@ (define-public python-pytorch
pthreadpool
protobuf
pybind11
+ shaderc
sleef
+ glslang spirv-headers ;spirv-tools ; not sure why this is needed
+ vulkan-headers
+ vulkan-loader
xnnpack
zstd))
(propagated-inputs
FAILED: bin/scalar_test ld: /tmp/guix-build-python-pytorch-1.12.0.drv-0/source/build/lib/libtorch_cpu.so: undefined reference to `glslang::TShader::setNanMinMaxClamp(bool)'
ld: /tmp/guix-build-python-pytorch-1.12.0.drv-0/source/build/lib/libtorch_cpu.so: undefined reference to `spvtools::CreateCompactIdsPass()'
[...]
So that's why it's not enabled.
There's a shaderc_combined.pc in shaderc, but it doesn't lists glslang or spirv as dependencies, even though shaderc uses glslang and spirv data structures and functions. Errrrr. How's that supposed to work?
There are actually much less rules than "normal" programming languages. I had started to write my own basic functions that would be in a "standard library" (trim(), trunc(), remove() etc) and got the hang of it after a couple hours.
The only thing I don't like about the language for "serious" work on massive projects is missing static types (may make it less readable though..), and the tooling (most people lean on Emacs) could be better.
So I'm guessing there's some additional conventions for function naming, besides appending ? to the name for boolean return and ! for functions that mutate.
[1]https://spritely.institute/static/papers/scheme-primer.html
It can't be a yaml file because it's essentially a program, it's a lot easier to write such things as a program.
Certain sections of nix files have special meaning (config), certain functions only work in those sections. Nice features like `//` for merging dicts goes out of the window as soon as you have nested dict, which is nearly every time.
It's poorly documented, “standard library” documentation is even worse. It's impossible to navigate because everything is mashed together in `nixpkgs`. Often show no examples on how to use a function and just say `it takes blahblah`, but what is blahblah?
Ah, so Nix needs lenses.
Edit: On the NixOS matrix I was told "there is lib.recursiveUpdate" when I mentioned your comment.
Does that fix this problem for you or improve things?
That's what I'm bitching about - discovering documentation sucks because nix-lang is a very niche language and `nixpkgs.lib` is mostly undocumented. That makes finding how to do something in it is harder compared if to any other language.
With homebrew, it's just ruby scripts. You write ruby, you can ask some ruby question from someone who never wrote a formula and get an answer because that's a general purpose language.
With guix, it's not the best, but it's still just GNU Guile, it's been around for almost 30 years. Yeah, it's probably not very popular, but it's still a general purpose language that has uses outside Guix.
Keeping this in mind, I find the Guile scheme syntax used by Guix quite intuitive and pleasant. It reduces the cognitive overhead to configure the system, at least for me.
Also each configuration file in Linux has already has their differences and quirks
- the module paths a bit weird (I’d find slash or dot separated more obvious)
- some values needed to be constructed with a call to list
- weird dsl for building was mixed in there
- it’s probably not obvious what should be string/symbol/whatever
I’m not sure anything should change though. It still feels a bit weird to me that it’s a program generating the package rather than a more direct ‘just data’ format.
There's no special syntax for any of those dot slash whatever stuff. (Program item) paths are always lists (like in CSS selectors). Likewise we don't use shell strings and then have to weirdly escape them, we just use an argv list to begin with. And so on. What's the use in pretending it's something it's not?
>- some values needed to be constructed with a call to list
It depends on how good you are with quoting; if you want to avoid quasiquote, "list" is easier to understand. Otherwise, (almost) no values need to be constructed with a call to "list".
(list 1 2 3) is equal to `(1 2 3)
(list 1 2 b) is equal to `(1 2 ,b)
(list 1 'b b) is equal to `(1 b ,b)
If you mean what I think you mean then it's a program running in the build container (in an extra process) (so it's quoted in the scm file of guix).
>- it’s probably not obvious what should be string/symbol/whatever
>I’m not sure anything should change though. It still feels a bit weird to me that it’s a program generating the package rather than a more direct ‘just data’ format.
It will always end up like that, even in packaging systems that start with a "just data" config format. They will always grow their own weird language for #if, #cfg, function calls and so on in the config format. Then they have weird generator scripts generating the config files, and there they are right where we are, just shittier.
Oh, and how can a package manager avoid dealing with build systems? There are almost as many different build systems as there are packages!
I continue to find it slightly odd that the package definition is not some serialisation of a package object but rather a full program that outputs a package object, and I think this is the root of a bunch of the confusing things about the definition. Consider that the commenter at the top of this thread thought yaml might be preferable. I think that means they didn’t realise they were looking at a program outputting the package definition but rather that they thought they were just looking at the definition itself.
If the whole thing were just data instead of data that’s actually code whose evaluation produces data, I think things would be more obvious. For example, some symbols are just syntax (e.g. in the alist syntax for constructing the package definition) whereas others are references to values of which some are other packages, some are just scheme functions, and some are guix-specific functions. And symbols have other meanings in the build-steps reader macro.
Building package definitions in this way can also make errors harder to pinpoint as the author can write any program they like rather than being restricted to a more limited configuration language. Needing to evaluate programs to get packages can also, I think, make the collection of all packages as a whole pretty opaque and hard to reason about compared to e.g. being able to have a database that contains all the package definitions.
I don’t think the ‘unfamiliar with lisp’ argument is good. I’m pretty familiar with lisp (though not so much guile/scheme) and I’ve been writing it for over ten years and I think this way of specifying packages is not great in the average case, though perhaps there are some complexities that cannot be handled without this general mechanism.
As for error messages, I don’t think that using a real programming language forces the error messages to be bad, or to be too generic. If the error messages are not good it is only because nobody has put in the time and elbow grease to make them good, not because of any fundamental limitation of turing–complete languages.
I do actually prefer s-expressions to JSON, but only when used as pure data. Once you turn it into scripting language it gets ugly fast.
Which is why a good text editor will balance parentheses automatically and highlight macros. That being said, it is well known that macros can be abused and overused. It is considered bad style to use macros if a function would do the job too.
*Edit:* Guix supports JSON specifications too, though you are of course not as flexible as with Scheme: https://guix.gnu.org/manual/en/html_node/Invoking-guix-packa...
[
{
"name": "myhello",
"version": "2.10",
"source": "mirror://gnu/hello/hello-2.10.tar.gz",
"build-system": "gnu",
"arguments": {
"tests?": false
}
"home-page": "https://www.gnu.org/software/hello/",
"synopsis": "Hello, GNU world: An example GNU package",
"description": "GNU Hello prints a greeting.",
"license": "GPL-3.0+",
"native-inputs": ["gettext"]
},
{
"name": "greeter",
"version": "1.0",
"source": "https://example.com/greeter-1.0.tar.gz",
"build-system": "gnu",
"arguments": {
"test-target": "foo",
"parallel-build?": false,
},
"home-page": "https://example.com/",
"synopsis": "Greeter using GNU Hello",
"description": "This is a wrapper around GNU Hello.",
"license": "GPL-3.0+",
"inputs": ["myhello", "hello"]
}
]Which is what Guix is doing. Just look at (package), why isn't that a function taking an associated list? It's this kind of ad hoc DSL magic that makes Lisp-like languages incredible hard to read, since you never know what you are looking at.
Compare the package macro:
(package
(name "guile-sdl2")
(version "0.7.0")
(source (git-checkout (url (dirname (current-filename)))))
…)
(package
`((name . "guile-sdl2")
(version . "0.7.0")
(source . ,(git-checkout (url (dirname (current-filename)))))
…)
Neither. But that proofs my point, Lisp is unreadable for the average user. If your first instinct is to not use the language itself, but write your own DSL on top of it to make it readable, maybe that language ain't such a good choice.
Plus, you can’t assume that using JSON or YAML will give your users some kind of magical intuition bonus; you’ll still have to train them to use JSON/YAML correctly. I have a friend (who isn’t a software engineer) who needed to do some work on JSON files, and it was pretty clear that he was just cargo–culting it. He was only doing it to accomplish some larger goal, not for the joy of knowing the JSON syntax. That’s a hurdle you will face no matter what you do.
JSON would at least work consistently and don't randomly redefine the meaning of the language midway through a config file. YAML numerous issues of its own, so I'd avoid that if possible.
This isn't a Guix specific problem, the GNU project has tried to make Guile a thing for the last 20 years or so, and it just never looked especially elegant to me in any context and never really gained any real adoption in the wild either.
And when it comes to package configuration, it's just an unnecessary issue, we already had Nix, which comes with it's own JSON-like language specifically build for this task and that is much nicer to work with than Scheme, because it's configuration language first, not a general purpose programming language turned into one via macros. Using Scheme instead just doesn't improve the situation in any way in my eyes, it just creates a lot of additional problems.
Tongue in cheek :)
Why does gnu-build-system use %standard-phases (a symbol with %)? You'd think it could just use standard-phases. What is the big namespace concern. The % prefix is a Lisp convention for implementation internal symbols, not to be used by user code. Kind of like double underscore in C.
It could simply be the mplicit default. If most rules use %standard-phases, it's silly to be repeating it all over the place.
modify-phases could have a syntax like:
(modify-phases {phases-object | clause }*)
(modify-phases
(add-after 'unpack 'bootstrap
(lambda _ (invoke "sh" "bootstrap")))
About the lambda, a simple idea would be that if the registered function is a list object (pair), then it's treated as list of exec arguments:
(modify-phases
(add-after 'unpack 'bootstrap '("sh" "bootstrap")))
A macro like (cmdf "sh bootstrap") could do the parsing and generate the lambda + invoke. The "f" reminds you that it doesn't run the command but returns a function.
> Why does gnu-build-system use %standard-phases (a symbol with %)
AFAIK %symbol indicates a constant in Scheme.
It's unfortunate that it isn't written in Common Lisp. CLOS would've been immensely useful there instead of the ad-hoc object system via Scheme records.
AFAIU, asdf is for providing particular versions of tools (e.g. python, etc.). In the original post, guix is also used to provide library dependencies.
Guix is the free software oriented, lisp flavored cousin of Nix. -- I've used Nix. It's got enough rough edges that writing nix code is pretty much only accessible to enthusiasts.. though, those who can get past the rough edges do recommend it.
The added complexity of guix/nix allow for having an OS distribution where the whole OS config is declared/reproducible.