GNU Make Guile Integration | Hacker News Reader

GNU Make Guile Integration

gnu.org

117 pointsgebt30 comments

GNU Make's Guile integration is such a cool feature.

The one hiccup with it is that isn't included in the default GNU Make package on any system I've ever used (although it can be installed as make-guile in a few distros).

I'm guessing it's because Guile is around the same installed size as Python, and it needs a lot of external dependencies - one of which (the garbage-collection library) is a little oddball. A tool as foundational as Make probably doesn't want to pull in a huge dependency tree if avoidable.

If we want to see Guile support added to Make by default, we'll need to reduce or eliminate Guile's external dependencies and slim down the install.

  : fred@tara ~ $; cat /etc/fedora-release 
  Fedora release 31 (Thirty One)
  : fred@tara ~ $; make -p | grep FEAT
  make: *** No targets specified and no makefile found.     Stop.
  .FEATURES := target-specific order-only second-expansion else-if shortest-stem undefine oneshell archives jobserver output-sync check-symlink guile load
  : fred@tara ~ $;

guile available -- so, its enabled in Fedora.

  : fred@tara ~ $; ls -l `which make`
  -rwxr-xr-x. 1 root root 248624 Dec 10 17:34 /usr/bin/make
  : fred@tara ~ $;

and, just to validate guile availability:

  : fred@tara ~ $; ldd `which make`
   linux-vdso.so.1 (0x00007ffce61cb000)
   libguile-2.2.so.1 => /lib64/libguile-2.2.so.1 (0x00007fc423432000)
   libdl.so.2 => /lib64/libdl.so.2 (0x00007fc42342b000)
   libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc423409000)
   libc.so.6 => /lib64/libc.so.6 (0x00007fc423240000)
   libgc.so.1 => /lib64/libgc.so.1 (0x00007fc4230e1000)
   libffi.so.6 => /lib64/libffi.so.6 (0x00007fc4230d6000)
   libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fc422f50000)
   libgmp.so.10 => /lib64/libgmp.so.10 (0x00007fc422ed3000)
   libltdl.so.7 => /lib64/libltdl.so.7 (0x00007fc422ec7000)
   libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fc422e8c000)
   libm.so.6 => /lib64/libm.so.6 (0x00007fc422d46000)
   /lib64/ld-linux-x86-64.so.2 (0x00007fc4235ed000)
   libatomic_ops.so.1 => /lib64/libatomic_ops.so.1 (0x00007fc422d41000)
   libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc422d25000)
  : fred@tara ~ $;

Yes, libgc and libgmp are needed (182k for libgc, 1.3mb for libgmp, 1.3mb for libguile itself) as opposed to 3.3m for python3. Seems comparable, including dependencies.

Awesome, I didn't know it had landed in F31. Thanks for the comment.

Re: size, I'm surprised by the library size. Is it possible that the library needs more files than just the .so? The guile-2.2-libs package is around 40-43 MB on the current Ubuntu LTS, plus a few more MB for supporting Guile packages.

(source: https://packages.ubuntu.com/bionic/guile-2.2-libs)

ubuntu LTS's make is a lot leaner:

    $ ldd `which make`
        linux-vdso.so.1 (0x00007ffff9e87000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f27cd690000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f27cd290000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f27cdc00000)
    $ ls -l `which make`
    -rwxr-xr-x 1 root root 222792 Apr 17  2018 /usr/bin/make
    $ make -p |grep FEAT
    make: *** No targets specified and no makefile found. Stop.
    .FEATURES := target-specific order-only second-expansion else-if shortest-stem 
    undefine oneshell archives jobserver output-sync check-symlink load

It's enabled in the `make` package on Arch Linux.

My Guile is around 40MB; my Python is around 80MB--both gargantuan compared to the 1.5MB Make.

What exactly is "oddball" about BDW GC?

What isn't oddball about it?

It's designed to work as a garbage-collected replacement for C's malloc. That alone is enough to establish its oddballness.

Any large system that makes the mistake of using it soon discovers its limitations. Not that it's a bad system per se - it achieves its design goals, it's just that its design goals are, to put it mildly, oddball.

Basically, it's designed to allow systems that really should be written in a garbage-collected language, to kinda-sorta get away without that - up to a point.

You should not think of it as malloc replacement but as an GC that can work with native code that does not use fixed stack and register layout. That you can use such a thing for code generated by C compiler is side effect of that. In fact, there are various garbage collected runtimes with bespoke GC's that use similar implementation strategies.

Edit: and one notable thing on BDW GC is its non-oddballness when it comes to building and/or packaging it. Also it is almost completely written in portable-ish C (it expects sane memory layout and relies on bunch of technically unspecified, but expected behaviors).

The point is that it's designed to work with native code in a language that wasn't designed to use garbage collection. That's going to result in a semantic gap no matter what, and it invariably does.

Every system I've ever seen using Boehm GC has issues as a result, except in certain special circumstances where the limitations are mitigated by the nature of the system. That's not an indictment of the GC itself, it's simply a consequence of trying to shoehorn automatic GC into a language that doesn't support it.

It also allows language implementations to get a GC that, while not perfect, is "good enough" in many cases - and can get them through the early growth period. The library has APIs to manually manage roots etc for this purpose.

I remember that Mono used it early on. Not really surprising that Guile does, as well. Nor is it surprising that they haven't switched to a better hand-rolled GC - given the relative obscurity of Guile, I would imagine that its maintainers have more important things to work on.

The license is a little weird, for one. It's also strange because I think it's the only non-GNU dependency in the whole chain. I guess I'd also have expected a GC scheme to be baked into the core Guile source-code instead of relying on an external GC library. It's also not fully portable C code.

I'm sure the library works great - it has a long history, and I'm definitely not knocking it. But it's a little bit of an unexpected dependency for sure.

Is it cheating if we count GNU Guix?

I have used this in one of my projects, but in the end had to abandon it: the problem is that Guile itself requires GNU make during build, which creates a circular dependency, and thus building it with e.g. FreeBSD ports required manual intervention every time you updated it. This combined with the fact that this integration is only available since GNU make 4.0, and some Linux distributions still have older make, made it a hassle to setup. This should improve over time, of course.

Still a great feature though. Without it there are e.g. manipulations on the file paths that you can not do at all, or can do but only as a hack. Case in point: "GNU Make Standard Library" (https://gmsl.sourceforge.io/) is a collections of such hacks; here's its definition of 'strlen':

    strlen = $(call assert_no_dollar,$0,$1)$(strip $(eval __temp := $(subst $(__gmsl_space),x,$1))$(foreach a,$(__gmsl_characters),$(eval __temp := $$(subst $$a,x,$(__temp))))$(eval __temp := $(subst x,x ,$(__temp)))$(words $(__temp)))

> only available since GNU make 4.0, and some Linux distributions still have older make, made it a hassle to setup. This should improve over time, of course.

That feels like an outdated statement, make 4.0 came out in 2013. The only non-EOL system I can think of off the top of my head that isn't shipping >=4.0 is macOS (because they won't upgrade to anything GPLv3).

However, unfortunately there's a popular CI system (cough CircleCI cough) that's still using Ubuntu 14.04 by default, which has been EOL for almost a year now. (And others, such as Travis-CI, used it by default right up until it was EOL).

Oh, if you appreciate gmsl, you might also appreciate a book by the same author ('jgrahamc), The GNU Make Book. Some of the things in gmsl are certainly hacks, but a lot of it is quite reasonable if you start reading Make macros as "lisp, but with a bunch of extra dollar-signs and commas".

> The only non-EOL system I can think of off the top of my head that isn't shipping >=4.0 is macOS (because they won't upgrade to anything GPLv3).

RHEL 7 has Make 3.82. Its regular life cycle ends in 2024, and I assume it will have an extended support period after that.

TIL that after the 5 year "full support" period (which is what I had in my head), RHEL has a 5 year "maintenance support" period (and that CentOS matches that 10-year total) (and then the "extended lifecycle support" arbitrarily long after that for whoever wants to pay enough).

(Aside: 3.82 is a surprising version to be stuck with... most distros went straight from 3.81 to 4.0, because by the time the show-stopper bugs in 3.82 were all patched, 4.0 was happening.)

Scientific Linux 6.10 has make 3.81, CentOS 7.7 has 3.82. These long-term-support distros have old software by their nature, and sometimes you're just stuck with one of them unfortunately.

Guile integration in Make is super useful. I've used it on Windows to replace expensive shell calls with Guile functions and achieved tremendous speed-up. Building Make with Guile-support on Windows seems to be pretty complicated, but luckily ezwinports provides a native build, and it works great.

For those of us stuck working with vanilla GNU Make, you can still use $(shell) to accomplish a lot of what the Guile integration provides.

As an example, string-length can farmed out by something like:

    strlen = $(shell printf "$1" | wc -c)

When combined with $(eval), you can use the $(shell) command to do some pretty complex stuff in your Makefile. I've used this for doing checksum-based dependencies on one oddball build.

It's not as nice as working with Guile, but it gets the job done in a pinch. And since the Guile interface doesn't have direct hooks into the DAG or anything, it doesn't really have access to tools that you couldn't provide by calling $(eval) on the result of a shell script.

Can someone who has used this featured give an example of what they used it for and how it improved their build system?

With a bit of creativity, you can achieve most things with the existing GNU make functions. Often the result is far from readable and relies on side-effects like sorting removing duplicates. But at least you don't need to worry about make being built with guile support.

Guile does give you access to most system calls. The most interesting thing I've done with this is have the gmake process create a fifo, fork and exec tee in the child process to save a copy of the build output. It doesn't entirely work if gmake re-execs itself after building included makefiles.

I've also tried doing a loadable module in C - to output timing to benchmark changes to my Makefile. I have a non-recursive setup which can take a few seconds at startup.

The guile scripts are mostly limited to running during the initial make stage where dependencies are all gathered. You can't avoid the shell forks during the build process. There are also limitations in the strings being passed back to gmake so it didn't help greatly with colourising output when I tried that.

make -p -f/dev/null | grep FEATURES

Thank you kindly for providing the way to actually do the documented: "You can determine if make contains support for Guile by examining the .FEATURES variable; it will contain the word guile if Guile support is available."

Unix - the documentation is excellent if you already know how to do it...

Seriously? Is it really that hard to figure out how to look at values of variables in general?

I have been using it with great success, but there has been some bother with portability.

I used it in ways that could have been done with make proper, but since I already know a lot of guile I reached for that.

Which platform did Guile have trouble on?

Not cross-platform, but cross-computer. It isn't available everywhere, and is quite a fuzz to enable elsewhere.

For those of you who don't have make-guile, your GNU make probably still supports loadable modules in C (or your favorite C-ABI-compatible language): https://www.gnu.org/software/make/manual/html_node/Loading-O...

I've been using that at work recently.

If anyone is interested in using a Scheme-like language with Make 3.81, take a look at: https://github.com/bhk/scam

Oh, like MSBuild with Powershell scripts. Each evil in their own ways.

Reply on news.ycombinator.com