GNU Autotools: A Tutorial [pdf]
elinux.org
elinux.org
The video shows an animation about how the autotools work (which I think helps understanding), and has a CLI demo of how to use it, all for the low price of free :-).
I thought I should be scared, and m4 looks daunting, but what they at least get right is that it's trivial to generate both shared and static libraries, cross-compilation is a breeze, and in the end all you need is a shell to configure and make to install.
CMake defaults to either shared or static libs, and sometimes you have to read the docs for a project to figure out what custom variables influence this; and cross-compilation is almost always guaranteed not to work because some find_x cmake script tries to compile and run a C source to figure out certain properties about the target (but it confuses that with the host system).
Some systems seem to try to absolutely minimize the amount of configuration code which is required for simple things, using "automatic configuration". If this is for clearly defined things like configuring a library to be static, this is fine. But when it becomes unclear and ambiguous what actually happens, or, worse, when you can't say what versions of dependencies are actually used, this is too much magic. And also when one needs twenty minutes or more to figure out how to switch on address sanitizer, or use a specific language standard. And this becomes worse when the configuration becomes more automagical with each iteration of the build software and the constructs and semantics start to depend on which version of the tool one is using.
autotools has the advantage that these things are quite explicit, and the configuration interface is very stable, and this helps to understand and maintain a system.
https://old.reddit.com/r/programming/comments/a02n5/trying_q...
This is all one needs to start a project. I actually think it is a good idea to use it quite regularly for hobby projects as well, just like "git init ." .
Recently, I got the job of supporting dozens of packages in an organizational environment. The first choice was cmake with Conan, but I found both to be quite ambiguous defined and it was really hard to figure out what was going on and how to get things to build. Then I found the autotools tutorials I linked before, and I needed less than one day to get going with autotools - and I find the build scripts easy to debug.
I think this is a larger issue with CMake - it has a lot of magic going on, but always leaving the user with fuzzy definitions and a really unclear concept of what is going on. And while I like the kind of magic which happens inside Emacs, for example, when it does just the right thing when I move the cursor around, I had this kind if magic which leaves me wondering what is going on and unable to fix issues based on systematic knowledge. I find it really, really time-consuming to work with such systems.
Another really nice, more detailed introduction is this one:
Probably the best technical book I bought in the last year. After the lecture, I have put aside all whispered-in prejudices and am convinced that it will still be worth betting on GNU autotools in the 2020s.
Support for Ninja files also seems to speed up builds in my experience. (Ninja is a Makefile "assembly language", minimal features, assuming that things like CMake files compile into Ninja). Ninja is fully multithreaded and focused on speed instead of features, and it really does show.
Since ./configure is single-threaded, a substantial amount of time needs to be spent whenever you make changes to your Autoconf / Automake stuff.
CMake has a few nice things: a built in basic testing framework (just interpreting the exit codes of programs: 0 is success and anything else is test-failure). Said testing framework has been integrated into various other test frameworks (like Googletest: https://cmake.org/cmake/help/latest/module/GoogleTest.html).
You can also run `make distcheck` to generate a source tarball, extract/build the tarball in isolation, and run `make check` on the result. A very handy way to make sure that you've packaged everything correctly, and everything builds/tests OK when executed out-of-tree.
I get that this is especially attractive for some companies earning money with Windows. But how important are contributions from developers on Windows for Linux distributions and the free software community? I think that most companies which use open source libraries on Windows are unlikely to give anything back.
Also, when it comes to things like networking and server software, Windows has many important differences, and is not that relevant at all any more. Well, every project will need to figure out on its own whether supporting Windows builds is worth the time and hassle.
But actually, one of the points made in the Calcote book really stuck with me: Build tools are not just about developer convenience; they're also about the user's convenience. It's more work on my end, but my users (who, admittedly, are on UNIX-likes) know to type:
./configure
make
From their standpoint, it just works as expected.It is astounding how many tools I have to install to get most tools to work. :(
The C and C++ ecosystem is crying out for a new build system, but there is no mechanism to converge on one. It's bad.
There is one alternative to make which I think is worth mentioning, because of its simplicity, brilliance, and excellent support for correctness: it is redo, as in Apenwarrs redo:
https://redo.readthedocs.io/en/latest/
And redo works with autotools!
Basically, autoconf generates a makefile which will work for your architecture. It does that via a script which is called "configure", which generates that makefile. The configure script is independent of your hardware and platform and does only require a standard POSIX shell, and it generates code for the version of "make" that is present on your platform. This means that in difference to other build systems, one does not have to install (and build!) the build system for the own platform, and also that there are no compatibility problems because the configure script always matches the source code distribution.
On top of that, the "configure" script is also automatically generated, typically using two files "config.ac" and "makefile.am", via a command which is now called "autoreconf", but this normally happens only on the developer's system.
This is also quite important when you happen to work on larger projects with many dozens or hundreds of modules and you need to build a specific part weeks after you have been looking at them. To have to look up every time how to build something is more than a nuisance.
- the macro language it uses is strongly standardized, so it produces identical semantics on all platforms. And because it uses macros, which are expanded into shell code, it does not require the macro interpreter to be present on the target systems neither, only the generated script.
- the configure script is included in the source distribution and as such "frozen", so one has no problem with changes or incompatible upgrades which could break builds.
- the philosophy of autoconf is based on feature testing, for example if a given compiler version produces code with a certain property, if a library with the name "foo" contains a function with the name "bar", and so on. This is the right way. It is also good for providing upgrades between large libraries, like Qt.
- testing for features individually is the only way to escape a nuclear-scale combinatorial explosion of feature combinations.
- feature testing also allows for support of evolving software, new functions that are buggy first and need to be worked around, then gradually fixed and turning perhaps into a special library, then perhaps an element in Boost, later you need perhaps a specific version of boost, and so on. This is well-suited to the bazaar-style development which is characteristic for Unix, with an open interchange and borrowing of ideas.
- as already mentioned, it is very well documented. People who just copy-paste existing code are doing it wrong.
- it is understandable, without too much magic build into it. cmake is affected by an excess of magic IMO - it is hard to understand what is going on, and as a consequence, it is hard to fix a failing script. In my experience, a fraction of the time was needed to come up to speed in comparison to learning only some of cmake.
- This also means that it is easier to maintain for a medium-to-long-term future. It is nuts to use a language which is constantly changing precisely for infrastructure- some poor people will have to read and maintain all that code years later! (And, this people could even be you!).
- You only need to support platforms and features you need, at the same time you are able to support a myriad of exotic platforms if you want it. That means that if you want your software to work only on Ubuntu 64 bit Linux, you only need a very small configure.ac script, but if you want to support older libraries on Debian, you can do that easily as well, and if you need to support an exotic architecture with 36 bit-words or a 16-bit embedded MCU, you can do that as well.
- It is so widely used so that almost every Unix developer knows how to build software with it, without much thinking.
- It tries to do one thing, and do it well, providing platform-specific build configuration. It does not try to do dependency management or package management, leaving that to specific package managers.
- and because of the previous point, it also works well with new distributions and package managers, like Arch or NixOS or GNU Guix - it does not interfere with them, and does not fight for the position to be the top dog in a packaging system.
- Both autoconf and make are, while supporting C and C++, not language-specific, so you can easily use it to distribute things like LaTeX documentation, pictures, or even fonts.
- you can use and generate a mix of shared and static libraries, and also use both pkg-conf as well as standard paths (or, for example, paths provided by GNU Guix). And it can use that for each package individually. In contrast, cmake find_package commands do have difficulties with mixing different retrieval mechanisms.
- Supporting all kinds of hardware and compilers will continue to be important. While there are without question obsolete platforms, it will be able to support different hardware architectures in the future, like CUDA on GPUs, or ARM or RISC architectures.
- It works for any library without requiring specific upstream code. This is in stark contrast to cmake, which often needs supporting find_package commands or target definitions for specific libraries. The latter means ultimately that cmake would need to become a meta-distribution which requires specific support for many packages one wants to use. I think that as the complexity of software continues to grow, this approach will probably not scale.
http://guix.gnu.org/cookbook/en/guix-cookbook.html#Advanced-...
With this, you can create isolated environments in which you can work on specific software, and use and go back to arbitrary versions, without cluttering the base system with local installations. One of the best features is that defining an additional package oneself is merely an extension of the package definitions available with Guix:
https://guix.gnu.org/cookbook/en/html_node/Packaging-Tutoria...
And, of course, it works well with autotools....
One of the advantages is that this functional package definitions define very clearly what is provided by a package. It also works well for other languages like Rust or Clojure.
I've done the whole custom makefiles as well, but by the time you have tests, installation, etc it just becomes a burden to maintain them.
Wouldn't it just be better to provide the source files (autoconf.ac and Makefile.am) and instruct developers to run "autoreconf" to generate the rest? That would it also make much clearer which files need to be changed if e.g. source files are added.
The whole point of autotools is that the output (./configure) is highly portable and can be run on basically any mostly-POSIX-compliant /bin/sh. You don't need to install anything beyond sh and make just to perform the build - compare with a design like Bazel where you need a piece of software to do the build.
If the generated files change rarely, checking in and committing the generated files means people who are just making code changes (or no changes at all) don't need to install autotools.
Also, there are functionality changes in certain versions of autotools; you might want to make sure your users get a feature that was implemented in a recent version of autotools, but support users who are running on an older OS that doesn't have that version packaged.
(If you expect the git repo to be primarily used by developers / contributors instead of end users who are just building it, then you should definitely gitgnore the generated files, but also make sure to include them in the release tarballs. "make dist" will do it. You should also guide end users to download that release tarball instead of an auto-generated tarball of your git sources - GitHub and GitLab both have support for file attachments in their release feature, these days.)
In our modern world, this might be less important because installing software (even specific versions of software) is a lot easier. But then our modern world probably deserves something other than autotools, because a lot of what makes autotools hard to work with is the fact that it's intended to translate down to sh/make. (Options I'm aware of that look fairly reasonable include Meson/Ninja, CMake, and I guess Bazel.)
you will sometimes run into subtle forward porting issues doing this.
that's why not doing this and using the generated Makefile and ./configure is a good idea (tm) for most consumers of packages.
in other words: if you're in for adventure and learning, go for autoreconf. https://wiki.debian.org/Autoreconf
CMake or Meson have both learned the lessons of Autotools and so you dont need to subject yourself to them anymore.
Why exactly is it getting harder?
`cmake .. -GMakefile` not enough for you?
anyway, the particular case here is something like an academic computing cluster, where you don't have administrator access, and you want to compile some package that will install into some place like $HOME/project/usr. The system has no meson or bazel or scons and only some older version of cmake that may not meet the claimed minimum requirement of the random package you want to compile.
Or it's some SBC that you dare not upgrade the operating system on because you don't know if there is some breaking kernel change with some hardware interface.
Sure, locally installing a new version of cmake or some other build system is not too hard, but it's certainly additional friction.
for cmake it's basically
wget https://github.com/Kitware/CMake/releases/download/v3.20.0/cmake-3.20.0-linux-x86_64.tar.gz
tar xaf cmake-3.20.0-linux-x86_64.tar.gz
export PATH=$PWD/cmake-3.20.0/bin:$PATH
definitely simpler than installing cygwin, msys or wsl :-) (and some software requires cl.exe on windows anyways)https://youtu.be/8Ut9o4OdSC0?t=459
In this section, he kind of complains about the number of files which are needed to use an autoconf install. But the thing is, almost all of these are generated files. So, one does not have to create them - they are created by autoconf, once, by the developer, and stored together with the rest of the source distribution for installation.
Then, in the seconds after, at
https://youtu.be/8Ut9o4OdSC0?t=582
he compares the speed of autoconf to CMake, and CMake comes out faster. But, well, this time, he also adds to the total time the time which is needed to run autoconf for the first time, which generates the above files. So, in practice, a developer which changes source files, or a user which installs a package, does not need to run the autoconf generation of the configure script, and a developer which does imporovements and testing in the source code also does not need to build the configure script each time. To sum up, at the one hand side, he counts the generated autoconf files into the number of files needed for a package, but then he counts again the generation time into the run time to use autoconf regularly. And this is a bit weird, because the configure script and the other files only have to be generated once when a change to the build system is made, not for install builds and not for normal development builds.
I agree that autoconf could run a tad faster and that generated files could be organized a bit tidier (it is possible to configure it so that it puts them into a subfolder). But I think the comparisonn in the video is a little bit biased.
And of course, a company needs to make money and sell its products, but if a presentation of a core product starts with such a biased comparison, it is not very convincing to me. Well, we'll see where CMake is in five years time - build systems are extremely complex and large things and need to account for a lot of stuff which is rarely used to work robustly, and then it is perhaps coming to an age where things like good architecture, good design, and clean code mattes. I am not worried that CMake does not manage to do highly automated builds for simple projects and happy paths in the most popular platforms, but how it copes with the hard stuff on which a lot of computing infrastructure depends. There is a good reason why most of this infrastructure stuff, be it the kernel, or things like glibc, or OpenSSL, is not made by companies.
In my early days of programming, I thought Autotools was the coolest thing in the whole world. It was essentially magic.
These days, with the demand for running on non-Linux UNIX at an all time low, why put yourself through all this?
This comes from someone who has invested a significant amount of time trying to implement pretty much exactly what PHK is advocating: https://github.com/wahern/autoguess I don't reach out for autoconf reflexively, but I fully appreciate why people do make use of it.
You can get it all, quickly and cleanly.
https://lwn.net/Articles/706404/
https://thiblahute.github.io/jpakkane.github.io/Simple-compa...
The thing is apparently, ninja is indeed faster, but this matters only for very large projects with tens of thousands of files.
Speed of incremental builds matters, and in this respect, ninja is an advance, but is not a day-and-night difference either.
What it matters for developers is the development time, and doing things in automake takes 3-4 times of Autotools. Not to mention that the configuration time you loose waiting for a single line change.