Stop the autoconf insanity – Why we need a new build system (2003)
freecode.com
freecode.com
All the complaints in the article are PRECISELY the kinds of complaints that autoconf programmers worked very hard to avoid. For example, it's IMPOSSIBLE to have version skew with autoconf because the configure script is completely pre-generated before being shipped. As far as the user is concerned it's a simple sh script.
ALL of these problems stem from the "automake" monstrosity that creates many more problems than it ever solved. Unfortunately people now tar autoconf with the same brush as automake and libtool whenever they run into problems with these. Sigh.
http://mosermichael.github.io/cstuff/all/projects/2011/06/17...
The catch is of course most cases ; other stuff is still possible to do , but things become hacky.
Another downside is cross compilation, my system does not do this effectively (though it works for Linux i686, x86_64 and Cygwin - many surprises between these setups !). Most build systems I know don't support cross compilation - except for automake, the terrible.
If you do debian packages then you are strongly encouraged to use automake - because of cross compilation.
The problem with build systems is they don't cover the entire scope required to actually build software reliably. Ideally, you want to take just your code as input and produce a target, but the reality is that you're taking your code, plus an environment with potentially infinite number of configuration options, and you're asking the build system to produce a target. This is effectively saying "Here's some code and some shit, please do your wizardry and make me something that looks like this." Is it any wonder that every proposed solution to this difficult (impossible?) problem gets it wrong?
The correct solution is to declare the target you want, then build up the dependencies you need as input to meet that target and declare them explicitly, then perform the build in that isolated environment to ensure that no undeclared configuration options can alter the result of the build. This is what Nix (http://nixos.org/nix/) does, and by derivation, Guix (https://www.gnu.org/software/guix/), and Debian's ReproducibleBuilds is also attempting. The build process becomes effectively a pure function, free of unwanted side-effects.
Nix doesn't actually perform builds in itself, except for trivial ./configure && make. It piggy-backs on existing build systems via bash scripts if needed, but it completely manages the environment in which such script will run, so it has local-side effects which are controlled, similar to how you might use the ST monad in place of IO in Haskell perhaps.
I find Guix a bit more interesting because it can take on the whole problem - including the build system. While it can piggy-back on existing build systems in the same way Nix can, you can also write your own in Guile (or some other language and invoke from Guile). It doesn't really make sense to depend on the old cruft under this new mentality, because things like pkg-config are obsoleted by known, explicitly declared dependencies. The build process can be simplified (and probably sped-up).
Nix itself should work on BSD (I think it's tested on FreeBSD), but Guix is linux-only so far, as it is still in early development. I'm pretty sure Guix intends to be fully portable, even to Hurd in the long term. The ideas behind Nix is platform independent though - there shouldn't be any reason it can't be ported to other platforms.
As for building individual software on other platforms - they require different targets. (Or in fact, just an alternative configuration of the same piece of software on the same platform requires a different target). This is intentional though - targets have an identity (a SHA1 of their entire inputs), so any new platform will require a new package.
The package configuration format is quite useful in that regard though, as you can derive new package definitions from existing ones, so it should be possible to make partial "generic" configurations, then derive the platform specific targets from them. There's quite a bit of configuration for each platform though, as the dependencies will all have different identities, all the way down to the compiler and the kernel.
Another potential option would be to create a template system to automatically generate the Nix configuration for multiple platforms, which would be somewhat similar to what autotools et al are doing today. The difference being, once one person has performed a build and tested it for a platform, anyone else should be able to reproduce it exactly, or to explicitly specify where he wants the target to differ.
In effect, Nix removes the "hidden knowledge" that goes into building software by requiring that you specify even the commands you used to perform the build.
Of course, Nix isn't a panacea - it introduces a new set of problems, including some social ones. It might take more effort to write portable targets for each platform - but for that trade off you get reliability and simplicity for the users of the software.
This is simply incorrect. If you already have a configure script, then you don't need autoconf installed at all, let alone a particular version of it. autoconf is used for generating the configure script, not running it.
To be fair, I've only actually seen this on scripts I was writing - never seen it happen to something that passed make dist.
I don't quite remember what the default is if you don't include AM_MAINTAINER_MODE in your configure.ac. So yes, you could have indeed seen that happen, but there are ways to make it not happen.
It can do everything autotools+make can and you only have to code your build in Python rather than Makefile + M4 + shell scripts. I've been using it for many years and it is technically superior to any build system I've seen. Shame the author of it doesn't market it more -- with some hype it would easily have been the de facto standard build system by now.
GNU make is actually two different languages coexisting in the same file (the makefile), a declarative language for describing the targets and their dependencies as well as a functional language for the variables/macros. Once you realise this, a whole new world of possibilities opens up. If more people took the time to actually understand proper use of makefiles, perhaps we'd see fewer poorly reinvented wheels.
Make is a great system for declaring dependencies - the basic syntax is wonderfully concise (not that that's everything) and clear, and the system is a great platform on which you can build your build system (unlike Scons which tries to abstract away all sorts of things and just ends up hiding them).
The problem is that when you are trying to do anything more complicated than simple rules you are lacking basic language features. You don't even get function calls - the templates work under many circumstances, but not all (and are a pain to debug). Also don't get me started on invisible syntax errors (tabs vs spaces), and the only error message Make has' Missing separator. Stop'.
Writing a build system in Python loses you that concise declaration syntax (but again characters typed isn't everything, as long as you maintain clarity it doesn't matter), but gets you ugh more flexibility to produce rules under complex circumstances, which saw at you need for portability.
Firstly, you call it "make script," suggesting that you view makefiles as a typical (procedural) script that is executed from top to bottom. I base this on my experiences working closely with others who also used the term "make script," all of whom suffered from this misconception.
Secondly, GNU make does have function calls. Look up the $(call ...) construct. When the built-in functionality is insufficient (and of course such cases come up), it is trivial to call an external script, written in your language of choice, using either the $(shell ...) syntax or as part of a target recipe.
As for the "Missing separator. Stop." it is not fair to dismiss an entire tool because of one slightly obscure error message.
I also find it ironic that you complain about tabs vs spaces, then go on to suggest using python, which also has invisible whitespace as part of its syntax.
Make's rule system completely breaks down when you need to create a zip file of all *.c files in a directory tree. But it's a side-point, for complex software, build variants, configuration, installation and distribution are much harder problems to solve than dependency-based compilation. Waf is a replacement for the whole autotools suite, not for GNU make.
Want a zip file? Use a command like "git archive -o foo.zip HEAD".
Why do you insist on having a single tool do all your tasks? A good craftsman has an arsenal of tools and picks the right one for each job.
What? Creating a compressed archive is exactly the sort of thing that Make was designed for. A binary file or library, if you squint the right way, is just an archive of object files, which are just transformations of .c files.
Make's biggest problems for me (and the reason that I'm slowly gravitating towards writing a replacement for my own purposes) come from the fact that it can't handle inferring dependency graphs correctly. The usual 'gcc -M' hacks that work for C source don't work for code where dependency tracking is more than just an optimization. For C, remember, as long as you have the appropriate headers and .c files, you can build a .o file completely order-independent.
If you have a dependency digraph that looks like this:
a -> b, c, d
b -> c, d
c -> d
d
d, c, b, a
depfile: $(SOURCES)
update-deps -o $@ $?
include depfile
Regarding your chained dependencies, how do you expect _any_ tool to figure this out without either being told or examining the inputs (which implies knowledge of their formats)? Relegating this to an external tool seems perfectly appropriate to me.
zipfile.zip: $(wildcard dir/*.c)
zip -ur $@ $?
2-3. This command does what you want: zip -R -FS foo.zip '*.c'
4. In practice timestamps work just fine unless you deliberately sabotage them. To use content hashes, the hashes of the input files when creating a target would need to be stored somewhere for future reference. This storage place could just as easily be corrupted. Make is a build tool, not a VCS. Treat it as such.
Yes, some things are specific to GNU make, but it is the de facto standard, something waf can only dream of. If you're going to complain about something being non-standard, your suggested alternative should be more of a standard, not less.
If you want bz2, why did you ask for zip?
I blame make's awful syntax.
I hate using autotools, but every time I touch a project that uses something else (and that is too complex for a ten-line Makefile), I inevitably end up wishing that it used autotools. I suppose it shows that sometimes a few decades of polishing a turd does sort of work.
I don't think there will ever be a standard build tool, simply shunts to get the needed functionality from others. Eventually all build tools will exist in each other and the only interface will be recursion.
I especially like that one file allows me to develop on XCode, somebody else on Visual Studio and finally deploy on Linux/gcc. One config file creates native projects on all 3 platforms, no manual hacking.
Makefiles work fine in the trivial case, the problem is that things quickly become complex. Automating that complexity seems like it should be easy but, as it turns out, it isn't.
There should be a way to $FOO. Well, there's a way to do it in a Makefile but figuring that out is harder than should be, and doesn't make sense when you finally do figure it out.
"Well that way is stupid. Here's how I'd do it:"
I like cmake, and used to use it exclusively, but after running into so many issues with their homebrew scripting/configuration language, I've mostly moved on. I wouldn't be surprised if I end up coming back, and I'd encourage everybody to give it a shot (the ease of typical simple Makefile generation, but with extra muscle), but the omission of Tcl as a control/config tool has always baffled me.
[0] http://en.wikipedia.org/wiki/CMake#History
[1] http://en.wikipedia.org/wiki/Insight_Segmentation_and_Registration_Toolkit
I've concluded that you can either embrace highly complex evolving systems and build tools which follow that evolution in such a way as to provide functions. Or you can make "point in time" sorts of things that are ephemeral in their ability to do what they do. Not a good choice but the only two that seem to be durable.
I completely agree that it is a challenging place to be.
Yet, there is one problem which has cost me lots of money for buying beer and getting myself filled up: why can't auto* check for all libraries and then output an aggregated "libx,liby and libz missing, libfoo outdated" instead of the configure - apt-get - configure - apt-get cycle?
e.g. no matter what your preferred source control tool is I think we can all agree that there are some pretty good options out there these days. compare that to the insanity that prevails in packaging in every corner.
Of course, I'm not talking about IDEs or web-based editors. You're right though, I do enjoy the innovation in web editors.
1. Out-of-the-box integration with modern toolsets (0-configuration documentation, smart autocomplete, build, debug, jump-to-definition, etc)
2. Present an interface that is easily discoverable by GUI-minded folk
3. Take advantage of the opportunities afforded by non-textual displays
but each of these is done "poorly" in E&V for a good reason, as per your text editor / IDE distinction. They can't change these things without stepping on the toes of their power users (i.e. all of them) and leaving behind part of what made them great, which is why this kind of innovation is happening away from E&V. Most of the apps I listed address between 1 and 3 of the above issues. Just because the young upstart editors don't support keyboard input and customization with the fluidity of E&V doesn't mean they are purely derivative. They do something that a lot of people want done that E&V don't do, and they do it well. That's what innovation is all about.
So I ask again what it is that you would consider innovation and why the projects I listed don't count.
I'll repeat that we're partly talking past each other, but you are correct that the kind of (valuable! useful!) innovations you list tend to happen by an upstart and then are copied into other programs. So it is healthy to have a field of contenders rather than two dinosaurs.
You're also pitching this as some kind of argument, which I think is your mistake.
Never did I say that people shouldn't make new text editors. I remarked that some problems seem to attract developers, and some don't. I think that structural issue is more interesting than editor feature evolution.
tl/dr: -DSL for builds with well defined semantics, leverages existing toolchain
-design constraints for the system that make sense, i.e. speed, portability, usability and common sense
-does not try to reinvent the wheel but rather simplify the usage of existing tools
- the build config it generates on my Ubuntu box seems to really deliver on the speed promise - I have no benchmarks to quote but a small c++ codebase using all sorts of dependencies including boost compiled nearly instantly
followed by
the auto tools are constantly a thorn in the side of users and developers alike.
Which is it? Do the autotools "usually [result] in a working installation" or are they "constantly a thorn in [our sides]?"
But on a serious note, I really want to see Gyp become more popular, for the simple reason that integrating multiple Gyp projects is essentially zero-effort.
It's a beautiful way of working, even though Gyp certainly has room for improvement.
The modern approach is: apt-get install (or your OS equivalent).
It's 2014. I can't remember the last time I needed to install a package from source. Worst case scenario I need to add a repo.
Regular users are not supposed to install software like this. The only sane way to administer a distribution is to make sure everything passes through the official package manager. This leaves the developers and package maintainers to deal with autotools and I don't hear them complaining (very loud[1]).
Autotools is simply good enough for the job and instead of being some sort of hated legacy it's being sometimes preferred for new projects (cough [2] cough).
[1]: https://blog.flameeyes.eu/tag/autotoolsmythbuster/
[2]: https://github.com/stefantalpalaru/vala-skeleton-autotools
Installing software from source is a good thing, you just need to go through the intermediate process of creating a package for your distribution first.
This is not always the case in reality, however.
- The versions in the repositories are always up-to-date with the latest numbered version.
- Users never want/need any features or bugfixes that haven't made it into the latest numbered version.
- Nobody ever needs to install an older version of anything, because new versions never introduce compatibility-breaking changes. Which of course are never necessary anyway, because software designers always have perfect foresight, so the initial design of every package is always something that will remain perfectly suited to it forever.
The only downside to that is that it's more work to create new packages and you end up with part user / part maintainer hybrids that no longer fit the "regular Joe User" model.
Now I avoid anything that needs anything like these tools or anything that has a step before running configure from a source code checkout.
Take a look how sane developers (nginx, cpython, etc) are using it.