Hacker News Reader

> Which makes me think... why did I just sit through that long "configure" script run, and why did the developer bother messing with autotools, if this project isn't even going to bother targeting anything but Linux with the GNU toolchain to build it?

Even if target is just Linux with GNU toolchain, there are still plenty of issues that needs to be managed by configure script - tests for mandatory and optional libraries / header files, some functions are in different libraries on different systems (e.g. tinfo vs ncurses, or librt), different versions of kernel headers, different versions of GCC (older ones may not support some options).

> tests for mandatory and optional libraries

I never understood the concept of "optional" library. Either your program requires a library or it doesn't. If it requires the library, the easiest thing to do is to assume that it is already installed and just link to it. In case it is not installed the compiler will fail with a very clear error message.

Say, your program is an editor of png files and it requires libpng. Thus in your code you will have

    #include <libpng.h>

and in your makefile you will link your executable with -lpng. And that is all! This works in linux, bsd and macOS. What the fucking else do you need?

>I never understood the concept of "optional" library.[...] In case it is not installed the compiler will fail with a very clear error message. [...] and in your makefile you will link your executable with -lpng. And that is all! This works in linux, bsd and macOS. What the fucking else do you need?

Interesting condescending tone there. I didn't downvote but if I can try to answer the core part of your misunderstanding: Many do _not_ want the build to fail when library png is missing.

Example projects using this type of build optionality are FFmpeg and Qt.

For example, ffmpeg lets you build with the optional Fraunhofer FDK AAC (libfdk_aac)[1] encoder but it is not GPL license compatible. If everyone followed your "assume libfdk_aac is already installed" advice, many people would be stuck at the "very clear error message" and not be able to have a working GPL ffmpeg.

Same idea with Qt optional libraries.

[1] https://trac.ffmpeg.org/wiki/Encode/AAC

Come on, then just edit a configuration file specifying what libraries you want to use and where they are! /s

Ok, humor me for a second: What's wrong with that?

I'd paste the output of ffmpeg's ./configure --help here to illustrate the point, but it would be a rather hostile HN post due to its sheer size. It's not really practical to expect someone to configure dozens or even hundreds of optional libraries in a configuration file. It would be a major hassle to have to get a config file correct rather than config detecting as much as possible. ffmpeg is an extreme case, but it's proportionally not that uncommon; many of the things people most want to compile by hand are naturally the things with a ton of optional libraries. The easy cases with 0 or a handful of options are handled by your package manager and you don't care, because there aren't any "options" for you to disagree with.

It's not "wrong". It's a choice.

You can provide some code and let the users do whatever is needed to get it to compile on their system.

Or you can try to automate the process with the best of intentions and create a monster.

In any case, the reason why autotools exists in the first place is that people were less than thrilled with the work required to make things work by hand.

Users should ask for something, and the build should build that, or fail if it can't.

Randomly linking against things if they happen to be available is a terrible idea.

Does any build ecosystem except C do this? I'm not sure even CMake does this.

Who is the user? Most "users" never build 99% of what they use from source. Sure a few things that they do build they would like control over. But randomly linking against things is the default because it make the package maintainer's job easier, and it is hard enough as it is to get people to maintain and update packages for distributions.

> Ok, humor me for a second: What's wrong with that?

There's absolutely nothing wrong with that, and in fact it's so simple that afterwards we can even write a script to automate setting those variables.

Who needs convoluted build systems anyway?

My sarcasm detector is on the fritz, what you are talking about is autotools, correct?

Say, your program is an editor of png files

Say you are writing a program to edit images that can potentially read lots of different formats. However most people don't need support for .sgi or .ecw files and won't have the libraries to read those formats installed. Your program is still perfectly usable without those libraries and so you should still be able to compile and run it, just without support for those formats.

Then you end up with a program where the users can never trust it to have any given feature. They use it all the time to edit .sgi files, and then install it on a new computer, and bam, suddenly, no support for .sgi files.

Or even worse, it is used as a dependency in a script, or it is a library linked by another program, but the author can never trust that any given installation has the feature he actually needs.

It is a mess. Just don't do it.

The disadvantage is very mild, and the advantage huge (modularity and ability to easily go embedded). If your dependency manager cannot express "program[+feature]", it is your dependency manager's problem.

I do not know of a single dependency manager out there that can express that.

MacPorts' variants:

* https://guide.macports.org/chunked/reference.variants.html

* https://guide.macports.org/chunked/using.variants.html

The default / most common variants have pre-built binaries that are downloaded via a port install, but if you specify something more 'obscure', things are built from source.

FreeBSD's Ports also allow for options:

* https://docs.freebsd.org/en/books/porters-handbook/#makefile...

Build-time and run-time dependencies are both handled.

> a single dependency manager out there that can express that.

If it existed, it would be a nightmare, because features are recursive, and you would end up with fragile monstrosities like

    foo[+bar[+baz[+qux],+quux]]

of course, each variable having version numbers and all

ebuilds¹ use essentially that syntax, as does setuptools². Others support that too(nix, bitbake, etc), but I have bookmarks on the first two ;)

¹ https://devmanual.gentoo.org/general-concepts/dependencies/i...

² https://setuptools.readthedocs.io/en/latest/userguide/depend...

Nix and guix.

Exactly.

By the way another thing I've come to hate is configure or make scripts which download stuff. This is really annoying if you want to build self-contained packages. Or if the downloaded material ends up being not available, or if you're not always connected to the internet.

> Then you end up with a program where the users can never trust it to have any given feature.

That's the "glass half empty" pov.

The "half glass full" pov is that anyone can still use their favourite piece of software even if an obscure and/or useless library is not available to them.

I like the idea, but the reality is that it fosters extreme complexity on all kinds of axes for marginal benefit.

Sure, for hobbyists it is fine but professional use.. I disagree.

> I never understood the concept of "optional" library.

Consider a scientific software, let's call it x. x can solve many problems and these problems cross-cut a lot of disciplines.

A researcher, user of x has a need for only subset of these features, and configures & compiles x with the features only he/she needs. This has many advantages from simpler setup to shorter build to easier verification.

When you're compiling software on remote, restricted systems; these optional components makes people's life a lot easier. Also, esp. in scientific community, every library needs something lower level so, trying to compile everything in results in exponential dependency hikes. Optional components solve this very neatly.

Another similar use case that is quite common is that your tool can use one of several different calculation kernels. For example there is one option that is fast but closed source, one that is open source but slower and one option that only works with Nvidia GPUs. Most people won't have all three options available, but as long as you have one of them the software works fine.

> I never understood the concept of "optional" library. Either your program requires a library or it doesn't.

The program may have an optional feature that requires a library. It does not make sense to force everyone to install it, if such feature is used by 1 % of users. Example might be mplayer, where there are common decoding libraries used by everyone and also plenty of obsure decoding libraries.

Even better is to use dlopen() to open the optional library in runtime, to fit in distribution package management paradigm, but this requires much more work than simple option to disable dependency at compile time, and even with it it is useful to have ability do disable optional dependency at compile time to avoid requiement for header files.

> Say, your program is an editor of png files and it requires libpng. Thus in your code you will have > > #include <libpng.h>

In addition to what other commenters wrote (failures later during compilation instead of at the start), you also need proper include path in CFLAGS.

For example, gdk-pixbuf is supposed to be included by '#include <gdk-pixbuf/gdk-pixbuf.h>'. That would assume the header file is in /usr/include/gdk-pixbuf/gdk-pixbuf.h . But distribution that i use installs it in /usr/include/gdk-pixbuf-2.0/gdk-pixbuf/gdk-pixbuf.h , likely to be able to have multiple versions of header files installed togethers.

You are supposed to use 'pkg-config' tool in configure script to add proper header file locations in CFLAGS to handle this (in this case -I/usr/include/gdk-pixbuf-2).

> and in your makefile you will link your executable with -lpng.

The library you are usimg may depend itself. on another library, you need to add -l flags for all recursively dependent libraries to linker. As this set of recursively dependent libraries may change in the future, you are supposed to use pkg-config to find proper link flags.

Note that all of this is not really dependent on configure, you can do that in a simple makefile (by just calling pkg-config directly from it). So it is more about using pkg-config instead of hardcoded paths than about using configure scripts.

> proper include path in CFLAGS

This makes no sense. Preprocessor options have no place in CFLAGS, which is for compiler options. I guess you meant CPPFLAGS? Putting a preprocessor option on CFLAGS is always wrong and expected to fail.

> But distribution that i use installs it in

If your distribution installs headers outside of the preprocessor path (or somewhere inside CPATH or C_INCLUDE_PATH), then I'd say that this distribution is broken: it has not installed the library, just copied the files somewhere. The makefile is then, correctly, expected to fail. It's just as if you had unzipped the library on /tmp, of course you would not expect the compiler to find it!

> This makes no sense. Preprocessor options have no place in CFLAGS, which is for compiler options.

Technically true, but pkg-config does not distinguish between CFLAGS and CPPFLAGS.

> If your distribution installs headers outside of the preprocessor path (or somewhere inside CPATH or C_INCLUDE_PATH), then I'd say that this distribution is broken

Both Debian and Red Hat use this path. These libraries are designed to be located with pkg-config, so you can install them anywhere, and everything works as expected.

This kind of opinions is why the dream of Desktop Linux will keep being a dream.

Is it though? How so?

Your parent poster IMO has a point: seems that part of the reason for autotools existence is to also cater to all distributions choosing weird places to put headers and libraries in. Shall we accommodate them forever?

As for Linux desktop, personally I'm skipping it due to the complete lack of security and sandboxing in X11: basically every program can be a keylogger is the typical egregious example. And Wayland is kind of funny; people are still trying hard to make it be 60FPS capable which is absurd in 2021.

Sure, it can be simple.

To give you a less clear example, think about writing a library that sends telemetry to some cloud service. The telemetry could be readings from a sensor attached to a microcontroller with 512K of RAM or it could be readings from a server sitting in a datacenter with 256GB of RAM.

You need some helper library to handle the protocol and there's a really nice full-featured library that comes with a bunch of handy debugging tools but it's too memory-hungry for that tiny microcontroller.

Another option is a minimalist library that uses very little memory but also has less flexibility in say TLS.

If you pick just one, one of your platforms ends up suffering needlessly.

Adding the options allows the user to decide what's best for them.

Sure, you can have different make targets if you really need to. Or Makefile.linux and Makefile.embedded everybody understands what to do in that case.

I simply don't understand the explosion of complexity associated to autotools, cmake, and the like.

Because the cartesian product of options quickly explodes and you're suddenly managing 256 targets or makefiles (though even in Make there's a better way to handle this, though it's made painful by the syntax). It's very common for larger pieces of software to have 30-50 different configuration options for building.

> I simply don't understand the explosion of complexity associated to autotools, cmake, and the like.

If you refer to tools like cmake as "explosion of complexity" then I have to say that you are using them entirely wrong, and you should review your practice thoroughly.

I mean, cmake is a makefile generator. You specify a high-level description of your project, and then you run cmake for it to generate your Makefile that performs the whole build.

With cmake you don't even care which compiler you use. You can simply state you require, say, c++ 14 with constexpr support and you're done for all platforms and all compiler picks.

If you believe cmake is more complex than a makefile then either you only work on single-foot projects or you have been cursed with hideously managed cmake projects.

It's not just about libraries.

Autoconf for example can abstract across different compilers from different vendors, with all their various incompatible command–line arguments. And it does that for a dozen or so different languages, not just C.

It can abstract across different function signatures in the libraries you're using. For example if you're calling a libc function where the arguments have changed, you can detect which version of the function you have and make your code work with either one.

It can do the same for structs and types as well.

It can test for system services, such as X Windows or the ability to run Perl scripts.

It can abstract over the differences between different implementations of common utility commands such as awk, grep, install, mkdir, and so on.

It does a lot of stuff!

> It does a lot of stuff!

Of course it does. My point was that I've never found it did anything that I was interested in.

Then it wasn't written for you. It was written for people who want to distribute software to users who do not all run the same Unix operating system (not to mention the radically incompatible hardware of the day), and whose Unix vendors are all mutually hostile to any form of cooperation with each other. If all you ever install your software on is Ubuntu running in a VM at your cloud provider, then of course this will all be irrelevant to you.

Large projects can take dozens of minutes up to hours to compile. It's extremely frustrating if the build system doesn't tell you in advance if it expects the compile to work. Having an error message once every 20 minutes, adding a package and restarting the build is terrible. (Also, for package systems like DEB or RPM, you can't just continue a compile that half-finished.) Now imagine you're a Linux distro and doing that for thousands of packages...

> It's extremely frustrating if the build system doesn't tell you in advance if it expects the compile to work.

I have only found this behavior in three different projects that used cmake. After several minutes of beautiful compilation lines in green, it turns out that the library (that cmake was supposed to check) was not installed after all. Of course it printed a beautiful error message in red. Beautiful but non-informative, that is.

And what if your program needs a version of libpng above a certain version? And maybe below another one? That can give compile errors that are quite alot harder to grok.

One of the frameworks I use optionally supports a commercial library. If the library is present it enables support, if it isn't it just compiles without it.

This is a nice feature since it avoids a lot of manual configuration. The downside is that support fails silently if the library is missing - since that is the expected case for most users.

the `./configure` often tests edge-cases from the 90s and early 2000s, that are not relevant anymore, but everyone is afraid to touch any of that.

...because almost nobody can really write any code in M4sh (M4 with bash spliced together) and other odd languages.

Really, the e-mail from the actual link is surprisingly sober and calls out the weaknesses of autotools well.

> ... but I'm not sure how much I should be blaming autotools versus the projects that use autotools.

Definitely the latter. Few projects treat `configure.ac` and its macros as serious as programming should be. They produce spaghetti code, duplicate code, patched code, slow code, unmaintainable code and they leave it there for decades only add code to make it worse, without ever feel the embarrassment to fix it. Of course blame the coder before blame the language.

I dunno, if a majority of projects using X come out bad, eventually it is a fault in X, not just bad users.

Not really. As an example, the configure script will often output a "config.h" header that will have a bunch of preprocessor macros that the developer can use to make decisions about what platform features are supported.

So you might have HAVE_FOO and HAVE_BAR in config.h. But if the developer doesn't do something like:

    #ifdef HAVE_FOO
    int ret = foo();
    #elif HAVE_BAR
    int ret = bar();
    #else
    int ret = do_some_fallback();
    #endif

... then that's on the developer for just working on a system that has foo() and doesn't consider other systems that only have bar(), or have neither.

Of course, that assumes they even put the tests for foo() and bar() in configure.ac in the first place!

I would still use autotools for a C project even if I was only targeting linux/x86_64, just so I could write a simple declarative Makefile.am file that lists sources and defines target outputs, and have autoconf/automake figure out everything else, and put it behind a standard interface (that is, one that respects prefix, destdir, etc.). Also being able to easily write tests for dependencies and get the appropriate cflags and ldflags makes life easier.

Yes really, if everyone does it wrong, it means something is lacking to put those peoples in the right track from the start.

Can you supply said statistic?

Are there any good examples of configure.ac macros that would be useful to study?

So you're saying it's the PHP of build tools?

...that kinda works, actually.

Yeah, the autotools model of portability didn't stand the test of time, but there's no shame in that. It's in good company. Isn't CMake the usual tool these days?

Autotools, during its lifetime, has provide tremendous value across innumerable scenarios and systems. What does it matter if it didn't stand the "test of time" — especially if it shone a light for its successors to follow?

The Autotools model of probing for system capabilities at build time isn't dead; it will never die, it will only be used in greater and lesser amounts and to complement other means of discovering system capabilities.

CMake seems to me like a reimplementation of the Autotoools model of build-time probing. The true competitors are binary distribution after building on precursor systems with full specification and no probing, and cross-compilation with full specification and no probing.

Saying that something is obsolete isn't a value judgement -- it's an observation of the differences between the state of the world when it was necessary and now. This describes autotools to a 't'. It's not that there are that many fewer build targets these days, but rather that there are many many many fewer installations of most of those build targets these days.

Cmake is only useful in very limited scenarios. I do maintain lots of projects with both, and cmake is always the second citizen, for msvc users only.

Serious users use autotools (with mingw), but the occasional windows noob cannot be helped. So he gets his limited cmake experience. Cmake usually only does a tenth of the autotools probes, edge cases will never be detected and worked around. Such as broken compiler versions, as with gcc or icc frequently. Cmake is a visual basic like hack, but Visual Basic at least had a proper design. Just lacked the library infrastructure.

Everybody uses autotools, show me a cmake-based distro.

All this autotools bashing is hilarious. Probing is costly yes, but you only do it once, and you rarely do it by yourself. Distro packagers do that for you.

This is a complete mischaracterisation of CMake usage.

Today, it's better at portability than most Autotools builds I've used. In projects which maintain both systems side-by-side, I've had Autotools fail on Solaris, MinGW and Cygwin while with CMake it worked without trouble. And that's for projects which hadn't specifically tested and fixed defects for portability issues on these platforms. Empirically, CMake had better platform coverage.

As for the probing of features, take an actual look at what CMake provides. Over an order of magnitude more feature tests. The problem with the Autoconf feature testing is that the vast majority of the tests are 15 years obsolete at this point. The CMake tests are mostly for contemporary platforms. That's where the value lies.

> Everybody uses autotools, show me a cmake-based distro.

The LLVM project exclusively uses cmake.

> Yeah, the autotools model of portability didn't stand the test of time, but there's no shame in that.

xmkmf joined the chatroom

> Which makes me think... why did I just sit through that long "configure" script run, and why did the developer bother messing with autotools, if this project isn't even going to bother targeting anything but Linux with the GNU toolchain to build it?

There's more than just compiling, it adds a bunch of common stuff "out of the box" (with a line or two of configuration). Test runners, pkg-config, installation, cleaning, out of tree builds, etc. Nothing magic that you couldn't do yourself, but IME by the time you do it your makefile is much less simple, it's taken a non-trivial amount of time and possibly has a bunch of subtle errors. Even on my own little projects that will never run on another computer let alone another OS I find autotools worthwhile.

If I had one major complaint it's that it still puts build artifacts in the root directory.

> One annoyance is that it often feels like "./configure ..." takes longer to run than the actual "make".

Agreed. The configure script doesn’t take advantage of concurrency, but make can.

But the number one cause of slow configure is that people forget to run configure -C. The -C option makes it cache the outcome of all of the tests, so that it doesn’t have to do them all again. This is a huge time–saver!

Since ./configure just probes the compiler it could be paralellized. Make has concurrency (-j4) why doesn't ./configure have the same feature

It’s also a shell script. It’s a shell script written purely in POSIX–compatible sh, that is generated by the worst M4 script ever written. The information which will be printed to stdout, to the log files, to the cache files, to the status file, and so on all needs to be buffered until the end of any asynchronous task, then written atomically so that it can be understood and will work consistently. That’s synchronizing writes to six or seven different file descriptors, just so that everything comes out in a comprehensible order. I bet you're already thinking about how to implement it though.

And it does quite a lot more than just probe the compiler.

And finally, it doesn’t support concurrent execution because nobody who was tried to write it has returned. Usually it starts with confident estimates and preparations, followed by departure and sporadic reports. Gradually the reports dwindle in frequency and comprehensibility, until eventually nothing more is heard.

So it's impossible?

Because typically make produces a number of independent targets (for example, object files of different source files) while configure only produces a single output file containing all variable assignments.

Which linux x86/amd64?

Do you mean the linux that puts libraries in

  /usr/lib/x86_64-linux-gnu

or the one that puts them in

   /usr/lib64

?

who cares? wherever they are, the linker finds them when it sees the option -lpng

I would assume autoconf is just checking for existence of packages, but yes I know other tools like cmake will actually try compiling/linking something to see if it's available.

As someone who ran a fair bit of FreeBSD and Interix/SFU/SUA, autotools was much more reliable than any other build system (though CMake came close). 99% of autotools projects would build fine. Approximately 0% of "simple Makefile"s would build at all.

I really like CMake... except at some point you realize you changed one set of problems for another.

the "modern cmake" movement is a step in the right direction

It is nice that windows and linux folks can both work on a project (which I see as windows developers being able to contribute to linux software)

The reality is that it's kind of messy when you actually try to develop something with it. (but not try-to-figure-out-m4 messy)

Likewise, except at some point I worked on cross compiling and embedded systems. When I had to build something using CMake, I would tremble. Autotools may be pure hell for the author side, but it is quite useful in the user side.

Lucky you. I rarely get autotools projects to compile cleanly even under idyllic circumstances.

> [...] I'm not sure how much I should be blaming autotools versus the projects that use autotools.

I'd say that a large portion of the bad name that autotools currently have is due to their users. On one hand developers, who as you say do not actually use all the deductions of configure scripts,. But also users (the ones who blindly type ./configure) are to blame a little. If you complain that you wait while configure checks to find whether your machine is 32 or 64 bits, etc., please read on how to specify "site defaults" once and then all your configure runs take a fraction of the time.

Same for me but there's no better alternative to autotools that I can rely upon