Hacker News Reader

Comment by enriquto | Hacker News Reader

> tests for mandatory and optional libraries

I never understood the concept of "optional" library. Either your program requires a library or it doesn't. If it requires the library, the easiest thing to do is to assume that it is already installed and just link to it. In case it is not installed the compiler will fail with a very clear error message.

Say, your program is an editor of png files and it requires libpng. Thus in your code you will have

    #include <libpng.h>

and in your makefile you will link your executable with -lpng. And that is all! This works in linux, bsd and macOS. What the fucking else do you need?

View on HN

>I never understood the concept of "optional" library.[...] In case it is not installed the compiler will fail with a very clear error message. [...] and in your makefile you will link your executable with -lpng. And that is all! This works in linux, bsd and macOS. What the fucking else do you need?

Interesting condescending tone there. I didn't downvote but if I can try to answer the core part of your misunderstanding: Many do _not_ want the build to fail when library png is missing.

Example projects using this type of build optionality are FFmpeg and Qt.

For example, ffmpeg lets you build with the optional Fraunhofer FDK AAC (libfdk_aac)[1] encoder but it is not GPL license compatible. If everyone followed your "assume libfdk_aac is already installed" advice, many people would be stuck at the "very clear error message" and not be able to have a working GPL ffmpeg.

Same idea with Qt optional libraries.

[1] https://trac.ffmpeg.org/wiki/Encode/AAC

Come on, then just edit a configuration file specifying what libraries you want to use and where they are! /s

Ok, humor me for a second: What's wrong with that?

I'd paste the output of ffmpeg's ./configure --help here to illustrate the point, but it would be a rather hostile HN post due to its sheer size. It's not really practical to expect someone to configure dozens or even hundreds of optional libraries in a configuration file. It would be a major hassle to have to get a config file correct rather than config detecting as much as possible. ffmpeg is an extreme case, but it's proportionally not that uncommon; many of the things people most want to compile by hand are naturally the things with a ton of optional libraries. The easy cases with 0 or a handful of options are handled by your package manager and you don't care, because there aren't any "options" for you to disagree with.

It's not "wrong". It's a choice.

You can provide some code and let the users do whatever is needed to get it to compile on their system.

Or you can try to automate the process with the best of intentions and create a monster.

In any case, the reason why autotools exists in the first place is that people were less than thrilled with the work required to make things work by hand.

Users should ask for something, and the build should build that, or fail if it can't.

Randomly linking against things if they happen to be available is a terrible idea.

Does any build ecosystem except C do this? I'm not sure even CMake does this.

Who is the user? Most "users" never build 99% of what they use from source. Sure a few things that they do build they would like control over. But randomly linking against things is the default because it make the package maintainer's job easier, and it is hard enough as it is to get people to maintain and update packages for distributions.

> Ok, humor me for a second: What's wrong with that?

There's absolutely nothing wrong with that, and in fact it's so simple that afterwards we can even write a script to automate setting those variables.

Who needs convoluted build systems anyway?

My sarcasm detector is on the fritz, what you are talking about is autotools, correct?

Say, your program is an editor of png files

Say you are writing a program to edit images that can potentially read lots of different formats. However most people don't need support for .sgi or .ecw files and won't have the libraries to read those formats installed. Your program is still perfectly usable without those libraries and so you should still be able to compile and run it, just without support for those formats.

Then you end up with a program where the users can never trust it to have any given feature. They use it all the time to edit .sgi files, and then install it on a new computer, and bam, suddenly, no support for .sgi files.

Or even worse, it is used as a dependency in a script, or it is a library linked by another program, but the author can never trust that any given installation has the feature he actually needs.

It is a mess. Just don't do it.

The disadvantage is very mild, and the advantage huge (modularity and ability to easily go embedded). If your dependency manager cannot express "program[+feature]", it is your dependency manager's problem.

I do not know of a single dependency manager out there that can express that.

MacPorts' variants:

* https://guide.macports.org/chunked/reference.variants.html

* https://guide.macports.org/chunked/using.variants.html

The default / most common variants have pre-built binaries that are downloaded via a port install, but if you specify something more 'obscure', things are built from source.

FreeBSD's Ports also allow for options:

* https://docs.freebsd.org/en/books/porters-handbook/#makefile...

Build-time and run-time dependencies are both handled.

> a single dependency manager out there that can express that.

If it existed, it would be a nightmare, because features are recursive, and you would end up with fragile monstrosities like

    foo[+bar[+baz[+qux],+quux]]

of course, each variable having version numbers and all

ebuilds¹ use essentially that syntax, as does setuptools². Others support that too(nix, bitbake, etc), but I have bookmarks on the first two ;)

¹ https://devmanual.gentoo.org/general-concepts/dependencies/i...

² https://setuptools.readthedocs.io/en/latest/userguide/depend...

Nix and guix.

Exactly.

By the way another thing I've come to hate is configure or make scripts which download stuff. This is really annoying if you want to build self-contained packages. Or if the downloaded material ends up being not available, or if you're not always connected to the internet.

> Then you end up with a program where the users can never trust it to have any given feature.

That's the "glass half empty" pov.

The "half glass full" pov is that anyone can still use their favourite piece of software even if an obscure and/or useless library is not available to them.

I like the idea, but the reality is that it fosters extreme complexity on all kinds of axes for marginal benefit.

Sure, for hobbyists it is fine but professional use.. I disagree.

> I never understood the concept of "optional" library.

Consider a scientific software, let's call it x. x can solve many problems and these problems cross-cut a lot of disciplines.

A researcher, user of x has a need for only subset of these features, and configures & compiles x with the features only he/she needs. This has many advantages from simpler setup to shorter build to easier verification.

When you're compiling software on remote, restricted systems; these optional components makes people's life a lot easier. Also, esp. in scientific community, every library needs something lower level so, trying to compile everything in results in exponential dependency hikes. Optional components solve this very neatly.

Another similar use case that is quite common is that your tool can use one of several different calculation kernels. For example there is one option that is fast but closed source, one that is open source but slower and one option that only works with Nvidia GPUs. Most people won't have all three options available, but as long as you have one of them the software works fine.

> I never understood the concept of "optional" library. Either your program requires a library or it doesn't.

The program may have an optional feature that requires a library. It does not make sense to force everyone to install it, if such feature is used by 1 % of users. Example might be mplayer, where there are common decoding libraries used by everyone and also plenty of obsure decoding libraries.

Even better is to use dlopen() to open the optional library in runtime, to fit in distribution package management paradigm, but this requires much more work than simple option to disable dependency at compile time, and even with it it is useful to have ability do disable optional dependency at compile time to avoid requiement for header files.

> Say, your program is an editor of png files and it requires libpng. Thus in your code you will have > > #include <libpng.h>

In addition to what other commenters wrote (failures later during compilation instead of at the start), you also need proper include path in CFLAGS.

For example, gdk-pixbuf is supposed to be included by '#include <gdk-pixbuf/gdk-pixbuf.h>'. That would assume the header file is in /usr/include/gdk-pixbuf/gdk-pixbuf.h . But distribution that i use installs it in /usr/include/gdk-pixbuf-2.0/gdk-pixbuf/gdk-pixbuf.h , likely to be able to have multiple versions of header files installed togethers.

You are supposed to use 'pkg-config' tool in configure script to add proper header file locations in CFLAGS to handle this (in this case -I/usr/include/gdk-pixbuf-2).

> and in your makefile you will link your executable with -lpng.

The library you are usimg may depend itself. on another library, you need to add -l flags for all recursively dependent libraries to linker. As this set of recursively dependent libraries may change in the future, you are supposed to use pkg-config to find proper link flags.

Note that all of this is not really dependent on configure, you can do that in a simple makefile (by just calling pkg-config directly from it). So it is more about using pkg-config instead of hardcoded paths than about using configure scripts.

> proper include path in CFLAGS

This makes no sense. Preprocessor options have no place in CFLAGS, which is for compiler options. I guess you meant CPPFLAGS? Putting a preprocessor option on CFLAGS is always wrong and expected to fail.

> But distribution that i use installs it in

If your distribution installs headers outside of the preprocessor path (or somewhere inside CPATH or C_INCLUDE_PATH), then I'd say that this distribution is broken: it has not installed the library, just copied the files somewhere. The makefile is then, correctly, expected to fail. It's just as if you had unzipped the library on /tmp, of course you would not expect the compiler to find it!

> This makes no sense. Preprocessor options have no place in CFLAGS, which is for compiler options.

Technically true, but pkg-config does not distinguish between CFLAGS and CPPFLAGS.

> If your distribution installs headers outside of the preprocessor path (or somewhere inside CPATH or C_INCLUDE_PATH), then I'd say that this distribution is broken

Both Debian and Red Hat use this path. These libraries are designed to be located with pkg-config, so you can install them anywhere, and everything works as expected.

This kind of opinions is why the dream of Desktop Linux will keep being a dream.

Is it though? How so?

Your parent poster IMO has a point: seems that part of the reason for autotools existence is to also cater to all distributions choosing weird places to put headers and libraries in. Shall we accommodate them forever?

As for Linux desktop, personally I'm skipping it due to the complete lack of security and sandboxing in X11: basically every program can be a keylogger is the typical egregious example. And Wayland is kind of funny; people are still trying hard to make it be 60FPS capable which is absurd in 2021.

Sure, it can be simple.

To give you a less clear example, think about writing a library that sends telemetry to some cloud service. The telemetry could be readings from a sensor attached to a microcontroller with 512K of RAM or it could be readings from a server sitting in a datacenter with 256GB of RAM.

You need some helper library to handle the protocol and there's a really nice full-featured library that comes with a bunch of handy debugging tools but it's too memory-hungry for that tiny microcontroller.

Another option is a minimalist library that uses very little memory but also has less flexibility in say TLS.

If you pick just one, one of your platforms ends up suffering needlessly.

Adding the options allows the user to decide what's best for them.

Sure, you can have different make targets if you really need to. Or Makefile.linux and Makefile.embedded everybody understands what to do in that case.

I simply don't understand the explosion of complexity associated to autotools, cmake, and the like.

Because the cartesian product of options quickly explodes and you're suddenly managing 256 targets or makefiles (though even in Make there's a better way to handle this, though it's made painful by the syntax). It's very common for larger pieces of software to have 30-50 different configuration options for building.

> I simply don't understand the explosion of complexity associated to autotools, cmake, and the like.

If you refer to tools like cmake as "explosion of complexity" then I have to say that you are using them entirely wrong, and you should review your practice thoroughly.

I mean, cmake is a makefile generator. You specify a high-level description of your project, and then you run cmake for it to generate your Makefile that performs the whole build.

With cmake you don't even care which compiler you use. You can simply state you require, say, c++ 14 with constexpr support and you're done for all platforms and all compiler picks.

If you believe cmake is more complex than a makefile then either you only work on single-foot projects or you have been cursed with hideously managed cmake projects.

It's not just about libraries.

Autoconf for example can abstract across different compilers from different vendors, with all their various incompatible command–line arguments. And it does that for a dozen or so different languages, not just C.

It can abstract across different function signatures in the libraries you're using. For example if you're calling a libc function where the arguments have changed, you can detect which version of the function you have and make your code work with either one.

It can do the same for structs and types as well.

It can test for system services, such as X Windows or the ability to run Perl scripts.

It can abstract over the differences between different implementations of common utility commands such as awk, grep, install, mkdir, and so on.

It does a lot of stuff!

> It does a lot of stuff!

Of course it does. My point was that I've never found it did anything that I was interested in.

Then it wasn't written for you. It was written for people who want to distribute software to users who do not all run the same Unix operating system (not to mention the radically incompatible hardware of the day), and whose Unix vendors are all mutually hostile to any form of cooperation with each other. If all you ever install your software on is Ubuntu running in a VM at your cloud provider, then of course this will all be irrelevant to you.

Large projects can take dozens of minutes up to hours to compile. It's extremely frustrating if the build system doesn't tell you in advance if it expects the compile to work. Having an error message once every 20 minutes, adding a package and restarting the build is terrible. (Also, for package systems like DEB or RPM, you can't just continue a compile that half-finished.) Now imagine you're a Linux distro and doing that for thousands of packages...

> It's extremely frustrating if the build system doesn't tell you in advance if it expects the compile to work.

I have only found this behavior in three different projects that used cmake. After several minutes of beautiful compilation lines in green, it turns out that the library (that cmake was supposed to check) was not installed after all. Of course it printed a beautiful error message in red. Beautiful but non-informative, that is.

And what if your program needs a version of libpng above a certain version? And maybe below another one? That can give compile errors that are quite alot harder to grok.

One of the frameworks I use optionally supports a commercial library. If the library is present it enables support, if it isn't it just compiles without it.

This is a nice feature since it avoids a lot of manual configuration. The downside is that support fails silently if the library is missing - since that is the expected case for most users.

Reply on news.ycombinator.com