Tips for C libraries on GNU/Linux
git.kernel.org
git.kernel.org
- We are all used to autotools, it works, nobody cares. - And really, anything but autotools is realy an option. Just get over it.
Everything else is an experiment, and it will come back
to you sooner or later. Why? think cross compilation, installation/
uninstallation, build root integration, separate object trees,
standard adherence, tarball handling, make distcheck, testing,
portability between distros, ...Among those of us that have attempted or have done that port, most will probably not look particularly favorably on autotools again.
One of the easiest approaches here involves reverse-engineering the autotools product from a test run performed on a GNU/Linux box, and then translating and building a platform-specific script.
From what I can tell of it, autotools makes substantial use of GNU/Linux features. Which is certainly fine for its intended platform and typical usage, but it's an approach that is Not Fun for porting that code.
Even on Windows, it works fairly well with MingW. Most porting pain comes from bad package management.
Many POSIX-compatible libraries using autotools/libtool can be compiled on OS X without changes, and often MingW too. That isn't true for many other systems.
More importantly, when you do need to make a change, it's a simple edit of shell code and some pretty straight-forward makefile variables. It could just be unfamiliarity, but I've tried customizing the output of CMake for some specific situations and the complexity of the its makefiles is pretty prohibitive. Trying to change the CMake source files for someone unfamiliar with its "language" and variable name conventions is annoying and error-prone.
If I had my druthers here, I'd like a build tool that wasn't a layer atop GNU/Linux/Unix — and that's not to imply how autotools works here is at all bad or particularly wrong, it's just an approach that's a bear to port the tools — and have that (portable) build tool then generate the platform-specific build script, build procedure, build-whatever equivalent.
Conceptually: to move the existing builds from a procedural, interpreted approach into a higher-level and object-oriented approach.
I think you mean Unix, not GNU/Linux. Porting from one flavor of Unix to another is the raison d'etre of autoconf. It's really quite valuable when you're the one in charge of babysitting 20-year-old AIX or HPUX boxes.
It's not even meaningful to talk about using the autotools on a non-Unix platform, since autoconf emits shell scripts. It sounds like you've run into problems with automake on a non-Unix platform generating Makefiles with shell code in it?
But now, people write Linux libraries, tested on a Linux box, for people running Linux, and only compiled on Linux. There may be OS X support. That's it. This is not the 1980s, where 10,000 Unixes bloomed. With 4 Makefiles, you could support Linux, OS X, Windows, and FreeBSD, and thus cover 99% of your audience. I'm sick of watching configure scripts take longer to run than the actual compilation.
Seriously, get over it and just use autotools.
Get over it and use punched card decks, timesharing is a fad.
Get over it and use FORTRAN.
Get over it and just use Windows.
So, when you take a broader view than "just support what everyone uses", you're not just helping niche platforms -- you're future-proofing.
And separate object trees is nothing new, all the newer systems don't even mention it as a feature as it's the default (for example, in cmake).
Standard adherence, you could say something for that, though cmake is a standard in a pretty large amount of projects too these days.
Installation/deinstallation. All of the tools support installation and setting install prefix. And nah, half of the time deinstallation doesn't work. I think that's the task of a package manager anyway (or just in case of experimental stuff install into /opt/XXX).
Their arguments are not very convincing IMO. Unlike the rest of the article which is pretty good this is just bikeshedding.
$ tar xf autotools-using-package.tar.bz2
$ cd autotools-using-package
$ ./autogen.sh
Error: possibly undefined macro AC_BLABLABA
$ ./autogen.sh
checking for build system type...
^C
$ make clean
$ make distclean
$ ./configure --prefix=$HOME/my_app ...
$ make -j9 install
...
install: no such file or directory blabla.la
$ make install
Let's submit this fix upstream. No problem, use diff.
$ mkdir temp
$ tar xf autotools-using-package.tar.bz2 -C temp
$ mv temp/autotools-using-package autotools-using-package.orig
$ diff -urN autotools-using-package.orig autotools-using-package
I know you're going to say I should be using the VCS checkout in the first place, which would hopefully be configured to ignore the autogenerated files. But as a user, or distribution maintainer, most of the time the bug you find is with a specific, packaged version of the software, and it may be quite an effort to figure out how to get the exact same version from the VCS server.
I always run "./autogen.sh --help" for exactly that reason; then if I see --help output from configure, I know that autogen.sh "helpfully" ran configure for me.
You can also usually just run "autoreconf -v -f -i" directly, unless the package has done something unusual that it needs to take extra steps in autogen for.
Also, autotools amazing features aren't all that. Builds out of the source dir? Eh.. This is 2012 and it doesn't impress anymore. Both waf and SCons can do it no problem. Autotools-projects, on the other hand, seem to always put the object files and linked library in the source directory. Sometimes the built library is put in a hidden directory for some reason I don't understand. Possibly it has something to do with libtool, another kludge on top of autotools one rather would do without. Since modern build tools does not pollute the source directories you basically get make distcheck for free. Waf has it builtin for example and it can easily be extended to also upload your tarballs to your distribution server, sending annoncement emails and what have you.
For a "real" project, https://github.com/ggreer/the_silver_searcher, I have the same set-up: Makefile.am and configure.ac with no generated code in the repo. It works fine as long as you have pkg-config, and most people do. It builds and runs on Solaris, OS X, FreeBSD, and any Linux distro you like.
Although I use autotools the "right way", I'm not a fan of it at all. There are multiple levels of generated files. Configure is generated from configure.in which is generated from configure.ac. Makefile is generated from configure and Makefile.in. Makefile.in is generated from Makefile.am. There are other files in the mix as well. Config.h, config.h.in, aclocal.m4, and various symlinks to autotools (compile, depcomp, install-sh, and missing) get generated. It's insane.
There are other problems. Minor versions of autotools can behave completely differently. AM_SILENT_MAKE was removed between automake 1.10 and 1.11. Instead of printing out minimal text during a build, scripts using that macro crash. Another example: 1.9 requires AM_PROG_CC_C_O to compile anything useful but 1.10 doesn't. What's crazy is that 1.9 actually spits out an error message telling you to add AM_PROG_CC_C_O to your configure.ac. It makes no sense.
A system this complicated can't be pruned without breaking backwards compatibility. For autotools, that's not feasible. There are too many projects that would need to be fixed. The next-best solution is to use something else for new projects and let autotools fade gently into history.
I know some folks like it in all it's python-y glory, but I found it opaque and horrible. Finding where things are done and how to change its behaviour was surprisingly hard work. Now this may have been at least partly due to the way the project was set up but... well just give me a nice Makefile any day.
But, for the love of all that is holy, do not CMake. It works fantastically... until you have to fix something. I tell you this as a distro packager. I've had to fight with build systems. Patching autoconf files, automake files, ant files, are all fairly comfortable for me. I dread the days when I have to figure out an issue with CMake.
Doing complex dependencies in CMake is a pain where as with Autmake, I can just go back to Make in the same file.
That's the issue, really ... each of these build systems (including the autotools) seems to have a sweet spot where it works pretty well, but can be a misery outside it.
None of them (that I've encountered) are even close to being universally good. Pick your poison wisely...
"The MPL's 'file-level' copyleft is designed to encourage contributors to share modifications they make to your code, while still allowing them to combine your code with code under other licenses (open or proprietary) with minimal restrictions."
With some software (Go code that is always statically linked by the official toolchain comes to mind) you are shooting yourself in the foot with GPL and even LGPL.
This is good advice especially if you want to write bindings for other languages, including callbacks (as I've found) adds a lot of difficulty to that stage.
However, two main projects I work on use callbacks extensively. The reason is that they are essentially event-based, and it would seem much less user-friendly to force the user to implement a huge switch statement, particularly when user-defined events are involved.
How else could callbacks be avoided? In some cases, they just seem like the most user-friendly option.
Although for bindings they are more difficult, using callbacks can be great when binding to higher-level languages, where the user can specify what should happen using a short lambda function (e.g. Python, etc.) A switch statement or whatever is much more obnoxious in those cases, so what other solutions are there?
Unfortunately if you're trying to be portable, you can't do this, and instead have to implement a complete event notification abstraction (unless you know you will only ever be dealing with this one file descriptor). E.g. user of the library needs to implement functions like addWait(fd, io_type, callback), delWait(wait_id), addTimeout(milliseconds, callback), delTimeout(timeout_id).
Most prominently Plan9 and friends are opposed to dynamically linked libraries.
I also agree that there are use cases where static linking solves some problems.
It also doesn't really work out if you're not buying that philosophy as a whole.
One of the authors of the linked piece is Lennart Poettering, creator of PulseAudio
Glass houses, after all. ;)
Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.
I work on Guile because I enjoy messing with compilers. I absolutely agree that very few projects use it (exceptions include Lilypond and GNUCash). However, that might change - Guile 2.0 switched from a simple interpreter to a virtual machine and a compiler to that VM. That means that Guile is competitive in speed with other scripting languages (and actually faster than some of them, I believe). It also means that it now supports multiple high-level languages. There is currently a good Emacs Lisp implementation and about half of an ECMAScript implementation.
I don't know if it'll become more widely used, but on the mailing lists you certainly see people writing libraries and contributing code, so I think there is a real chance of it. My sense is that Guile is now coming out of a period of stagnation. I don't know where it's going.
edit: http://lists.gnu.org/archive/html/guile-user/2012-01/msg0006...
That's exciting, though - it would be really cool if Makefiles could do more computation. Maybe then we'd get an easier-to-use autoconfiguration system!
And an easier-to-use autotools? Bring it on!
It can be argued that this is not the best solution because every place where fork() is called needs to be patched, and this could be in libraries. But the same applies to O_CLOEXEC flags; every place where file descriptors are created needs to be patched. Further, there are probably many more places where fd's are created than where fork() is called.
So if you want to be super careful library, you should do both. Yes, I know the article advises against fork() from libraries. But sometimes you really need it. It's not bad per-se, just bad when done in *nix because of the broken design of OS interfaces.
[1] http://code.google.com/p/badvpn/source/browse/trunk/system/B...
CLOEXEC approaches are the only race free solutions.
Thats a sign something needs to be taken out in the yard and shot if there ever was one.
If you're not going to do that, then use an alternative that provides the same command-line interface as autotools (so that things like "./configure --prefix=FOO && make -j4 && make install DESTDIR=BAR" still work). As far as I know, there's no such thing as of yet.
Can anyone explain this piece of advice?
That is similar how the STL does it, but not like stdio
I'm not sure I fully agree. - For simple libraries it's likely good advice. - But it encourages big locks and poor scaling. It may be right for desktop apps, but not necessarily for server code that needs to scale. For some things that's fine, but you don't want that for the big tree or hash table that your multi threaded server is built around on. - It avoids the problems of locks being non composable, that is the caller may need to know which order the locks need to be called, to avoid deadlock. Actually it doesn't avoid it, just pushes it to someone else. However if you make sure the library is always the leaf and never calls back the library locks will be generally at the bottom of the lock hierarchy.
Let's be honest. Most multithreaded programs evolve from programs that are more-or-less single threaded. Then, threads are added in an attempt to improve performance, and high-contention locks are broken into finer grained locks when profiling shows lock contention in the critical path. I would argue it's better to either design for minimal mutable global state from the start. Failing that, it's often better to re-factor the code when you start scaling up the number of threads, before you start investing a lot of time into locking and breaking down your big locks into finer and finer grained locks.
I'm sure you're not one of those programmers who often leans on mutexes/semaphores/etc. as a crutch to prop up poor design, but there are a lot of programmers who do.