How to Write Portable C Without Complicating Your Build
nullprogram.com
nullprogram.com
I've been working on and off with Make, C, autoconf, etc for ten years. Maybe if I were a full-time C or C++ person I'd have less trouble with this, but it seems rather excessive. The thing that most discourages me from contributing to certain open source projects is when I clone the repository and the build fails and I can't get the project to build in an hour or less of build-debugging time (and I'm not on a weird setup, I'm on a modern Lenovo with Debian).
Worst case scenario, the SlackBuild I'm writing has to include some call to patch or sed in order to tweak the installation path, or I just have to cp the build output into the package's root directory (something which I'll likely have to do when I get around to updating the CouchDB SlackBuild, but have been putting off since 1.6.x works well enough on Slackware).
Though I don't see the point in reinventing your own build system, exactly. Some people I trust seem to be getting into Meson these days but I havent tried it.
On the other hand, using the autotools in a modern way is dead easy. You don't need to add many many tests if you don't intend to support old stuff. You get access to automake which is a fantastic tool on its own.
Don't read old tutorials, don't look at how big established projects are doing, look at the Autotools Mythbuster instead (autotools.io) and start with a minimal configure.ac.
On the other hand, when something goes wrong with autotools, the first step is to pour yourself a drink.
I really feel a need for something like AC_TRUST_HOST or AC_TRUST_DISTRO or something like that would not check for each header or each function in some library. Instead it will check the version of GCC, libc, and compile some (hopefully just one) program and test.
Of course the common tests shall be skipped only in popular distributions. Say, Debian, Arch, Fedora (and their derivatives), to name a few.
The author's point is to do <em>without</em> a configure step entirely, not write your own.
A standard Makefile is perfectly capable of implementing these things in a manner which is clean, reasonably portable and without the level of indirection that makes things hard to debug.
My own experience is that autotools has evolved to feel less standard than the modern platforms it purports to smooth over the differences in; I seem to frequently find that I can't generate a 'configure' for a 3rd party software or libraries from Git repositories because I of the wrong autotools versions. Efforts to investigate this by unpicking the various macros etc. provided in the build have almost always been unsuccessful, leaving me building from distributed .tar.gz files. With something that feels like such a moving target after 30 years, I'm glad to see people realising the benefit of a simple Makefile that's easy to customise for the edge cases.
I don't object that many projects have convulated and dated autotools scripts notably because they don't age well. It's easy to accumulate a lot of crap.
Most projects don't get autotools because they don't know how to use plain Makefile. They do because autotools bring much more than that.
OTOH, while I'm not a big fan of autotools internals (especially libtool), being able to run "./configure && make install" on tens of thousands of F/OSS packages is something I'd hate to loose. In fact, the extreme consistency in installation procedures (including use of pkg_conf etc.) and discipline in directory layout accross so many F/OSS packages is something I very much admire, given it's an unexpected outcome of a "Bazaar" style development model.
You forgot 'make'!
I'm tempted to turn this into a cheap shot about how the consistency obviously isn't worth much, but I won't.
./configure && make && sudo make install
./configure && sudo make install
(*)(void*, const void*, const void*)
(*)(const void*, const void*, void*)
That's part of the argument: with standards compliant code, you don't need to do "what autotools does".
Well, you're responding to something that wasn't actually said; "these things" quotes specifically the list of requirements given by the previous poster -- CFLAGS etc.
Your example is indeed valid, it's an ugly case and I agree that testing for the feature specifically (rather than arch, compiler) is always preferred. But let's look at the practicalities -- how many of these cases do I actually need in a codebase to tip the balance that would justify use of autoconf? In your example, a simple #ifdef against the platform (Windows/BSD/Linux) and it's gone. qsort_r is off limits for massively portable program anyway, so autoconf's ability to help is limited anyway.
That's equivalent to the uname trick the GP was complaining about. That means your software won't compile in a lot of platforms.
On a small scale, I don't think that's a big problem. Somebody on those platforms only has to add the correct conditions to your #ifdef forest and if he sends the update back, other people on the same platform won't even have the same problem. It's not much different from software not working on untested platforms.
It starts becoming a problem when done often, or on high level (at the calling stack) code.
Is won't anyway; qsort_r has already limited me to a tiny number of platforms. If I care about further portability, autotools can't do anything to help me; my next step is not autotools, it's "don't use qsort_r"
> (*)(void*, const void*, const void*)
> (*)(const void*, const void*, void*)
Also, isn't the first supposed to be:
void*, size_t, size_t, int (*cmp)(const void*, const void*)
My SFINAE code was to handle a similar problem picking between a GNU-specific strerror_r and a POSIX strerror_r that have different signatures (both are available on Linux, determined by whether certain macros are set: https://linux.die.net/man/3/strerror_r , I couldn't just rely on those macros because I wanted my code to compile on any POSIX platform).
Both Linux and BSD chose to add a non-standard qsort_r(), each choosing different ways to do it.
So why not get a tool with 1/10 the complexity and 1/100 the legacy stuff of Autotool, but the same interface otherwise?
People who need the old checks 100% could continue to use Autotool-old, people who don't care for such BS could use the new.
Autotools is a huge pain in the ass for the dev, unfortunately I haven't found any alternative that didn't end up being an even bigger annoyance.
A decent, simple, easily customizable and portable C/C++ build system is still very much a unsolved problem as far as I'm concerned (and I've tried quite a few of them). At least autotools are supported basically everywhere.
CMake is faster than autotools and works fine with all the compilers talked about so far and all the windows compilers I know of.
Creating a CMakeLists.txt covering moderately complex build that links against a few libraries (but doesn't need and custom logic for moving files or other uncommon stuff) is normally just a few lines of code. Usually just one line of code per source file and per library (depending on how you feel about automatically including files this can be further shortened), then a little bit declaring the language and other settings. There are plenty of 10 line CMakeLists that can build large and seemingly complex projects.
By having autotools for the projects that need "everything" (legacy BS checks) and this leaner version for the projects that don't need them.
The same way there's vim and neovim without the legacy stuff.
As a rewrite using the same user interface but a completely different developer interface, there is mklove (https://github.com/edenhill/mklove). However, this just covers autoconf: you don't get the flexibility of automake. Also, I don't know how complete it is.
A rewrite just to speed up things a little is a bit useless. Autotools are not that slow. And if a rewrite was done, it would be a shame to keep the horrible syntax. What needs to be kept is
From the developer's perspective, it is an absolute nightmare – and I consider myself a seasoned professional with the various unix-like systems and their shells. Even with this deliberate care, dedication, and time spent preparing the scripts beforehand, the build usually doesn't work properly out of the box on a new platform.
My mistake was in thinking that the value in autotools system is that you can build a system that will compile for anything. I imagined all you need to do is use (or create) feature tests to sniff out any differences between platforms, and your code will build and work anywhere.
The real value of autotools is for the user. The user can run "./configure && make && make check && make install".
Don't make the mistake I did, thinking that autotools will save you time. It absolutely will not.
https://github.com/SirCmpwn/scmake
It's definitely not something you should use, but I hope new things take a similar approach. It's small, give it a read. Here's a somewhat more complicated project that uses it:
https://github.com/SirCmpwn/libccmd
I think from a usability standpoint (both for devs and users) it's really great but the internals could use some work.
http://stackoverflow.com/questions/600274/alternatives-to-au...
If you’re coding to POSIX, you must define the
_POSIX_C_SOURCE feature test macro to the standard you
intend to use prior to any system header includes:
Take it from someone who religiously keeps many of his very complex libraries portable across many systems (AIX, FreeBSD, Linux/glibc, Linux/musl, NetBSD, OpenBSD, Solaris, and others), the _last_ thing you want to do is define the POSIX portability macros.
Once you do that, you're in for a _world_ of hurt trying to use any extension, including routines from newer POSIX standards that are almost universally supported, but because some systems don't claim 100% compliance to the latest POSIX release, become hidden once you start using the user-definable feature macros.
Most systems (particularly the BSDs) make _everything_ available by default. On Linux, however, you should define _GNU_SOURCE; on Solaris, define __EXTENSIONS__ and _POSIX_PTHREAD_SEMANTICS; on AIX define _ALL_SOURCE; on Minix define _MINIX. This way, anything you might possibly want to use is available.
There are a few cases where a native extension conflicts with a POSIX routine. The classic case is strerror_r on glibc. Dealing with these cases is easier to deal with than fumbling with feature macros.
Remember, you'll _always_ want to use extensions unless you're writing pure ANSI C (C90) code. Portable is not the same thing as POSIX compliant. And many POSIX routines are effectively extensions on systems not yet certified for the latest POSIX standard.
All the books on learning languages I've read just skip it entirely, or provide a default setup with no explanation. Which is somewhat understandable. But then the documentation for various projects also do the same thing. I'm left reading the man pages for various tools that some project says I need and going "Why do I need this?" on a higher level than the man pages.
And in my professional life it's much the same. On one project we have one or two people who understand how the system is setup, reject any kind of change or improvement, and just keep throwing bailing wire at it to keep the process going.
On my own, partly as a result to institutional intertia, we have an ancient TFS build controller running a powershell script running gulp running webpack running babel, and then deploy it using robocopy to network share in IIS (using iisnode). It's amazing it all works. It often breaks, rarely with the same error. It also takes 10 times as long as building on my local machine.
Speaking of someone who occasionally ports software to less common environments I have a few things to say that I was hoping to find in this article.
First the case for and against autotools:
autotools are well supported by every packaging system out there (freebsd ports, pkgsrc, apt, rpm, etc) where it's really just a line or three to do the build in the package definition.
The main things that are not usually thought about by developers but are loved by porters and packagers (and come out of the box with autotools) are: DESTDIR support (installing into an intermediate path, so DESTDIR/PREFIX/usr/bin/myapp) and cross-compilation, which is one of the biggest strengths of using C in the first place!
The biggest argument against autotools is that feature discovery at build time is complete bullshit. I hate it. It reduces determinism and ties execution environment to the build environment.
So if you can support DESTDIR and cross compilation without autotools, go for it! It's not that much extra work!
---
The second point I was hoping this article would address is project layout. In the last few years a nice, standard-ish layout for portable projects has emerged but package maintainers end up having to teach it to every new (big) project.
Here are examples where it has gone okay: https://github.com/nodejs/node/tree/63243bcb330408d511b3945c...
https://github.com/golang/go/tree/964639cc338db650ccadeafb74...
Where you are writing to non-portable parts of unix you can build non-portable stuff into their own files named simply after the platforms and include the correct stuff higher up in a .h
and an exmaple of it not going so well: https://github.com/dart-lang/sdk/issues/10260
(fwiw you can find https://github.com/mulander all over this "teaching" effort, so points to him)
Sometimes I need to build C code for iOS and Android. Libraries with a nice clean setup like the author of this article describes are great. Libraries that use autotools are more trouble than they're worth.
EDIT - I should clarify. I google UWP and it could be Universal Windows Platform. If so, how is that different than the windows builds that Rust does support?
It is the same programming model as .NET, but built on top of COM and just the minimal set of Win32 APIs to support COM based APIs.
It is also sandboxed, just like on OS X.
For Rust to support UWP, it needs to be able to consume COM with the new UWP semantics and at the same time expose traits as COM interfaces accessible to any programming language able to consume UWP libraries.
My use case here is reading Vyatta config files. So lots of files that are generally (but not always) short. Mostly it's a fun teach myself Rust project, and it's hit lots of interesting edge cases in Rust, but otherwise I'd be all over just writing it in Python or Ruby.
You can use MingW's make but compile with cl instead of gcc and that problem is not present. You will have to check if all compiler flags are supported though. Or use a wrapper around cl which translates compiler flags, pretty sure that exists already.
My preferred approach lately is an amalgamation build
That won't scale well though. For large projects you'll really want to use something as mentioned above, or have seperate Makefiles and VS projects, or use a build file generator like CMake.
edit another thing I wanted to add: instead of using
#if defined(_WIN32)
void foo() {
//win implementation
}
#else
void foo() {
//unix implementation
}
#endif
I use CMake with Ninja, which is faster and nicer than plain old Make.
CMake completely falls apart if you're trying to do anything else than normal user space applications that link with commonly used libraries. Some years ago I tried building a bare metal project with CMake and the job got done but the experience was atrocious.
But CMake really shines in building "normal" applications and in particular cross compiling them.
I would recommend against that as well.
I have a directory structure where there's a module/hello.c (stub or generic, platform independent code) and then there's module/platform/hello.c which my Makefile selects over the generic one based on the platform. Primary issue is that there's some code duplication, and it's hard to keep .h files general across platforms all the time (not possible all the time), but in general it does make life a lot easier - especially if platform number grows over time or certain are removed.
Related: http://doc.cat-v.org/henry_spencer/ifdef_considered_harmful
I always thought that was funny. You are using the most powerful (of the widely used) language available for transforming text, but must fall back to the C preprocessor.
If I would to do it again (I'm not active developer anymore though - it's more for personal use now) I would do it again with make. There's certain straightforwardness to it when writing it, and from my experience there's zero to almost none maintenance, but ymmv of course. I would probably look into CMake if I were using vastly different compiler tools (configuration-wise), but I'm not.
Every time I see software without a make uninstall, I have to manually create a textfile with the stuff the application installed so I can do a clean upgrade... it sucks.
Hell, most Windows software ships (mostly crappy) uninstallers, and most OS X software can be uninstalled by dragging the app folder into the Trash, but there are loads of .nix programs without uninstall support? What the f..k?
Oh, and .nix software using standard build systems has another advantage: it's usually trivial to package them in .deb/.rpm if you want to distribute e.g. custom ffmpeg/vlc builds on a fleet of servers.
GCC itself is portable, so?
secure_getenv() first appeared in glibc 2.17.
Though I've recently warmed to autoconf, I still agree it's overkill for most cases. For many years I have, like many others, maintained an ad hoc library of purely preprocessor-based feature checks that I would copy from project to project. When recently writing a comprehensive Unix system API module for Lua I basically ended up with a ridiculous number of feature checks. I broke those out into a separate project I called autoguess.
https://github.com/wahern/autoguess
autoguess doesn't provide any compatibility routines--just the detection. (For compatibility routines see my Lua module, lunix. Compatibility interfaces can be tricky, and context matters, so I've learned to stay away from trying to comprehensively "solve" that problem. One needn't look further than Gnulib to understand the pitfalls of trying to maintain such a beast.)
Most of the code in the autoguess repository is a framework for running autoconf checks and comparing feature detection results with the autoguess header. The library is just the single file "config.h.guess".
Autoguess uses the same naming conventions as autoconf, though in the autoconf universe consistency can be poor and HAVE_FOO names can differ from project-to-project. However, autoguess recommends to use "#if HAVE_FOO" arithmetic conditionals, not "#ifdef HAVE_FOO". That makes it possible to override feature detection macros from CPPFLAGS. Autoconf's rule about using #ifdef is an archaic solution to a mostly non-existent problem these days--broken C preprocessors not evaluating an undefined macro as 0 in arithmetic expressions. (There's a 6 line preamble at the top of configure.ac in my repository that will fix how autoconf generates config.h files.)
Of course, as new operating system releases are made, the autoguess checks can become outdated. Though many autoguess checks aren't directly reliant on version numbers, autoconf feature checking is, I agree, a more robust method. But autoconf checks aren't immune to regressions, either. There's no substitute for regularly building your software on various platforms.
Autoguess does much more than detect system and compiler versions, but I've benefitted greatly over the years from this project:
https://sourceforge.net/p/predef/wiki/Home/It's just that C has been around for 40 years, it'll stay around for at least another 40 years. You can still compile source code from 20-30 years ago with little or no modifications.
C gets the job done fast and efficient.
That said, I'm waiting for a good excuse to learn Rust. Prior to that there have been very few alternatives to C.
It does not get the job done fast and efficient, if you consider the develpment costs. The fast code the compiler generates could be genrated from other sources just as well, it's not especially the merit of that language.
It's written in C because the tooling to analyze and certify it for security in embedded/automotive/aerospace is targetting C (or C++). For some industries, Ada might be an alternative.
It may be paradoxical, but you're not going to be able to write safety critical software in Rust because the tooling to certify it for security doesn't exist and it'll still take years to get there.
Do I like this situation? No, I do not. Do I think it's a big problem? No, not big enough that it can't be solved by pouring money and engineering resources on it.
These certifications unfortunately often are on the level of the golden shields CAs provide to customers to put on their sites as sign of trustworthiness.
So yes, blessed is a nice word for it.
I see these "certified" stamps less as "secure" and more of a huge slowing down of progress and change - which strangely can translate to more secure since you know what you are dealing with.
If I can dream, I'd take a "certified" GNU Ada, or Rust, or something over C. But that is bound to take many years to happen if ever. I think we sooner will see Rust compiled to C which is fed to something like the abominable Greenhill so security consultants can pour over the intermediate C output and put it through the abominable Greenhill.
For example, generally, people use a lot of static arrays in C. Arrays are efficient. The way you program with fix-sized structures like arrays tends to be different to how you program with dynamically-sized structures and that style of programming tends to be more efficient.
In the vast majority of cases, a C or FORTRAN program will be more efficient than other languages not because they are C or FORTRAN, but because they force you to use an efficient paradigm.
Even other compiled languages like C++ or D are, more often than not, slower than the C counterpart UNLESS they are programmed in a "C style".
e: To be clear, the fact that they are compiled low-level languages obviously does help, but I'm trying to make the point that it's not the only reason.
There were, but the market chose otherwise.
At Xerox PARC they moved from BCPL into Mesa, used to write Xerox Star and Pilot OSes. Also one of the first IDEs, also known as Xerox Development Environment (XDE). The year was 1976.
Mesa eventually got automatic memory management support (RC with a local tracing GC for collecting cycles) and became known as Mesa/Cedar.
Niklaus Wirth created Modula-2 in 1976 after his first sabbatical at Xerox given his experience with Mesa, used it to create the Lilith workstation at ETHZ, this was followed a few years later by Oberon for the Ceres workstation, inspired by Mesa/Cedar after his second sabbatical at Xerox.
The OOP extensions that Borland added into Turbo Pascal are actually from Apple's Object Pascal, used to create Lisa's OS and the initial versions of Mac OS, before Apple decided to make the development tools appealing to the growing UNIX workstation market and introduced Macintosh Programmer's Workshop.
On MS-DOS compatible systems, which were written in Assembly, there was a plethora of Basic, Pascal, Modula-2, C and C++ compilers to choose from. Plus business languages like Cobol and xBase.
It was only with the success of Watcom C++ adoption among game developers, thanks to its DOS extender, and the move to OS/2 and Windows 3.1 that C and C++ started to grow in adoption.
However most developers on OS/2 and Windows 3.1 were actually adopting C++ frameworks like CSet++, OWL and MFC, or alternative environments like TPW, Delphi or VB. Mac guys had Powerplant.
On Windows 3.1, C++ patterns like RAII were already common place, and even though each compiler had its own library, all of them provided support for safe strings, vectors and some form of smart pointers.
Writing pure C on Windows, besides Microsoft themselves, has always been mostly done by those porting UNIX stuff into Windows.
Even Microsoft by the time they released the Windows 3.1 SDK, a new set of macros was introduced to try to make it safer to code in plain C.
https://support.microsoft.com/en-us/help/83456/introduction-...
That depends on what you mean by alternative to C. If you are just looking for a "portable assembly" or systems programming language: there were many alternatives.
If you are looking for the systems programming language that has a chance building and running the same source code on multiple different systems then you have to wait until Posix which is tied to C.
Of course you can well argue that once you have Posix, C is not actually required. You just need a language that can call C style functions (this is a trivial exercise for many languages) with compilers for the platforms you are interested in (not trivial, but it is straight forward work that has been done often enough that it is well understood).
Most people mean Posix+C when they say there were no alternatives to C, and in this form they are correct. The other choices may be better for most definitions of better, but there are few alternatives that let you write code and have a chance of it running on something else.
In a way, I always though of POSIX as the C batteries that ANSI didn't want to make part of ANSI C, to make it easier to create compliant compilers.
Which was kind of wasted effort, because to make it easier to port code, many C compilers outside UNIX always bundled a subset of POSIX with them.
Agreed, but that increased the cost of porting a language to the new platform. The larger (and richer) the library the more expensive it is. Unless your large library is built on a smaller internal library (like Posix).
Note that I'm arguing that C was the first to really achieve this. I'm not arguing C is the best choice, not am I arguing that the other languages couldn't have reached that. There are other languages (some better than C) that could have done just as well, but for some reason didn't.
Or protection.